Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add API endpoint to export accuracy metrics from user feedback + created_at timestamp #803

Merged
merged 9 commits into from
Feb 15, 2021

Conversation

tholor
Copy link
Member

@tholor tholor commented Feb 3, 2021

User feedback collected at runtime can be utilized for two use case:
a) fine-tuning models on this custom training data (see https://haystack.deepset.ai/docs/latest/domain_adaptationmd)
b) evaluating models in production (i.e. creating accuracy metrics from the user feedback)

This PR implements basic functionality for b) by adding a new API endpoint /eval-doc-qa-feedback. When calling this endpoint we calculate simple accuracy metrics for answer and document results based on the given user feedback. Optionally, you can also supply filters to narrow down the labels that shall be used:

Example Request:

curl --location --request POST 'http://127.0.0.1:8000/eval-doc-qa-feedback' \
--header 'Content-Type: application/json' \
--data-raw '{
    "filters": {"document_id": ["a52c4f5b-2221-4653-8dc8-31eee2cf0ac4"]}
}'

Example Response:

{
    "answer_accuracy": 0.0,
    "document_accuracy": 0.0,
    "n_feedback": 1
}

Breaking change:

The labels have now a field called created_at that includes an automatic timestamp.

Before:

  • In SQLDocumentStore and FAISSDocumentStore all objects had a field / column created
  • In ElasticsearchDocumentStore labels did not have a field created

After:

  • Labels in SQLDocumentStore, FAISSDocumentStore and ElasticsearchDocumentStore all have a field created_at

@tholor tholor requested a review from tanaysoni February 8, 2021 13:52
@tholor tholor changed the title WIP Add API endpoint to export accuracy metrics from user feedback Add API endpoint to export accuracy metrics from user feedback + created_at timestamp Feb 8, 2021
haystack/schema.py Outdated Show resolved Hide resolved
@tholor tholor merged commit 6798192 into master Feb 15, 2021
@tholor tholor deleted the user_feedback_metrics branch February 15, 2021 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants