You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The error is coming from FARM inferencer. The document_id is missing in the returned json.
Currently this breaks our evaluation.
Tutorial5_Evaluation.py needs a document_id coming back from FARM.
Haystack eval first checks weather the document_id is corresponding to the gold truth, then it checks if the answer string is correct. Since the document_id coming back from FARM is None it never enters the answer string check.
input to the FARM inferencer is a dict:
cur = {
"text": doc.text,
"questions": [question],
"document_id": doc.id
}
output of farm inferencer has set document_id in all cases to None.
The text was updated successfully, but these errors were encountered:
The error is coming from FARM inferencer. The document_id is missing in the returned json.
Currently this breaks our evaluation.
Tutorial5_Evaluation.py needs a document_id coming back from FARM.
Haystack eval first checks weather the document_id is corresponding to the gold truth, then it checks if the answer string is correct. Since the document_id coming back from FARM is None it never enters the answer string check.
input to the FARM inferencer is a dict:
output of farm inferencer has set document_id in all cases to None.
The text was updated successfully, but these errors were encountered: