Removing probability field from answers in favor of score field

[julian-risch]

Proposed changes:
This PR drops the field probability from answers and documents in favor of the field score. The default behavior is now that the field score contains the value that was previously stored in probability. However, the FARMReader also allows switching back to the old scores by setting use_confidence_scores to False (default is True).
closes #1220

Main changes:

• no_answer prediction now uses score ranging between 0 and 1:

  haystack/haystack/reader/base.py

  Line 45 in dda0e85

  "score": float(expit(np.asarray(no_ans_score) / 8)), # just a pseudo prob for now

• FARMReader allows switching between new scores ranging between 0 and 1 (confidence scores) and old scores:

  haystack/haystack/reader/farm.py

  Line 55 in dda0e85

  use_confidence_scores: bool = True

• FARMReader extracts new or old scores from QAPreds depending on this switch:

  haystack/haystack/reader/farm.py

  Line 577 in dda0e85

  "score": ans.confidence if self.use_confidence_scores else ans.score,

• I relaxed an assertion in one of the test cases. Previously, it checked that the correct answer is the top prediction. It had the best score but only the second best probability so it worked in the past. Now, I changed the test so that it checks whether the correct predictions is within the top 2 predictions as the order of the ranked predictions slightly changed.

@tholor I came across some points in the code where a plain dictionary could be replaced with an Answer object:

haystack/haystack/reader/transformers.py

Line 134 in dda0e85

answers.append({

haystack/haystack/generator/transformers.py

Line 278 in dda0e85

cur_answer = {

haystack/haystack/reader/base.py

Line 44 in dda0e85

no_ans_prediction = {"answer": None,

haystack/haystack/reader/farm.py

Line 576 in dda0e85

"answer": ans.answer,

haystack/haystack/pipeline.py

Line 602 in dda0e85

cur_answer = {

Status (please check what you already did):

• First draft (up for discussions & feedback)
• Final code
• Added tests -> I added one test case that checks the behavior of the newly introduced switch use_confidence_scores in FARMReader. If use_confidence_scores is set to False, the old scores are used, which can be larger than 1.0
• Updated documentation -> I removed mentions of "probability" from docs

[tholor]

Looks all good to me. Two questions:

• I assume Summarizer, Re-Ranker and classifier did not use any probabilities before?
• Should we scale score always between 0-1 now? I think this would simplify communication to users a lot. I assume this scaling is currently not the case for Elasticsearch (see comment).

[julian-risch]

Looks all good to me. Two questions:

• I assume Summarizer, Re-Ranker and classifier did not use any probabilities before?
• Should we scale score always between 0-1 now? I think this would simplify communication to users a lot. I assume this scaling is currently not the case for Elasticsearch (see comment).

Thanks for the feedback! Summarizer, Re-Ranker and Classifier don't have a probability field in the returned Document, correct.

The requested change in the elasticsearch document store is included now and I reverted the manual changes to the reader.md file. Further, I implemented the normalization of weights in the JoinDocuments node (as discussed).