Add run_batch method to all nodes and Pipeline to allow batch querying

[bogdankostic]

This PR facilitates batch processing with query Pipelines. This is done by adding a run_batch and a corresponding predict_batch/retrieve_batch/... method to each of the nodes. In order to achieve this, transformers dependency has been upgraded to the newest version, because in the version currently used, batch processing for transformers' question-answering-pipeline is not supported.

As decided in #1239, the Pipeline's run_batch method can take a single query or a list of queries as its queries argument and a single list of Documents or a list of lists of Documents as its documents argument. The behavior of the Pipeline depends on whether a single value or list of values are provided. In general, the following applies:

	Single Query	List of Queries
List of Docs	Single query is applied to each doc individually -> answers are returned for each single Document	Each query is applied to each doc individually -> answers are returned for each query-doc-pair
List of Lists of Docs	Single query is applied to each list of Documents -> aggregation of answers on each list of Documents	Each query is applied to its corresponding doc list (based on same index in the provided lists of values) -> aggregation of answers on each list of Documents

if run_batch is called on an indexing pipeline, the pipeline's run method will be called.

Breaking changes

The input and output of the FARMReader's predict_batch method changed. (Therefore, the output of the QuestionAnswerGenerationPipeline also changed.)

Old way

# Old input
FARMReader.predict_batch(
    query_doc_list=[
        {"question": "Some sample query", "docs": [Document(content="sample doc1"), Document(content="sample doc 2")]},
        {"question": "Some sample query", "docs": [Document(content="sample doc3"), Document(content="sample doc 4")]}
    ]
)

# Old output
[
    {"query": "Some sample query", "no_ans_gap": max_no_ans_gap, "answers": Answers from doc1 and doc2, "label": cur_label},
    {"query": "Some sample query", "no_ans_gap": max_no_ans_gap, "answers": Answers from doc3 and doc4, "label": cur_label}
]

New way

# New input
FARMReader.predict_batch(
    queries=["Some sample query", "Some sample query"]
    documents=[[Document(content="sample doc1"), Document(content="sample doc 2")], [Document(content="sample doc3"), Document(content="sample doc 4")]]

# New output
{"queries": ["Some sample query", "Some sample query"], "answers": [[Answers from doc1 and doc2], [Answers from doc3 and doc4]], "no_ans_gaps": [[no_ans_gaps from query1], [no_ans_gaps from query2]]}

Closes #1239

[julian-risch]

Looks quite good to me! I just have a couple of things that we should discuss/change before merging. Here is a list (see my individual comments for details). Happy to have a call anytime.

• batch_size should maybe be an attribute of every node and it should be possible to set it in the init method of each node (yaml pipeline definition)
• Maybe two of the cases list of docs/queries can actually be joined?
• Make error message more extensive and print not only expected type but also given type
• Allow a different filter per query in queries?
• truncation_warning every time predict is executed
• multiple_doc_lists naming: more than one list? FOr example in batch_prediction_single_query_multiple_doc_lists
• Use distilled, tiny model instead of roberta-base-squad in test case
• Transformers upgrade separately would be better in separate PR
• Check whether tutorials are still working (QuestionGenerator could break I think)

[julian-risch]

If the handling of the edge case with a single query makes the implementation much more complex, we could discuss whether queries must be a list in all cases. The edge case would be a list with a single item then.

[julian-risch]

As discussed, we can address this question and decide what to do in a separate PR.

[julian-risch]

This case does essentially the same as "# Docs case 1: single list of Documents -> apply single query to all Documents". So if queries is a string, we could make it a list [queries] and then use the same code for both cases with the for loop for query in queries:. Merging these two cases could simplify the code a little bit. What do you think? I like the readability of the current version. 👍

[julian-risch]

As discussed, we can address this question and decide what to do in a separate PR.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[julian-risch]

Could we maybe add this without much overhead before the next release? Is there an issue already? It would be a shame if we can't use all the batch processing in dc because of that. What we need for that is a self.client.query_batch() call here and then also a query_batch() implementation in IndexClient.

[bogdankostic]

I would suggest to just call query here multiple times, as there is no batch_query endpoint in dc (yet). (In query_batch()in IndexClient we would need to call the documents-query endpoint multiple times.) Once dc has a batch_query endpoint, we can have a query_batch() implementation for the IndexClient. What do you think?

[julian-risch]

Yes, agreed. 👍

[julian-risch]

Looks great to me! 👍 Let's just not forget about query_batch in DCDocumentStore so that we can make use of these nice new batch processing features.