Refactor communication between Pipeline Components

[oryx1729]

Passing data between Components

Handling of kwargs

The nodes' run() methods no longer need to deal with kwargs.

The signature of the run() should include input data primitives(query, documents, answers, ...) and params (top_k, filters, ...) that it needs to work with.

Passing Parameters

Parameters for the pipeline execution must be passed as a param dict in the pipeline.run() method. By default, Pipeline attempts to match all params to each node.

To target a specific node, it must be explicitly specified by the node name. For instance, pipeline.run(query="Why?")

Validating Parameters

All targeted parameters(as described above) are now validated. For instance, supplying top_p instead of top_k will now return an error.

Adding "debug" information

Nodes can return a _debug key in the output that gets appended in the final output of a request. For example,

class MyComponent:
    def run(....):
        output = {..., "_debug": "my debug information ..."}  # the _debug key can also be a dict
        return output, "output_1"

will get transformed in final response like,

{"answers": ..., "_debug": {"MyComponent": "my debug information ..."}}

⚠️ Breaking Changes

• component params like top_k, no_ans_boost for Pipeline.run() must be passed in a params dict
• component specific top_ks like top_k_reader, top_k_retriever are now replaced with top_k. To disambiguate, the params can be "targeted" to a specific node. For instance, pipeline.run(query="Why?", params={"Retriever": {"top_k": 10}, "Reader": {"top_k": 5})

[tholor]

Nice. Good first step :)

Left a few comments

• the "graph validation" part before calling pipe.run() is still missing

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[tholor]

Looking great!

Left a few minor comments. Mostly around documentation.

• I think it would also make sense to add a few more test cases (e.g. testing the _debug field).
• Did you test the tutorials? especially the ones on pipeline (11) and query classifier (14)? I only had a superficial review there...
• We need to update documentation (especially the usage page in the docs + the pipeline tutorial). I am fine with doing that in a separate PR, but we should do it right away after merging this one so that others can understand how to use it + the content is still very fresh in our memory.

[tholor]

We could probably also use the new Doc2Answer node instead here. But I think it's not really the scope of this PR

[tholor]

LGTM!