Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load Pipeline with YAML config file #785

Merged
merged 6 commits into from
Feb 2, 2021
Merged

Load Pipeline with YAML config file #785

merged 6 commits into from
Feb 2, 2021

Conversation

tanaysoni
Copy link
Contributor

@tanaysoni tanaysoni commented Jan 28, 2021

This PR adds support to load a Pipeline with a YAML configuration file.

Here's a sample Pipeline YAML:

version: '0.7'

components:    # define all the building-blocks for Pipeline
  - name: MyReader       # custom-name for the component; helpful for visualization & debugging
    type: FARMReader    # Haystack Class name for the component
    params:
      no_ans_boost: -10
      model_name_or_path: deepset/roberta-base-squad2
  - name: MyESRetriever
    type: ElasticsearchRetriever
    params:
      document_store: MyDocumentStore    # params can reference other components defined in the YAML
      custom_query: null
  - name: MyDocumentStore
    type: ElasticsearchDocumentStore
    params:
      index: haystack_test

pipelines:    # multiple Pipelines can be defined using the components from above
  - name: my_query_pipeline    # a simple extractive-qa Pipeline
    nodes:
      - name: MyESRetriever
        inputs: [Query]
      - name: MyReader
        inputs: [MyESRetriever]

To load a Pipeline:

from pathlib import Path
from haystack import Pipeline

pipeline = Pipeline.load_from_yaml(Path("sample.yaml"))
prediction = pipeline.run(query="What is the capital of Germany?", top_k_retriever=10, top_k_reader=3)

@tanaysoni tanaysoni changed the title WIP: YAML configuration for Pipelines Load Pipeline with YAML config file Jan 29, 2021
@tanaysoni tanaysoni requested a review from tholor January 29, 2021 13:32
@tanaysoni tanaysoni force-pushed the pipeline-yaml branch 6 times, most recently from fecce8b to f32501e Compare February 1, 2021 13:11
Copy link
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking very good already. Added a few smaller comments ...

haystack/pipeline.py Show resolved Hide resolved
haystack/pipeline.py Show resolved Hide resolved
haystack/pipeline.py Outdated Show resolved Hide resolved
Copy link
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to merge after formatting the code snipped properly (see comment)

haystack/pipeline.py Outdated Show resolved Hide resolved
@tanaysoni tanaysoni merged commit 8a5dc8f into master Feb 2, 2021
@tanaysoni tanaysoni deleted the pipeline-yaml branch February 2, 2021 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants