-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for indexing pipelines #816
Conversation
521f907
to
deceb82
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the design. Differentiation into pipeline types makes sense. Just not sure about the repetition of all instance attributes in the respective methods. This can create quite a mess in the long run...
def clean(self, document: dict) -> dict: | ||
def process( | ||
self, | ||
document: dict, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see why we want the option to overwrite instance attributes here, but it will create quite a mess if we apply this pattern to all potential pipeline nodes (retriever, reader ...). Maybe we can find another, cleaner workaround 🤔
166483a
to
14cfcdb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
The Pipelines were introduced to help build directed graphs from Haystack components to process a query.
This PR extends the Pipelines for building indexing workflows using components like file converters, preprocessors, retrievers, and document stores.
Proposed changes
In the YAML configuration of Pipeline, a new field
type
is added. This can be used in the current version to specifyQuery
&Indexing
pipelines. In the future, additional types, for example,RayQuery
can be added that builds a query pipeline with the Ray framework. This newtype
fields allows components like the Retriever to be "polymorphic". For instance, in the case ofQuery
, it embeds a query, while, in anIndexing
pipeline, it creates embeddings for the given list of documents.There can be different root nodes based on the pipeline. For example, a
Query
pipeline has aQuery
root node, while anIndexing
Pipeline has aFile
root node. For cases where a user might want to use a text blob instead of a file, the REST API module can convert the text blob to a .txt file before passing it on to the indexing pipeline.The components should allow run-time parameters, e.g.,
remove_numeric_tables
to be also configurable during component initialization. This enables defining the default values as part of the Pipeline configuration that can later be overridden by individual requests.Sample Indexing Pipeline