Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor replicas config for Ray Pipelines #1378

Merged
merged 3 commits into from
Aug 31, 2021
Merged

Conversation

oryx1729
Copy link
Contributor

In the current implementation of RayPipeline, replicas can be configured for each component in the YAML. For instance,

version: '0.9'

components:
  - name: ESRetriever
    type: ElasticsearchRetriever
    replicas: 2   # create replicas
    params:
      document_store: DocumentStore
  - name: DocumentStore
    type: ElasticsearchDocumentStore

pipelines:
  - name: ray_query_pipeline
    type: RayPipeline
    nodes:
      - name: ESRetriever
        inputs: [ Query ]

The replicas is a configuration property for the pipelines rather than of the individual component. This PR updates that and adds a test case. With this, the YAML would look like this:

version: '0.9'

components:
  - name: ESRetriever
    type: ElasticsearchRetriever
    params:
      document_store: DocumentStore
  - name: DocumentStore
    type: ElasticsearchDocumentStore

pipelines:
  - name: ray_query_pipeline
    type: RayPipeline
    nodes:
      - name: ESRetriever
        replicas: 2   # replicas are now configured here
        inputs: [ Query ]

@lalitpagaria
Copy link
Contributor

Can we please update/add documentation as well.

@oryx1729 oryx1729 requested a review from tholor August 30, 2021 15:18
Copy link
Contributor

@lalitpagaria lalitpagaria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@oryx1729 oryx1729 merged commit a71180a into master Aug 31, 2021
@oryx1729 oryx1729 deleted the fix-ray-replicas branch August 31, 2021 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants