Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[agents] New agent 'dispatch' for message routing #536

Merged
merged 8 commits into from
Oct 5, 2023

Conversation

eolivelli
Copy link
Member

@eolivelli eolivelli commented Oct 5, 2023

Fixes #454

Summary:

  • this patch adds a new processor 'dispatch' that is able to dispatch the record to different topics depending on a condition

This is an example:

topics:
  - name: "input-topic"
    creation-mode: create-if-not-exists
  - name: "topic1"
    creation-mode: create-if-not-exists
  - name: "topic2"
    creation-mode: create-if-not-exists
  - name: "default-topic"
    creation-mode: create-if-not-exists
pipeline:
  - name: "Dispatch"
    type: "dispatch"
    input: input-topic
    output: default-topic
    id: step1
    configuration:
      routes:
         - when: properties.language == "en"
           destination: topic1
         - when: properties.language == "fr"
           destination: topic2
           action: dispatch
         - when: properties.language == "none"
           action: drop

Important notes:

  • the routes are evaluated in order, as soon as one condition is verified the message is sent to the destination and the processing stops
  • if no route matches then the record is sent to the "output", that can be a topic or another agent, as usual
  • for each route you can define an action: dispatch or drop, the default is "dispatch" that means to send the message, with "drop" the message is silently discarded (not emitted to the downstream agent)

@eolivelli eolivelli marked this pull request as ready for review October 5, 2023 09:08
Copy link
Member

@nicoloboschi nicoloboschi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just not very sure about the empty destination

@eolivelli
Copy link
Member Author

@nicoloboschi I added support for "action: dispatch|drop" PTAL

@ConfigProperty(
description =
"""
Destination of the message. If the destination is empty the message is discarded
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please update here as well

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

name = "Text extractor",
description =
"""
Extracts text content from different document formats like PDF, JSON, XML, ODF, HTML and many others.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you fix it ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

routes.forEach(
r -> {
String when = ConfigurationUtils.getString("when", "", r);
String destination = ConfigurationUtils.getString("destination", "", r);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we check for blank text and raise error if action == dispatch

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, also on the planner side

@eolivelli eolivelli merged commit 28763dd into main Oct 5, 2023
8 checks passed
@eolivelli eolivelli deleted the impl/routing-agents branch October 5, 2023 12:41
benfrank241 pushed a commit to vectorize-io/langstream that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add agent that can write to one of many possible topics
2 participants