Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[agents] Add support for using Camel components as sources #644

Merged
merged 10 commits into from
Oct 25, 2023
Merged

Conversation

eolivelli
Copy link
Member

@eolivelli eolivelli commented Oct 24, 2023

This is still a POC.

Summary:

  • add a camel-source agent that allows you to use any Camel component as a Source for LangStream
  • add a sample application that uses The GitHub camel connector
  • it is possible to choose an header coming from the Camel message and set is as key of the output record
  • the camel-source implements the Camel exchange protocol and it acks the messages completely processed by the pipeline, and it is also notifying failures

With Camel components LangStream fully support at-least once semantics

Notes:
we are not bundling any Camel connectors, you have to add the connector as a dependency, as it is shown in the sample application. The "dependency" mechanism that you use for JDBC and for Kafka Connect works also for Camel

@eolivelli eolivelli marked this pull request as ready for review October 25, 2023 10:25
String paramName = entry.getKey();
Object value = entry.getValue();
if (value != null) {
if (componentUri.contains("?")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be done with java.net.URI ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if Camel components URI are real URIs, IIRC there are some tricks, like RAW(xxx)

@eolivelli eolivelli merged commit 9c8ef1a into main Oct 25, 2023
10 checks passed
@eolivelli eolivelli deleted the impl/camel branch October 25, 2023 15:39
benfrank241 pushed a commit to vectorize-io/langstream that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants