Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: move Kafka and Pulsar clients to nar files #395

Merged
merged 14 commits into from
Sep 12, 2023

Conversation

eolivelli
Copy link
Member

@eolivelli eolivelli commented Sep 12, 2023

Summary:

  • the Pulsar client and the Kafka client import too many third party deps into the main classpath
  • move all the clients to .nar files
  • split the langstream-kafka and langstream-pulsar modules into two (langstream-kafka-runtime and langstream-pulsar-runtime)
  • move the "deploy" and "delete" operations of the StreamingCluster (that is executed on the planner) interface to TopicsConnectionRuntime (that contains the client)
  • reworked the loading of the "custom jars", the additional jars must be loaded on a classloader that is on top of the NarClassloader, otherwise the Kafka Connect adapters will try to load the Kafka client and the Kafka Connect APIs from the Connectors (if they bundled them by mistake)

@eolivelli eolivelli marked this pull request as ready for review September 12, 2023 15:18
@eolivelli eolivelli merged commit e170deb into main Sep 12, 2023
8 checks passed
@eolivelli eolivelli deleted the impl/topic-runtime-nar branch September 12, 2023 20:32
benfrank241 pushed a commit to vectorize-io/langstream that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants