Jetstream Support and Broadway Integration Officially

[yordis]

Hey there,

As a programmer, I would love to see the nats-io organization try to support Jetstream and Broadway's integration officially.

@mmmries already created the integration located at https://github.com/mmmries/jetstream. @mmmries is already the top-core contributor of nats.ex. Therefore, not much would change about Who is maintaining but the commitment from the nats-io organization (and potential Synadia 💜) to the ecosystem and hopefully to him.

I noticed and appreciated from Nats the strong support and commitment to owning the ecosystem of libraries. After a while in the industry, many of us are hungry to see that more often, especially with your amazing leadership.

The Broadway integration, Broadway has become a critical library for data processing (probably one of the essential packages in the ecosystem), and alternatives like JetStream as the Messaging Component would be excellent.

• https://github.com/dashbitco/broadway
• https://github.com/mmmries/jetstream

[mmmries]

@yordis thank you for opening this issue. We initially tackled the jetstream/broadway project in a separate project because we wanted to do a lot of small releases and iterate on ideas between 2 different companies that were each wanting to use it.

I think it has started to stabilize and it's probably a good time for me to sync up with the contributors to jetstream/broadway to ask the question of whether we should bring that functionality back into this library directly, or if we want to keep them separate.

@gcolliso are there are any open source policies around having two elixir libraries in the nats-io group that support base nats and jetstream separately? I think most languages put it into a single library and we might want to do that, but I just wanted to check if there are any general guidelines for this from Synada?

[gcolliso]

@mmmries I don't think there are any policies but the clients would normally have JetStream support added to the base clients and then it is enabled if needed.
@ColinSullivan1 any additional thoughts about this?

[ColinSullivan1]

It's really up to the implementors, and it sounds like separate clients was a good decision to start with!

That being said, most nats.io clients include JetStream APIs in the core client. The rationale there is that there may someday be checks at the protocol level for feature flags, ease of use by the community (one stop shopping), simplified release management, and to signal to the community and larger ecosystem that JetStream is a primary feature of NATS.

Unless it defies elixir idioms and would drastically impact the Elixir NATS community, I'd suggest integrating jetstream into the core client and deprecating the older one in lieu of the core NATS client with JetStream.

[yordis]

Hey peeps, any update here? Anything I can do to help anyhow?

[bruth]

Hi @mmmries, I wanted to check in on this and echo what @ColinSullivan1 said as well as in terms of having one client which makes it much simpler to communicate, document, install, etc. I am actively reworking the docs, and putting the basic method of installation and various links to the client resources (docs, package, source).

Happy to assist in any way to move this forward.

[mmmries]

Hi all, thank you for sharing your input. I believe that we are getting close to our first production usages of the separate jetstream library which I would like to complete so that we can learn from any deployment/configuration pains before finalizing the API for the jetstream areas.

Once that is live, I think we'll want to do a final pass of looking at the documentation and client APIs being exposed and ask for feedback. Once we've incorporated any feedback and validated any changes, I think we'll be ready to pull the jetstream functionality into this library and have it released together with optional dependencies for anything jetstream specific (ie broadway and connection).

[cocoaway]

Hello,
is there any update about jetstream integration in Nats elixir client ?

[cocoaway]

Hi all, thank you for sharing your input. I believe that we are getting close to our first production usages of the separate jetstream library which I would like to complete so that we can learn from any deployment/configuration pains before finalizing the API for the jetstream areas.

Once that is live, I think we'll want to do a final pass of looking at the documentation and client APIs being exposed and ask for feedback. Once we've incorporated any feedback and validated any changes, I think we'll be ready to pull the jetstream functionality into this library and have it released together with optional dependencies for anything jetstream specific (ie broadway and connection).

HI mmmries, any update about jetstream integration in elixir client ? Thx.

[acco]

Hey @mmmries, our team is super interested in this. Curious how the first production usages of jetstream are going (and therefore the merge of the two projects)? We'll be going to prod with Nats/Jetstream with Elixir soon and are happy to provide whatever testing or feedback

[mmmries]

Hi @acco (and everyone else). Spiff now has some usage of Jetstream in production and we've been using NATS for quite a while. No significant challenges have come up so far and we would love to get any feedback on how you get on with using this library plus the separate jetstream library for the moment.

Recently @autodidaddict started working with Synadia and he has previous experience working with the separatejetstream library as well. He'll be taking a more active role in this library and you can see some of our ideas about how to unify the two libraries here: #62 (comment)

No timeline to share just yet, but I expect to see some progress in the coming months. One thing that would be really helpful in the meantime is proposals for how a unified API should work? Would you prefer to have a lot of individual functions for things like a buffered publish vs an immediate publish? Would the Jetstream API live in the same module as everything else?

If you have ideas, I would love to see some examples of how you would like to interact with NATS and Jetstream in your own project. Pseudo-code, partial elixir files, etc would all be welcome input.