PIP-205: Reactive Java client for Apache Pulsar

[lhotari]

Status: Delivered - https://github.com/apache/pulsar-client-reactive
Mailing list thread: https://lists.apache.org/thread/9n961zvr5y4g77n7pfsptg796r2obfo1

Motivation

There's a need to "go reactive from end-to-end" when building modern reactive applications with platforms such as Spring Reactive.
There are ways to adapt the Apache Pulsar Java client async API calls to Reactive Streams with a few lines of code.
However, a lot will be missing and achieving the complete solution will require much more effort.

A better solution would be to have first-class support Reactive Streams in Apache Pulsar.

Reactive Streams is an interoperability specification and there are multiple implementations for the JVM.
It's not about a single programming language.
For example, a Reactive client for Apache Pulsar supporting Reactive Streams can be used together with Project Reactor / Spring Reactive, Akka Streams, RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others.

Goal

Provide Reactive Java client for Apache Pulsar

The Reactive Java client for Apache Pulsar exposes a Reactive Streams compatible Reactive client API for Apache Pulsar.
Reactive programming is about non-blocking applications that are asynchronous and event-driven and require a small number of threads to scale. The Reactive Java client for Apache Pulsar supports non-blocking reactive asynchronous back pressure for producing and consuming messages so that the producing or consuming pipeline doesn't get overwhelmed by producing or consuming.
Libraries that support Reactive Streams provide a programming model that is efficient and optimal for message producing and consuming (processing) use cases.

API Changes

Establish a Reactive Streams compatible client API for Apache Pulsar.
This client will be published in Maven central as a library.

Implementation

There's an existing proof-of-concept available at https://github.com/datastax/reactive-pulsar .
This implementation will be used as a reference for an entirely new implementation that is started as a new repository under the Apache Pulsar project.

The proposal for the repository location is https://github.com/apache/pulsar-client-reactive .
The Maven central group Id is "org.apache.pulsar" and the main artifact id is "pulsar-client-reactive".
The root package name is "org.apache.pulsar.reactive.client".

The implementation will provide an interface module that abstracts the Reactive client API.
This interface is implemented by wrapping the current Apache Pulsar Java client and adapts the existing async Java API to the the Reactive client API.
The reason for this particular detail is that it is possible to provide a native Reactive client later while having the possibility to start developing applications immediately using the Reactive client API. Applications depending on the API will be able to migrate to use the native Reactive client with minor or no changes when it becomes available.

Anything else?

By having an official Reactive Java client for Apache Pulsar, it will provide a way to contribute and improve the official client.
Other opensource projects might want to provide support for using Apache Pulsar within reactive application frameworks. Without an official reactive client, this becomes hard, since open source projects would like to use stable client dependencies instead of a hobby project provided by an individual.
There are several members within the existing Apache Pulsar contributors and committers that have expressed the desire to contribute to a Reactive client for Apache Pulsar and are willing to maintain the new repository. With the new repository and sub-project we will most likely see new active contributors and could possibly appoint new Apache Pulsar committers to the project to empower the developers working on this new sub-project.

[tisonkun]

@lhotari Cool! Several comments here:

The root package name is "org.apache.pulsar.reactive.client".

org.apache.pulsar.client.reactive?

For example, a Reactive client for Apache Pulsar supporting Reactive Streams can be used together with Project Reactor / Spring Reactive, Akka Streams, RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others.

Does the test plan include integration with all or part of the framework listed here?

There are several members within the existing Apache Pulsar contributors and committers that have expressed the desire to contribute to a Reactive client for Apache Pulsar and are willing to maintain the new repository.

Could you share the links for these statements?

[lhotari]

@lhotari Cool! Several comments here:

The root package name is "org.apache.pulsar.reactive.client".

org.apache.pulsar.client.reactive?

thanks for the comments @tisonkun

I was thinking about that option. However, I intentional proposed org.apache.pulsar.reactive.client.
I was simply thinking that it reads better. For the module and repository name I'd rather have pulsar-reactive-client than pulsar-client-reactive, but since there's an existing naming convention for the client modules, I thought it would be good to follow that.
For the package name I was thinking that there's no need to make it match the module name exactly.
Another reason in the back of my head was to make the package name globally unique. Both are unique, but it felt more unique when using org.apache.pulsar.reactive.client.
What is your concern about the package name?

For example, a Reactive client for Apache Pulsar supporting Reactive Streams can be used together with Project Reactor / Spring Reactive, Akka Streams, RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others.

Does the test plan include integration with all or part of the framework listed here?

This is an open source project and it's open for contributions. It's possible to make changes after we get things started. The main proposal is about making the decision to start the work and open it for contributions.

Reactive Streams implementations use the Reactive Streams TCK to ensure that they have been implemented according to a spec. Interoperability across implementations is supported on all implementations. There could be some value in providing optional libraries that add direct support for a particular library, let's say Akka Streams, without having the need to expose types of the Reactive Streams implementation used to implement the Reactive client. This optional library could also shade the implementation library completely if that is useful for some reason. However this is not part of the initial scope of the Reactive client and the Apache Pulsar project will decide how it is developed further. Akka Streams can be used also without any of this kind of special adaptation. Docs for Akka Streams Reactive Streams interop. Similar solutions exist for all Reactive Streams implementations.

There are several members within the existing Apache Pulsar contributors and committers that have expressed the desire to contribute to a Reactive client for Apache Pulsar and are willing to maintain the new repository.

Could you share the links for these statements?

That's not something that typically is expressed in a statement that can be shared by a link. :) What are you specifically interested to know?

[tisonkun]

What is your concern about the package name?

Yep. I can see the reason that reactive.client is more suitable to be globally unique. I'm interested to learn your thoughts on the alternative. It sounds reasonable to me.

open it for contributions

Agree. A proposal can go with a test plan, I'm trying to ask for the plan you'll cover in this turn.

What are you specifically interested to know?

I'd expect conversations on issues or a public channel where more contributors show their interests :) Although I support this proposal, I didn't see other users who ask for such a functionality.

[lhotari]

What are you specifically interested to know?

I'd expect conversations on issues or a public channel where more contributors show their interests :) Although I support this proposal, I didn't see other users who ask for such a functionality.

We have a mailing list thread for this PIP, it's https://lists.apache.org/thread/xkfl5px7qgt5rrxw5pj0g318r6mbdlz1 .

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[hpvd]

if I get it right this can be found in its own repository now:
https://github.com/apache/pulsar-client-reactive

[tisonkun]

@hpvd Thanks for bringing this issue up. I agree to close this issue as completed since we even released a few versions of pulsar-client-reactive already :)