3.x: Potential migration/compatibility issues

[dariuszseweryn]

Intro:

Since RxJava has a very compelling interface it is widely used as such. Some libraries also use it internally. RxJava 2 and 3 have some breaking changes but they both live in same packages.

Problem:

(I am not an expert in this field but here it goes)
Dependencies in Java are resolved by the full name (i.e. package + class name). Since the package name has not changed between RxJava 2 and 3 there are duplicate classes found during compilation. If two libraries (first level dependencies) have a dependency different major versions of RxJava (as a second level dependency to your app) they will not compile together.

Using gradle it is possible to make a first level dependency use a different second level dependency than it declares by using the below code in build.gradle file:

resolutionStrategy.dependencySubstitution {
    substitute module('io.reactivex.rxjava2:rxjava') with module('io.reactivex.rxjava3:rxjava:3.0.0-RC1')
}

But then there is a good chance that some of those libraries will face binary incompatibilities introduced between RxJava 2 and 3.

This poses a potential logistics problem for the applications consuming RxJava based libraries. There is no way to ensure that those libs dependent on RxJava 2 and 3 will work together — the app developer would need to substitute libs that do not allow to be used with one a specific RxJava version or postpone migration. Developers would potentially need to wait until all their dependencies will migrate to RxJava 3 before they can switch. There is no way to create an interop between RxJava 2 and 3 like it was the case with RxJava 1 and 2.

Question:

Has this problem been considered?

Ideas:

1. Add new RxJava 3 operators to RxJava 2.3.x. E.g.:

• Supplier<T>, Observable/Single.defer(Supplier<T>) but keep Observable/Single.defer(Callable<T>) (mark it as @Deprecated and point to the new function), etc.
• Observable.startWithItem(T) but keep Observable.startWith(T)
• others if possible
  This could make a usable subset of RxJava 2 binary compatible with RxJava 3 allowing it to easily swapped at a later point when adoption is high enough.

2. Keep only Observable, Flowable, Single, Completable and their .subcribe() functions in the main package so they could be exposed as an API of all libraries that are of RxLibrary kind and the operator implementations in some extensions / other packages. This could make the usage more cumbersome in Java (but Kotlin language has extensions so this could be mitigated somewhat). This could make the Rx API more future-proof

This issue is a place for discussion (while somewhat related to #6524)

[akarnokd]

RxJava 3 is developed as a replacement to RxJava 2 so yes, you have to wait for libraries depending on such changed APIs to be upgraded.

This has been discussed before and keeping everything in the same package is the least problematic path for 3.x as it allows libraries not depending on changed components to keep functioning. With a completely separate package, you have to wait for such libraries to upgrade too.

Add new RxJava 3 operators to RxJava 2.3.x. E.g.

RxJava 2 will not get any new minor release or new features.

Which libraries do you have problems with?

[digitalbuddha]

I have an android app that depends on 12 libraries all depending on rxjava2. I don't understand my path forward.

Do I need to wait for all 12 to upgrade if one does and uses a new api?

[ZacSweers]

Why are the new maven coordinates necessary? Every option not involving package renaming will be awkward, the least awkward option would just be to have a major version bump on the existing coordinates. Then you can avoid making consumers do all this build system finagling just to avoid classpath conflicts.

…

On Wed, Jul 31, 2019, 1:43 PM David Karnok ***@***.***> wrote: RxJava 3 is developed as a replacement to RxJava 2 so yes, you have to wait for libraries depending on such changed APIs to be upgraded. This has been discussed before and keeping everything in the same package is the least problematic path for 3.x as it allows libraries not depending on changed components to keep functioning. With a completely separate package, you have to wait for such libraries to upgrade too. Add new RxJava 3 operators to RxJava 2.3.x. E.g. RxJava 2 will not get any new minor release or new features. Which libraries do you have problems with? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6606?email_source=notifications&email_token=AAKMJPR53ZFIK6JWYA6425LQCHFLHA5CNFSM4IIIW5WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3IALBI#issuecomment-516949381>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKMJPQJMF6JSFRJS6DVEA3QCHFLHANCNFSM4IIIW5WA> .

[JakeWharton]

Or, phrased another way, the Maven groupId change was necessary for 1.x to 2.x to ensure that build systems treated them as separate artifacts because they existed in separate packages. Otherwise, a transitive dep on 2.x would have overridden 1.x and you'd get a bunch of NoClassDefFoundErrors.

Now, in 2.x to 3.x, the opposite is true. These exist in the same package and therefore should likely retain the same groupId because the behavior of the build system treating them as providing the same API in the same package is desired (modulo the inevitable-but-hopefully-rare NoSuchMethodErrors).

[dariuszseweryn]

Which libraries do you have problems with?

Basically all libraries that internally depend on RxJava 2 functions that do not have their equivalent in RxJava 3 or vice versa.

With a completely separate package, you have to wait for such libraries to upgrade too.

That is true but it allows for mixing and matching libraries that depend on RxJava 2 and 3 in one application.

RxJava 3 is developed as a replacement to RxJava 2

Currently it looks more like an evolution of RxJava 2, there are only few incompatible changes. As a creator of library I understand the need of dropping support for APIs that are inferior.

The problem is that RxJava is wildly popular and not all libraries will get updated at once. There is gonna be a transition period — which has a potential to be a major pain (recently there was a similar case in React Native world related to gradle update — compile/api/implementation case). In which library maintainers would potentially need to support two versions.

This could be relieved to a degree by adding binary compatible versions of functions to RxJava 2 and @Deprecate those that are about to be removed. Then maintainers could prepare their libraries to be binary compatible with both versions.

Question: What is the advantage of having io.reactivex.rxjava3 groupId instead of having only a major version bump with existing groupId?

[akarnokd]

The groupId for RxJava 2 had its major number added so the pattern is followed for 3.

There is about 6 months until release. Libraries can start the adaptation right now by depending on the release candidates.

Basically all libraries that internally depend on

Again, which libraries are these? I'm sure I can help with making those libraries version agnostic. For example, the defer(Callable) could be implemented locally thus not depending on Rx' implementation. TestSubscriber can be also localized with APIs as needed.

[ZacSweers]

I think it is far better to eat the cost of the number staying the same in maven coordinates. OkHttp just did this recently with good success

…

On Thu, Aug 1, 2019, 11:37 AM David Karnok ***@***.***> wrote: The groupId for RxJava 2 had its major number added so the pattern is followed for 3. There is about 6 months until release. Libraries can start the adaptation right now by depending on the release candidates. Basically all libraries that internally depend on Again, which libraries are these? I'm sure I can help with making those libraries version agnostic. For example, the defer(Callable) could be implemented locally thus not depending on Rx' implementation. TestSubscriber can be also localized with APIs as needed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6606?email_source=notifications&email_token=AAKMJPQN7KNOSABCD2JSNZ3QCL7KZA5CNFSM4IIIW5WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3K7VHA#issuecomment-517339804>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKMJPUYI3LEHS7DB4ERIXDQCL7KZANCNFSM4IIIW5WA> .

[dariuszseweryn]

The groupId for RxJava 2 had its major number added so the pattern is followed for 3

Exactly this move in case of a single dependency migration to RxJava 3 will make the compiler to complain if resolutionStrategy.dependencySubstitution will not be used. Whether or not the rest of RxLibs are compatible with both versions of RxJava.

Again, which libraries are these?

I personally maintain https://github.com/Polidea/RxAndroidBle
It is near impossible to track all libraries that use RxJava under the hood.

For example, the defer(Callable) could be implemented locally thus not depending on Rx' implementation.

As for this moment I do not have an idea how to do that using the APIs that RxJava 2 provides. Maybe a (pre)migration guide would help.

Making the middle consumers (RxLib maintainers) to reimplement parts of RxJava to keep binary compatibility both ways is not ideal. It allows for more implementation errors where the same work could be added to another version of 2.3.x (I know, this won't happen)

Technically whole RxJava could be reimplemented/shadowed internally by each library and only bridge signals at the very thin layer below its API. This would reduce the amount of friction on RxJava major bumps. This is along the lines of my second idea from the original post where we would have only a reactive interface (Observable, Flowable, Single, Completable, Maybe, subscribe, compose) in the main artifact/package that maintains a contract and the plugin operator implementations would live in a separate artifact/package. Consumers would not need to think what engine drives the API.

(I may be drifting offtopic, sorry in that case)

[akarnokd]

What about the following case? We release io.reactivex.rxjava2:rxjava:3.0.0, then any dependency on "latest" or library explicitly targeting RxJava 3 will auto-upgrade in your entire project to 3 silently and libraries which were dependent on 2.x will now be incompatible under the hood, causing runtime failures. This upgrade could even happen with the release candidates being offered by smart IDEs, doesn't it?

[JakeWharton]

Even with a separate groupId that case can still happen. The JVM picks the first occurrence of duplicate classes on the classpath so in the current configuration when both 3.x and 2.x inevitably end up on the classpath it's a tossup based on tiny variants as to which is first. Thankfully if you're using the modulepath or Android your build just up and fails and you spend a chunk of time figuring out how to resolve this, not that the experience is dramatically better than getting a random version.

For people depending on latest this is what they signed up for. I'm not sympathetic to the case that their app will break since that's a reflection on the choice to break binary compatibility, not their choice of a dynamic dependency. If they were concerned with this case they would have specified a range that kept the major version at 2.

[JakeWharton]

And in that first case, even with an explicit RxJava 3.x dependency, the classpath might have 2.x from a transitive dependency that winds up on the classpath first which means you'll run on 2.x at runtime.

Both of the classpath cases are going to be extremely poor user experiences.

[dariuszseweryn]

What about the following case? We release io.reactivex.rxjava2:rxjava:3.0.0, then any dependency on "latest" or library explicitly targeting RxJava 3 will auto-upgrade in your entire project to 3 silently and libraries which were dependent on 2.x will now be incompatible under the hood, causing runtime failures. This upgrade could even happen with the release candidates being offered by smart IDEs, doesn't it?

This is to be expected when following Semantic Versioning pattern. I have a similar thoughts that JakeWharton stated above — using wildcard dependencies is an anti-pattern and asking for troubles on major version changes.
Changing a groupId would end up with an obscure error during compile time and users would need to understand it first. The only advantage I can see is that the failure will be found during the compile time, though I am not sure if gradle will not complain about conflicting major versions of a resolved dependency — if both transitive dependencies use a concrete version of it.

The question evolves — what for we have a major version in Semantic Versioning? We could always create a new library (or publish under new groupId) if backwards compatibility is broken.

[ZacSweers]

From uber/AutoDispose#366 - a good example of the kinds of build system hoops users will have to jump through

api 'io.reactivex.rxjava3:rxjava:3.0.0-RC1'
api 'io.reactivex.rxjava2:rxandroid:2.1.1'
api('com.uber.autodispose:autodispose-ktx:1.2.0') {
       exclude group: 'io.reactivex.rxjava2'
       exclude group: 'io.reactivex.rxjava2:rxandroid'
}
api('com.uber.autodispose:autodispose-android-ktx:1.2.0') {
        exclude group: 'io.reactivex.rxjava2'
        exclude group: 'io.reactivex.rxjava2:rxandroid'
}
api('com.uber.autodispose:autodispose-android-archcomponents-ktx:1.2.0') {
        exclude group: 'io.reactivex.rxjava2'
        exclude group: 'io.reactivex.rxjava2:rxandroid'
}

[akarnokd]

@ZacSweers That wouldn't be much better if we had io.reactivex.rxjava2 and the breaking changes.

RxJava has to move forward and the least murky solution is to have it under a standalone id, founding a fresh set of dependent libraries. One end of it is libraries like RxAndroid that only needs to build and release under io.reactivex.rxjava3 as it does not have to add new features. The other end is libraries that constantly add features and bugfixes where supporting two RxJava versions can become daunting indeed.

In Zac's case and this latter case, there is a question: why would your users switch to RxJava 3 so soon? How long would you have to support 2 versions? Why not follow RxJava 3 and feature freeze your RxJava 2 support?

[JakeWharton]

The breaking changes are not related to the groupId change and we need to stop conflating the two problems.

The binary incompatibility is a problem no matter what groupId is chosen. You've offered:

I'm sure I can help with making those libraries version agnostic.

And the underlying cause of Zac's issue is just that.

However, what Zac was drawing attention to is the hoop-jumping now required in the build system to ensure that two copies of RxJava do not end up on the classpath. This is entirely unrelated to the binary incompatibility and would be required even if 3.x was 100% compatible with 2.x.

The package name and the groupId are fundamentally linked. If one changes, the other should. If one does not change, the other should not.

Either RxJava 3 should reside in io.reactive.rxjava2 groupId or it should change it package name.

[akarnokd]

After careful considerations of all target environments (Android/Desktop/Server) as well as any future major versions targeting Java 9+, I decided the path with the least problems will be having RxJava 3 reside in the new group id and in a new package entirely:

Group ID: io.reactivex.rxjava3
Package: io.reactivex.rxjava3.**.

In addition, the base classes and interfaces (i.e., Flowable, FlowableSubscriber, Observable) will be moved to a subpackage core: io.reactivex.rxjava3.core.Flowable. A reason for this is that with modules, opening up io.reactivex.rxjava3 opens up all subpackages, including internal which can't be hidden then on as far as I know.

Since Flowables are Publisher's, interoperation between the v2 and v3 variants is a given: either use them as is with parameters/arguments declared as Publisher or use Flowable.fromPublisher.

The other types won't be such lucky as ObservableSource is present in both v2 and v3 and isn't external unlike Reactive Streams. Those can't talk to each other without an actual (trivial) bridge library.

[artem-zinnatullin]

I'm glad this is getting resolved, as of interop issues I have a proposal.

Over a cup of coffee w/ @Tagakov we've figured a way to make v2 <-> v3 interop smoother.
Here is the schema:

• RxJava v2 declares <optional>true</optional> dependency on v3 maven artifact
• RxJava v3 declares <optional>true</optional> dependency on v2 maven artifact
• <optional>true</optional> maven dependency means that such a dependency won't appear in the final binary of an Android app or a backend Service/etc if it's packed as uber-jar or ran with a build system nor will it be present on the compilation classpath, unless it has been listed as a non-optional dependency.
• Both RxJava versions will include basic reactive type conversion methods/operators (naming is not final):
  • RxJava v2 example: io.reactivex.Observable.toV3Observable() returning io.reactivex.rxjava3.core.Observable
  • RxJava v3 example: io.reactivex.rxjava3.core.Observable.toV2Observable() returning io.reactivex.Observable
  • Implementation of the conversion methods will live in separate class(es) to prevent ClassLoader from trying to resolve classes that may not be on classpath, return type works fine
• I've checked with Gradle and Buck build systems that they're fine with remote maven dependencies having a dependency cycle between them, will need to check with Maven and Bazel too.
• Sources:
  • Playground project (has README with instructions): https://github.com/artem-zinnatullin/RxJava2To3InteropIntegrationTest
  • RxJava v2 w/ v3 interop branch: https://github.com/artem-zinnatullin/RxJava/tree/az/2.x-to-3.x-interop
  • RxJava v3 w/ v2 interop branch: https://github.com/artem-zinnatullin/RxJava/tree/az/3.x-to-2.x-interop
• Developer experience:
  • Projects using only RxJava v2 will see methods like toV3Observable() but won't be able to call them: IDE will highlight them as errors, build system will fail compilation, v3 won't be included into the binary and v3 interop code will be removed from the final binary if a tool like ProGuard/R8 is used
  • Projects using only RxJava v3 will see methods like toV2Observable() but won't be able to call them: IDE will highlight them as errors, build system will fail compilation, v2 won't be included into the binary and v2 interop code will be removed from the final binary if tool like ProGuard/R8 is used
  • Projects using both RxJava v2 and v3 will be able to convert basic reactive types with interop operators with experience similar to using extension functions in Kotlin for those who remember v1 to v2 interop

I've used this techinique in past in the StorIO project where we had compileOnly dependency on RxJava in a similar way to make RxJava integration optional for the users, Retrofit 1.x used to do similar thing with `true, it worked great.

Caveats:

• The only caveat I can think of is that <optional>true</optional> dependencies are not participating in version conflict resolution in Maven (the build system) User can define optional features, which map to Maven's <optional> dependencies gradle/gradle#867 (comment), this however shouldn't be an issue as any version of RxJava v2 or v3 will have core reactive types present in them. It was an issue with Retrofit 1.x when it started to use some newer RxJava operators while user's version was lower if I remember correctly.

If this all looks good, I'm happy to make PRs to RxJava to make it happen.

[artem-zinnatullin]

To add to that, we've also explored a path where v2 reactive types would extend v3 ones or vice versa with similar <optional>true</optional> dependency so that a v2 ObservableSource for example would be accepted in methods requiring v3 ObservableSource, but this route quickly failed:

• due to child-parent relationship that doesn't work in the other direction: you can't pass v3 ObservableSource as v2 ObservableSource
• due to use of concrete types like Observable instead of ObservableSource in real-life projects that are hard to extend without issues unlike ObservableSource/etc interfaces

[akarnokd]

The new package structure has been released with 3.0.0-RC2 and there is a support library so that v2 and v3 can talk to each other without hidden or overt compilation/runtime problems from before.
This also means that module override tricks no longer work so you have to bridge AndroidSchedulers manually or convert from v2 sources used in Retrofit until these (and many other) libraries start supporting v3.