Roadmap to 1.0

[benjchristensen]

I want everyone to know where we're heading and what's left before we hit 1.0.

From the beginning we have allowed ourselves to do breaking changes on each 0.x release as we were aware that the design was not finished when we started this project. We are getting close to being done. Our goal is to release 1.0 in the coming months after stabilizing the API so that it can be relied upon without breaking every couple months.

Project Structure

When we hit 1.0 we intend on splitting out the language adaptors into their own top-level projects such as RxScala, RxClojure, RxGroovy, RxKotlin, RxJRuby etc.

This will allow each project to iterate as needed at their own pace, especially since some will need to continue iterating while the RxJava core stabilizes. For example, if RxScala needs breaking changes it can bump it's major version while RxJava does not. This is particularly important to RxScala for handling changes such as Scala 2.10 -> 2.11 -> 2.12 etc.

Major contrib modules will also be moved out, such as RxAndroid which also needs its own life-cycle.

Outstanding Work

The major items of work to be finished before 1.0 are:

• Backpressure: Backpressure #1000 [Completed]
• Serialization Behavior: Serialization Behavior #998 [Completed in new merge implementation]
• Scheduler API: Proposed Scheduler Interface Change for 0.18 (yes, again) #997 Completed
• Remove all deprecated methods/classes [Completed] #1053 #1621
• Finish migrating all operators to using lift and chaining the Subscription via Subscriber (current priority) [Completed]

The primary goal is to nail down the public API. New functionality can come in 1.1, 1.2, etc. The secondary goal is for all operators to work as advertised (regarding unsubscribe, back pressure and non-blocking). There will always be bugs, that's why 1.x.y will still be active after release, but the desire is to not need to ship 2.x soon after 1.x as this is a low level library that once entrenched becomes hard to migrate (we create significant pain at Netflix on each 0.x release).

Going Forward

Please comment if you feel there are other critical things to achieve before 1.0. The fastest way to getting us to 1.0 is helping us achieve the work stated above.

[benjchristensen]

Based on discussion with @headinthebox we will probably move rx.joins into a separate module prior to 1.0 since this API does not yet feel finalized. If over time it matures it can be brought back into core in 1.1, 1.2, etc ... but we can't leave things in the core that may need public API changes.

---

Completed in #1049

[benjchristensen]

Removing deprecated methods/classes. A few still remain to be done before 1.0 but can't be done until all operators are migrated:

• Observable.OnSubscribeFunc
• Observable.create(OnSubscribeFunc)
• SafeObservableSubscription
• SafeObserver

[benjchristensen]

Migrating operators to using lift and Subscriber is now the priority for making progress to 1.0.

I will personally be tackling the back pressure prototypes and eventually propose the change via a pull request. This work is being tracked at #1000.

[akarnokd]

I'd like to start working on these. Is the Scheduler API stable for now, so any changes to it won't require me to rebase? In addition, could you setup an issue with checkboxes dedicated to each remaining operator?

[benjchristensen]

Thanks @akarnokd, I really appreciate your help on this.

I believe the Scheduler changes are stable (unless someone quickly comes up with a better name for Inner). I'm just waiting on @mattpodwysocki to review the changes so we are in sync with RxJS/Rx.Net going forward before releasing 0.18.

On the topic of migrating all of the operators, I think there are a few things we should nail down as patterns we want to follow so operator implementations are similar (to ease ongoing maintenance).

Things I can think of are:

• a common, well tuned ring buffer (SPSC)
  • ring buffers will become useful in the async operators as well once back pressure allows us to be bounded
• how we apply locking/synchronization
  • we have various different approaches and styles right now
  • when should state machines be used?
  • when should synchronized be used?
  • when should lock be used?
• when should OnSubscribe vs Operator + lift be used?
• how should error handling be done?
  • see map as an example

Perhaps we should create a README inside the rx.operators package meant as an internal coding guide for decisions to these and other things we determine as practices we want to follow?

What patterns do you think we should be following to the above questions and others you are considering?

[benjchristensen]

Version 0.18.0 was released with the Scheduler changes, removal of rx.joins and majority of deprecation cleanup.

[akarnokd]

Sorry for the delay, but I needed to implement a few operators to establish a pattern for myself.

Ring buffer
Whenever buffering is required with fixed size, and there is only addition and removal, one should use ArrayDeque instead of LinkedList. Note however that ArrayDeque grows by doubling its capacity and may consume extra memory beyond the intended fixed size. Eventually, we might need a custom ringbuffer implementation.

Locking & Synchronization
For light contention, such as when an observable and a scheduled action races for a shared object, there isn't much difference between the two. Therefore, I prefer using synchronized as it is more readable. For heavy contention, state machines with randomized backoffs on CAS failures seem to be the way to go, but implementing such logic is not trivial. So in case a new concurrent operator needs to be implemented, start off by using synchronized blocks to establish the base behavior then let one of the experts rewrite it to a state machine.

OnSubscribe vs. Lift
Use OnSubscribe if the operator works as a primary source of events (from) or there are many source observables that need to be mangled together (merge). Use lift if the operator "modulates" a primary source of events (map).

[daschl]

@akarnokd wrt to backpressure and groing. Shouldn't - if you want to mirror a ringbuffer - the ArrayDequeue or whatever be initalized to the required size upfront so you dont need to grow it? You can preallocate the slots more or less, so you also have less garbage flying around (like with the LinkedList).

[akarnokd]

From the JDK source code ArrayDeque appears to round up the initial capacity to the next power-of-2 and doubles it everytime it grows. In addition, the usage pattern in RxJava is as follows: add, check-size, remove, i.e., if it is full, it will add the latest item, discover that it has overgrown the expected capacity and remove on item. If the capacity happens top be power-of-two, adding one element doubles the size which is then wasted.

[benjchristensen]

wrt to backpressure and groing

Once we implement #1000 there should be very few places left that require unbounded buffers and we can leverage ring buffers even in places such as observeOn and zip.

[benjchristensen]

@akarnokd

Thanks for the feedback, I'll take a stab at writing a README to record this for our sake and anyone else getting involved.

[zsxwing]

One more question: when we need to handle the case that the source Observable doesn't obey the Rx contract?

In my opinion, if SafeSubscriber can not help handle the misbehaving Observable, we need to handle it in the operator.

[benjchristensen]

What use case are you referring to? Operators assume sequential, correct behavior.

[zsxwing]

The comment from @akarnokd here: #1115 (comment)

[benjchristensen]

Answered the specific question on that issue, more generally ... most operators should not need to concern themselves with unsubscribe behavior, unless their purpose is to unsubscribe, such as take. Operators should also be able to assume sequential notifications and contract compliance. The serialize operator is for making a source be sequential if it's not. We don't however have anything to force an operator to not emit anything after onComplete or onError. Some operators will stop functioning after a terminal state (such as groupBy) while others will work just fine (like map).

[benjchristensen]

With the release of 0.18 I am not aware of any further major breaking changes beyond the final removal of the last remaining deprecated signatures (such as Observable.OnSubscribeFunc).

There are a few operators (such as buffer) that @headinthebox is reviewing for correctness and possibly change their behavior from Rx.Net (which we have tried to match) based on his experience over the past couple years.

The goal is that within a few months we can have a stable API and release 1.0 and let the 1.x version live for a long time without further breaking changes.

[ldaley]

@benjchristensen I'm happy to help with any Gradle related stuff.

[benjchristensen]

Thanks @alkemist. I don't know enough quite yet to know what to ask for, but once I do I'll update here.

[benjchristensen]

For all contributors involved in this ... for a variety of reasons I'd really like to push towards having a 1.0 Release Candidate with the API locked down and all major features/bugs resolved by July 1st. This is only ~5 weeks away, but I intend on focusing as much as I can towards making this happen.

Once we hit the 1.0RC phase, I expect we'll stay on RC for a while (month or two?) to ensure testing so that 1.0 Final can be considered battle tested.

Issue #1000 is the biggest hurdle to achieving this as all of the other issues seem to almost be resolved.

As we approach 1.0RC we will also split the major sub-modules into their own top-level projects.

I'd appreciate input on this.

[akarnokd]

A few issues with rxjava-core is covered by PRs and there seems to be things not properly exposed in Scala, but otherwise the the current standing is okay. I don't know about the standing with the new computation scheduler. However, backpressure is a huge undertaking; most of us need to learn a new paradigm and way of coding, and it will certainly affect all operators. Even if I had the time, I couldn't make it under a month, not to mention, in a battle-ready fashion.

[benjchristensen]

A few issues with rxjava-core is covered by PRs and there seems to be things not properly exposed in Scala, but otherwise the the current standing is okay.

We can let RxScala continue pre 1.0 if we want? Our intent is to split it out.

I don't know about the standing with the new computation scheduler.

I think we can get this stuff solved fairly soon and then let it bake for a while. None of that requires API changes though.

backpressure is a huge undertaking;

It is indeed non-trivial but we've done enough prototyping that it's not completely unknown.

most of us need to learn a new paradigm and way of coding, and it will certainly affect all operators.

Moved over to #1000 (comment) for discussion instead of hijacking this issue with a long discussion on backpressure.

[headinthebox]

Scala bindings are just catchup, nothing fundamental, but I think it is fine to start separating the various bindings so they can move at their own pace such that we can stabilize the core.

[zsxwing]

I think I should be able to finish the remaining methods of RxScala in 1-2 weeks.

[benjchristensen]

Deprecations are being removed in #1621

[akarnokd]

Before locking the API down, it might worth considering replacing Action0 with Runnable across all use places. Doing this would reduce the need to wrap one into the other in schedulers (and thus less memory allocation) and the API would get a bit closer to Java 8.

[benjchristensen]

I'd rather not replace Action0. The API has worked very well and is consistent by having Action0, Action1, etc ... instead of Runnable (with run()) and Action1 etc (with call()).

It is very rare to be working with Runnable directly when using RxJava, and the memory benefit would not be realized anyways since we always wrap the Action0 in ScheduledAction for functional reasons, and it implements Runnable.

[headinthebox]

The consistency of having Actioni is hard to beat.

[benjchristensen]

First 1.0 release candidate has been released: https://github.com/ReactiveX/RxJava/releases/tag/v1.0.0-rc.1

[ldaley]

@benjchristensen will 1.0 include reactive stream adapters?

[benjchristensen]

I just created the project for it: https://github.com/ReactiveX/RxJavaReactiveStreams

It will be released as artifact rxjava-reactive-streams and depend on rxjava-1.+ and reactive-streams-* and have its own release cycle so it can rev however it needs to separate from RxJava itself.

I'm working on getting RxJava 1.0.0 Final and getting the many separated projects building again so likely won't focus on this adapter until after that.

[ldaley]

Great thanks, I’ll do my best to get it started. 

[benjchristensen]

My intent is to have a class such as RxReactiveStreams with methods on it converting in both directions.

[benjchristensen]

As part of the move to 1.0 I'm considering moving the join and groupJoin operators from RxJava into RxJavaJoins (https://github.com/ReactiveX/RxJavaJoins) as I don't feel confident they are battle tested.

Anyone have a reason not to do this?

/cc @headinthebox

[benjchristensen]

Status Update: September 19, 2014

Build and Release: I have been a bottleneck to getting all the split projects released with the new Maven Central io.reactivex IDs using the new build process. Theoretically most projects should be able to get working quickly now that RxJava itself is building and releasing on the new process. The RxScala and possibly RxAndroid projects may have some extra work to achieve their goals.

Release Candidates: It seems there will need to be some further time on release candidates due to the backlog of items we have accrued as shown here: https://github.com/ReactiveX/RxJava/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.0

There is also one blocker affecting Netflix production that is not yet identified enough to have an issue opened. We found it while upgrading from 0.19 to 0.20. Canary tests showed that 0.20 worked, but after a longer period of testing we experienced a very hard to identity bug and the symptoms go away when we rollback to 0.19. Due to this and the other open issues linked to above I am not yet confident calling 1.0 "Final" as I want the 1.0 release to mean it is stable and can be trusted, even though we will surely have 1.x.y releases with further bug fixes, features and improvements.

My personal time the past couple weeks has been pulled in different directions as well so I have caused things to slow down. I hope this won't last much longer so I can help get through the final work for 1.0 Final.

[benjchristensen]

Builds are now working for the following projects:

https://github.com/ReactiveX/RxScala/releases/tag/0.21.1
https://github.com/ReactiveX/RxGroovy/releases/tag/v1.0.0-rc.1
https://github.com/ReactiveX/RxKotlin/releases/tag/v0.21.0
https://github.com/ReactiveX/RxApacheHttp/releases/tag/v0.21.0
https://github.com/ReactiveX/RxRoboVM/releases/tag/v0.21.0
https://github.com/ReactiveX/RxQuasar/releases/tag/v0.21.0
https://github.com/ReactiveX/RxJavaDebug/releases/tag/v0.21.0
https://github.com/ReactiveX/RxJavaAsyncUtil/releases/tag/v0.21.0
https://github.com/ReactiveX/RxJavaComputationExpressions/releases/tag/v0.21.0
https://github.com/ReactiveX/RxJavaJoins/releases/tag/v0.21.0
https://github.com/ReactiveX/RxJavaMath/releases/tag/v0.21.0
https://github.com/ReactiveX/RxJavaString/releases/tag/v0.21.0

Not all of them are on Maven Central yet but are on their way. They can be seen on Bintray: https://bintray.com/reactivex/RxJava

The following are not yet released:

• RxSwing: due to KeyEventSourceTest.testObservingPressedKeys Failing RxSwing#4
• RxJavaFx: Build Failing RxJavaFX#3
• RxJRuby: Build Failing RxJRuby#2
• RxAndroid: RxAndroid Build RxAndroid#2

[headinthebox]

Like!

[benjchristensen]

RxAndroid has been released on the new io.reactivex artifact: https://github.com/ReactiveX/RxAndroid/releases/tag/v0.21.0

[benjchristensen]

@headinthebox and I are targeting November 18th as the release date for RxJava 1.0 Final.

All blocking issues appear to be resolved as of 1.0.0-rc.5. The next month will be for other bug fixes found while using RxJava 1.0 release candidates and to provide time for testing and usage.

If anyone has critical bugs or problems with the public API (method or class signatures) please report them now.

[benjchristensen]

1.0.0-rc.10 is released with all known bugs fixed and API changes made. There are new features and backpressure work that will continue in 1.0.x (https://github.com/ReactiveX/RxJava/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.0.x) but unless we find something we've missed in the next week 1.0.0-rc.10 is either what will release as 1.0.0 or very close to it.

Please provide feedback on either:

a) critical bugs
b) mistakes in the public API

I would appreciate you trying 1.0.0-rc.10 in your apps to make sure it works.

Thank you.

[benjchristensen]

I think we're ready for 1.0 Final as of 1.0.0-rc.11. Does anyone know of any reason not to release 1.0 Final on Tuesday?

There are obviously still bugs, performance improvements we can pursue and functionality to explore and add, but we need to stabilize and commit to the API at some point, so all of that will come in future releases.

Is there anything more that must be done before 1.0 Final?

Here are the near-term milestones:

• 1.0 https://github.com/ReactiveX/RxJava/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.0
• 1.0.x https://github.com/ReactiveX/RxJava/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.0.x
• 1.1 https://github.com/ReactiveX/RxJava/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.1

[benjchristensen]

/cc @zsxwing @samuelgruetter @akarnokd @jmhofer @mairbek @mttkay @mattrjacobs @davidmoten @jbripley @abersnaze @JakeWharton @headinthebox @daschl and anyone else watching this project.

Any reason not to release 1.0 Final in 3 days on Tuesday, November 18th?

Here is the Javadoc with the public API as of 1.0.0-rc.11 http://reactivex.io/RxJava/javadoc/rx/Observable.html

[daschl]

No objections from my side - we'll then already ship 1.0.0 with 2.0.2 first werk of december (currently 2.0.1 is on rc9)

Awesome work from all involved :) and special kudos to you @benjchristensen of course

[mttkay]

+1

On Sat Nov 15 2014 at 10:29:11 PM Michael Nitschinger <
notifications@github.com> wrote:

No objections from my side - we'll then already ship 1.0.0 with 2.0.2
first werk of december (currently 2.0.1 is on rc9)

Awesome work from all involved :) and special kudos to you @benchristensen
https://github.com/benchristensen of course

—
Reply to this email directly or view it on GitHub
#1001 (comment).

[akarnokd]

I think the merge bug needs to be resolved, but otherwise the release can proceed.

[zsxwing]

-1. @benjchristensen I tried to test rc.11. However, it cannot be found in maven. I checked https://bintray.com/reactivex/RxJava/RxJava/1.0.0-rc.11/view/files/io/reactivex/rxjava/1.0.0-rc.11 here and found the pom file didn't publish.

[YannRobert]

Please include the #1886 bugfix by @akarnokd in 1.0

[zsxwing]

+1. Tested RxScala with rc12.

[benjchristensen]

@YannRobert @akarnokd #1886 was included in rc.12 and will be part of 1.0
@zsxwing Thanks for confirming.

I'm proceeding to release 1.0.0