Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark 9/19 #2

Open
akarnokd opened this Issue Sep 19, 2016 · 5 comments

Comments

Projects
None yet
4 participants
@akarnokd
Copy link
Owner

akarnokd commented Sep 19, 2016

See the most recent comparison in #7.

Comparative benchmark

  • RxJava 1.2 (9/17)
  • RxJava 2.0.0-DP0-SNAPSHOT (9/15)
  • Reactor Core 3.0.3-BUILD.SNAPSHOT (9/19)

Environment:

  • i7 4790 stock settings
  • Windows 7 x64
  • Java 8u102
  • JMH 1.13
  • -XX:MaxInlineLevel=20

Green is good, red is not so good.

Subscribe to a simple source

image

Just comparing 1.x and 2.x

image

Remarks:

  • Subscribing to any type in 2.x has lower overhead than 1.x.

Streaming multiple values (synchronous)

image

Remarks:

  • There seems to be some extra overhead in the backpressured concatMapXRange (both 2.x and Reactor).
  • Also flatMapJust has extra overhead for all 2.x types (could be unavoidable atomics).
  • flatMapXRange 1M (which is flatMapJust 1M essentially) suffers the same overhead.
  • 2.x Observable.flatten is not optimized yet and delegates to plain flatMap
  • Otherwise, nice improvements all over the board.

Asynchronous and pipelined streaming

image

Remarks:

  • There seems to be some extra overhead or missing optimization from Flowable async 1.
  • 2.x Observable lack of optimization (1-case) and likely buffer-bloat in longer streams.
  • Generally not bad, but such benchmarks can be hectic (on Windows especially).

Memory usage

image

Remarks:

  • Generally half the memory usage.
  • empty() seems to be occupying too much memory as if there 1M unique empty() instance.
  • Behavior and Unicast seem to occupy double the memory for some reason.

Subject/Processor synchronous streaming

image

Remarks:

  • Async, Publish and small Replay is definitely faster in 2.x
  • Larger Replay, any Unicast and any Behavior are half-to-quarter slower for some unknown reason.

Blocking for an item (or completion)

image

image

image

Remarks:

  • All seem to better at blockingly retrieve one element, except 2.x Observable for some reason.
  • Reactor is optimized on blocking empty() retrieval

Reactor 3 vs RxJava 2

  • They have the same underlying architecture style.
  • Operator internals are practically the same for most Flowable/Flux types.
  • Reactor delegates some of its Mono operators to Flux operator internals.
  • Reactor has more operators fusion-enabled, RxJava has most important operators fusion-enabled, its Observable has not been extensively optimized though.
  • Reactor uses queue-suppliers, RxJava creates specific queues in operators.
  • Reactor uses atomic field updaters, RxJava extends AtomicXXX classes but otherwise uses AtomicXXX instance fields.
  • Reactor has only an onAssembly hook, RxJava has a subscribe-time hook and indirection on top of onAssembly hook.

Subscribing

image

Remarks:

  • Reactor uses a weaker form of backpressure in just and thus slightly better.

Streaming synchronously

image

Remarks:

  • Most operators are within the +/- 5% error margin.
  • Reactor's flatMap seems to be slightly faster (5-15%)
  • Reactor's flatten seems to be reasonably faster (1.5x to 2x).
  • Reactor's iterable seems to be slightly slower (5-15%)

Async streaming

image

Remarks:

  • The async benchmark is a bit hectic, inconclusive which is better
  • Reactor likes longer pipelined streams but dislikes shorter pipelined streams.

Memory use

image

Remarks:

  • Inconclusive, Reactor has more overhead in places and less in others.
  • Reactor's processors are more compacted but leak API.
  • Reactor has some drastic Replay overhead.

Processor overhead

image

Remarks:

  • Reactor is faster in general but slightly slower in large Replay and Unicast

Blocking

image

Remarks:

  • Reactor is optimized for blocking on an empty()

image

Remarks:

  • Blocking for the first element is the same.

image

Remarks:

  • Blocking for the last element is slower in Reactor.

Conclusion

  • RxJava 2.x is generally better in terms of performance and memory utilization than 1.x; most lower performing components can be optimized further.
  • Reactor is generally 10-50% better in a Java 8 environment due to more inlined nature and even fewer allocations than RxJava 2.
@smaldini

This comment has been minimized.

Copy link
Contributor

smaldini commented Sep 19, 2016

There's room for improvement in reactor I think given some latest design improvements but this is a massively complete report, congrats @akarnokd !

@daschl

This comment has been minimized.

Copy link

daschl commented Sep 19, 2016

great info, that makes me even more wanting to move the couchbase SDK over to 2.x at some point, but still not sure how to figure out a smooth migration path :)

@ondreju93

This comment has been minimized.

Copy link

ondreju93 commented Mar 16, 2018

Hi
How to understand the first table (Subscribe to a simple source) ?
What are these compared values for RxJava 1.x and RxJava 2.x ?
Are they time values? What is the unit? In the latter tables i can see information about count of operations. What is the count of operations in the first table?

@akarnokd

This comment has been minimized.

Copy link
Owner Author

akarnokd commented Mar 16, 2018

@ondreju93

How to understand the first table (Subscribe to a simple source) ?

These are operations per second, larger is better tables. There is a baseline value chosen in each table and the other columns are measured relative to it. Green background indicate improvements, red mean worse performance.

What are these compared values for RxJava 1.x and RxJava 2.x ?

In the first table the, it shows how many times various, simple reactive types can be subscribed to in a second.

What is the count of operations in the first table?

There isn't one. They are simple sources with 0 or 1 element, depending on the type. Later tables feature longer sequences as those show how the infrastructure overhead is amortized over many items.

@ondreju93

This comment has been minimized.

Copy link

ondreju93 commented Mar 16, 2018

@akarnokd
Thanks for the explanations
Now everything is clear for me!
Congrats for creating such a great benchmarks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.