Added benchmarks for common java idioms #1456

mping · 2016-07-21T18:33:43Z

Hopefully will have the following benchmarks:

For comprehension
Tuple vs Pair class?
Try
Pattern Matching

mping · 2016-07-21T18:34:17Z

@danieldietrich this is just preliminary to see if the general outline of the test looks good.
I've actually never used javaslang in my projects so let me know if it doesn't look good.

paplorinc · 2016-07-23T19:50:20Z

javaslang-benchmark/src/test/java/javaslang/idiom/ForBenchmark.java

+
+        @Benchmark
+        public Object javaslang_for() {
+            Integer result = For(ELEMENTS_ARRAY).yield(i -> i).reduce((i, j) -> i + j);


Thanks for providing alternative benchmarks :).
It would be nice if the build would be passing before asking for code reviews.
Also, please use the same style as the other benchmarks do (e.g. final local variables, method naming, encapsulation in separate classes with a base class)

will update, didnt notice it was triggering CI.

mping · 2016-07-27T09:28:03Z

Tests are failing because of OOM, since For comprehension essentially does a cross product I'll use smaller arrays.

paplorinc · 2016-07-27T09:38:50Z

javaslang-benchmark/src/test/java/javaslang/idiom/ForBenchmark.java

+            for (Integer i : ELEMENTS_ARRAY) {
+                for (Integer j : ELEMENTS_ARRAY) {
+                    for (Integer k : ELEMENTS_ARRAY) {
+                        result.add(i * j * k * 0);


Just needed something to ensure the assert is simple.

mping · 2016-07-27T19:25:24Z

javaslang-benchmark/src/test/java/javaslang/idiom/ForBenchmark.java

+            ELEMENTS_ARRAY = Arrays.asList(ELEMENTS);
+        }
+
+        protected Object consume(Blackhole bh, Integer i) {


Blackhole is from JMH and is designed for the case where you don't need/want to check the result but you do want to prevent DCE, so it's basically something to emulate a side-effect function.

mping · 2016-07-27T19:25:57Z

Tests should be simpler now, here's what I have (cannot explain the 2nd result)

Ratios slang / <alternative_impl>
Target        Operation                   Ratio                              10 
ForBenchmark  ForComprehension1Iterator   javaslang_for/java_for          0,88x
ForBenchmark  ForComprehension2Iterators  javaslang_for/java_for         11,89x

paplorinc · 2016-07-27T19:39:31Z

Will check it out next week (I'm very busy, sorry), until then, please simplify everything as much as possible.
If you're testing the speed of for loops, don't include everything else in it.
Just iterate over the array and either consume it via the blackhole, or rather collect it via a cheap operation like xor so that we can have some assertions that fail if we mess it up (like they failed until now when you made an error, that's exactly why we need them :)

codecov-io · 2016-07-27T19:54:22Z

Current coverage is 96.84% (diff: 100%)

Merging #1456 into master will not change coverage

@@             master      #1456   diff @@
==========================================
  Files            88         88          
  Lines         10675      10675          
  Methods           0          0          
  Messages          0          0          
  Branches       1818       1818          
==========================================
  Hits          10338      10338          
  Misses          194        194          
  Partials        143        143

Powered by Codecov. Last update 403226f...cd1b8e0

mping · 2016-07-27T20:09:39Z

Will use a xor, thanks.

paplorinc · 2016-07-29T12:07:50Z

javaslang-benchmark/src/test/java/javaslang/idiom/ForBenchmark.java

+        public int CONTAINER_SIZE;
+
+        Integer[] ELEMENTS;
+        List<Integer> ELEMENTS_ARRAY;


this doesn't seem like an array :)
Do you need both?

You're right, I just need a list.

mping · 2016-07-31T23:24:05Z

Geez, I'm getting horrible results with the XOR trick. @paplorinc can you share you idea with some code? I'm currently doing this:

Optional<Integer> res = For(ELEMENTS, ELEMENTS).yield((i, j) -> i ^ j).collect(XOR);

But the results are horrible in comparison to the nested for. I'm guessing I am doing something wrong, feel free to point me in the right direction.

paplorinc · 2016-08-01T12:01:54Z

I would be surprised if the simple for loop (which is often optimized by JIT anyway) wouldn't be at least 10x faster ... maybe even 100x.
However, you should get the same results with or without the xor (actually any constant size operation would only bring the values closer together).
Please try to debug it manually, I would do the same :).

mping · 2016-08-01T13:45:01Z

Cool, I will proceed then, and add more benchmarks, for the rest of the idioms.
The For().yield implementation is quite simple via flatmap but I was wondering if it wouldn't be better to just nest the loops to squeeze more performance.

paplorinc · 2016-08-01T13:46:14Z

Try it out and report your findings :)

mping · 2016-08-02T21:33:04Z

Added a couple more benchmarks, now only pattern matching is missing.

Ratios <alternative_impl> / slang
Target        Operation                                                                          Ratio           

-- try -- 
TryBenchmark  TryThrowsCatchBenchmark         java_try_actual_exception/javaslang_try_actual_exception     2,20x
TryBenchmark  TryNoThrowsCatchBenchmark               java_try_no_exception/javaslang_try_no_exception     0,87x

-- tuple --
TupleBenchmark  Tuple2Benchmark       java_tuple2_creation/javaslang_tuple2_creation     1,00x


-- for comprehension  --
ForBenchmark  ForComprehensionNestedFor       java_for/javaslang_for    11,47x

danieldietrich · 2016-08-02T22:09:29Z

README.md


 ```
-mvn clean test -Pbenchmark
+mvn clean test -pl javaslang-benchmark -Dtest=TryBenchmark


Do we already run a specific microbenchmark here? See line 37 and line 40.

Oops, too much copy/paste. Will fix.

mping · 2016-08-24T13:16:49Z

No I don't. Will check it out today. Was an assert that failed :(
@danieldietrich is there a report for the benchmarks generated anywhere? would love to write a blog post on this.

danieldietrich · 2016-08-24T13:58:19Z

A blog post would be awesome! I would appreciate to review it before publishing because interpreting benchmarks is a difficult topic. @paplorinc and others may also be interested to take a look!

Actually I don't know how to run the benchmarks :-/ The recent command mvn test -Pbenchmark -pl javaslang does not work any more. I asked @paplorinc for the details and will keep you up-to-date. We will document it, see #1527

paplorinc · 2016-08-24T20:26:29Z

Sure, a blog post would be nice, but a lot of things are very hard to interpret here currently, I really urge you to understand JMH well first, it's a lot more difficult than it seems - and as I tried to point out in these benchmarks, they are still far from being finished.

mping · 2016-08-24T22:51:58Z

I appreciate everyone's advice and input, I realize it's very tricky to interpret the results but my goal was never to provide an accurate measurement. I just want to get an idea of what to expect in terms of performance. I know using JMH is hard, but I think this is a good and practical way to learn

danieldietrich · 2016-08-25T06:12:13Z

@mping Yes, we need that blog post. I would greatly appreciate if you would write a guest post on the Javaslang blog, including your Bio (/Photo etc) :-) The format is would be markdown.

paplorinc · 2016-08-25T21:02:46Z

javaslang-benchmark/src/test/java/javaslang/idiom/ForBenchmark.java

+
+    @State(Scope.Benchmark)
+    public static class Base {
+        @Param({ "10" })


Please provide more params so that we can observe a trend

- For comprehension - Tuple - Try - Pattern Matching

danieldietrich · 2016-10-23T19:33:18Z

I will pull this PR in now - we need to burn down our issues and PRs in order to get forward. Optimizations of the Benchmarks can occur later.

@mping Many thanks!!

mping · 2016-10-24T10:33:07Z

@danieldietrich yay thanks!

AnthonyKot · 2017-08-16T19:09:37Z

JFYI, looks like the error was mentioned by danieldietrich on 24 Aug 2016 is still here:
"Test set: io.vavr.idiom.ForBenchmark

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.475 sec <<< FAILURE! - in io.vavr.idiom.ForBenchmark
testAsserts(io.vavr.idiom.ForBenchmark) Time elapsed: 0.475 sec <<< ERROR!
java.lang.RuntimeException: org.openjdk.jmh.runner.RunnerException: Benchmark caught the exception
at io.vavr.idiom.ForBenchmark.testAsserts(ForBenchmark.java:38)
Caused by: org.openjdk.jmh.runner.RunnerException: Benchmark caught the exception
at io.vavr.idiom.ForBenchmark.testAsserts(ForBenchmark.java:38)
Caused by: org.openjdk.jmh.runner.BenchmarkException: Benchmark error during the run
at io.vavr.idiom.ForBenchmark.testAsserts(ForBenchmark.java:38)"

danieldietrich · 2017-08-17T13:58:52Z

@AnthonyKot thank you, I will create an issue

mping mentioned this pull request Jul 21, 2016

suggestion: add benchmarks for common java idioms against javaslang #1441

Closed

paplorinc reviewed Jul 23, 2016
View reviewed changes

mping force-pushed the idioms-benchmark branch from 9a43706 to 2f17b04 Compare July 25, 2016 22:32

paplorinc reviewed Jul 27, 2016
View reviewed changes

mping force-pushed the idioms-benchmark branch from 2f17b04 to 2acaf96 Compare July 27, 2016 19:23

mping reviewed Jul 27, 2016
View reviewed changes

paplorinc reviewed Jul 29, 2016
View reviewed changes

mping force-pushed the idioms-benchmark branch from 2acaf96 to 3ad6d8c Compare August 2, 2016 21:32

danieldietrich reviewed Aug 2, 2016
View reviewed changes

mping force-pushed the idioms-benchmark branch from d0ac73b to 8745c74 Compare August 24, 2016 22:45

paplorinc reviewed Aug 25, 2016
View reviewed changes

danieldietrich modified the milestones: 2.2.0, 2.1.0 Aug 26, 2016

Added benchmarks for common java idioms

cd1b8e0

- For comprehension - Tuple - Try - Pattern Matching

mping force-pushed the idioms-benchmark branch from 8745c74 to cd1b8e0 Compare September 25, 2016 18:10

danieldietrich modified the milestones: 2.1.0, 2.2.0 Oct 23, 2016

danieldietrich merged commit 120f85f into vavr-io:master Oct 23, 2016

danieldietrich added «non-public» «cross cutting» and removed «non-public» labels Nov 13, 2016

danieldietrich mentioned this pull request Aug 17, 2017

For-idiom benchmark fails (sporadic?) #2055

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added benchmarks for common java idioms #1456

Added benchmarks for common java idioms #1456

mping commented Jul 21, 2016

mping commented Jul 21, 2016 •

edited

paplorinc Jul 23, 2016

mping Jul 25, 2016

mping commented Jul 27, 2016

paplorinc Jul 27, 2016

mping Jul 27, 2016

mping Jul 27, 2016

mping commented Jul 27, 2016 •

edited

paplorinc commented Jul 27, 2016

codecov-io commented Jul 27, 2016 •

edited

mping commented Jul 27, 2016

paplorinc Jul 29, 2016

mping Jul 29, 2016

mping commented Jul 31, 2016

paplorinc commented Aug 1, 2016

mping commented Aug 1, 2016

paplorinc commented Aug 1, 2016

mping commented Aug 2, 2016

danieldietrich Aug 2, 2016

mping Aug 3, 2016

mping commented Aug 24, 2016

danieldietrich commented Aug 24, 2016 •

edited

paplorinc commented Aug 24, 2016 •

edited

mping commented Aug 24, 2016

danieldietrich commented Aug 25, 2016

paplorinc Aug 25, 2016

danieldietrich commented Oct 23, 2016

mping commented Oct 24, 2016

AnthonyKot commented Aug 16, 2017 •

edited

danieldietrich commented Aug 17, 2017

Added benchmarks for common java idioms #1456

Added benchmarks for common java idioms #1456

Conversation

mping commented Jul 21, 2016

mping commented Jul 21, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mping commented Jul 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mping commented Jul 27, 2016 • edited

paplorinc commented Jul 27, 2016

codecov-io commented Jul 27, 2016 • edited

Current coverage is 96.84% (diff: 100%)

mping commented Jul 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mping commented Jul 31, 2016

paplorinc commented Aug 1, 2016

mping commented Aug 1, 2016

paplorinc commented Aug 1, 2016

mping commented Aug 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mping commented Aug 24, 2016

danieldietrich commented Aug 24, 2016 • edited

paplorinc commented Aug 24, 2016 • edited

mping commented Aug 24, 2016

danieldietrich commented Aug 25, 2016

Choose a reason for hiding this comment

danieldietrich commented Oct 23, 2016

mping commented Oct 24, 2016

AnthonyKot commented Aug 16, 2017 • edited

danieldietrich commented Aug 17, 2017

mping commented Jul 21, 2016 •

edited

mping commented Jul 27, 2016 •

edited

codecov-io commented Jul 27, 2016 •

edited

danieldietrich commented Aug 24, 2016 •

edited

paplorinc commented Aug 24, 2016 •

edited

AnthonyKot commented Aug 16, 2017 •

edited