Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add BumbleBench for Perf Testing #1379

Closed
5 of 6 tasks
piyush286 opened this issue Oct 1, 2019 · 0 comments
Closed
5 of 6 tasks

Add BumbleBench for Perf Testing #1379

piyush286 opened this issue Oct 1, 2019 · 0 comments
Labels
enhancement
Projects
aqa-tests
Milestone
October 2019

Comments

@piyush286
Copy link
Contributor

piyush286 commented Oct 1, 2019 •
edited
Loading

In order to extend performance testing, we should use BumbleBench to run various microbenchmarks, focusing on different aspects such as GPU, string and math.

Repo

https://github.com/AdoptOpenJDK/bumblebench

About BumbleBench

BumbleBench is a microbenchmark tool intended to make it as easy as possible to avoid common pitfalls when microbenchmarking Java. It is intended to make sure that test runs spend most of their time running the desired piece of code compiled at the highest possible level of quality. This is surprisingly tricky in an environment with dynamic compilation employing aggressive speculative optimizations.

Benchmarks to Measure

Should use levels such as sanity, extended or special to have coverage for different components on each level.

  • Math

  • String

  • Lambda

  • GPU

  • Crypto

  • TBD

Assigned Contributor

I'll get started by adding a few tests and then continue to expand the suite.
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 1, 2019
@piyush286
Add BumbleBench
52a300f 
adoptium#1379

- Add SIMD and string tests

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Oct 1, 2019
Merged
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 11, 2019
@piyush286
Add Lambda Benchmarks From BumbleBench
1dca156 
• Added parameters for changing BumbleBench repo and branch
• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 11, 2019
@piyush286
Add Lambda Benchmarks From BumbleBench
1a3c65c 
• Added parameters for changing BumbleBench repo and branch
• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Oct 11, 2019
Merged
@karianna karianna added this to TODO in aqa-tests via automation Oct 15, 2019
@karianna karianna added this to the October 2019 milestone Oct 15, 2019
@karianna karianna moved this from TODO to Done in aqa-tests Oct 15, 2019
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 17, 2019
@piyush286
Add Lambda Benchmarks From BumbleBench
9fe8b0e 
• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 17, 2019
@piyush286
Add Lambda Benchmarks From BumbleBench
17c9e5d 
• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
smlambert pushed a commit that referenced this issue Oct 20, 2019
@piyush286
Add Lambda Benchmarks From BumbleBench (#1393)
c53a8cf 
• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: #1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
M-Davies pushed a commit to M-Davies/openjdk-tests that referenced this issue Oct 25, 2019
@piyush286 @M-Davies
Add Lambda Benchmarks From BumbleBench (adoptium#1393)
21583b9 
• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
M-Davies added a commit to M-Davies/openjdk-tests that referenced this issue Oct 25, 2019
@M-Davies
parent edbfd8b
93f583c 
author Morgan Davies <morgandavies2020@gmail.com> 1570612741 +0100
committer Morgan Davies <morgandavies2020@gmail.com> 1572013868 +0100

Removed CanAccessTest.java

Fixed in eclipse-openj9/openj9#7311

Removed DropLookupModeTest.java

Fixed in eclipse-openj9/openj9#7153

Updated problem list links

Perm exclude ModuleSetAccessibleTest.java

Removed DropLookupModeTest.java

Fixed in eclipse-openj9/openj9#6802

Removed InvokeMethodHandleWithBadArgument.java

Fixed in eclipse-openj9/openj9#6943

Updated links

Perm exclude TrySetAccessibleTest.java

squashed jdk-tests commits

skip B6206527 and B6521014 on macOS (adoptium#1390)

Remove block to re-download openjdk-test repo for debveloper branch (adoptium#1389)

Signed-off-by: Mesbah_Alam@ca.ibm.com <Mesbah_Alam@ca.ibm.com>

Make JAVA_BIN variable usable in MBCS_Tests  (adoptium#1387)

* Fix test environment check scripts and some test shells in MBCS_Tests

* Create set_variable.bat for test2.bat

Triage (adoptium#1392)

* Unexcluded RuntimeThreadInheritanceLeak

* Unexclude TestTerminatingThreadLocal

* Added issue for FilterUSRTest

Add Jenkins file to test Linux x86 32 bits platforms (adoptium#1395)

Add groovy script for Linux x86 32 bits test pipeline.

Signed-off-by: Violeta Sebe <vsebe@ca.ibm.com>

skip remaining ipv6 tests on macOS and skip TestZoneInfo310.java on all (adoptium#1398)

Add Lambda Benchmarks From BumbleBench (adoptium#1393)

• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>

Set correct test repo for aarch64 builds (adoptium#1403)

Fix git repo for jdk8 aarch64 (adoptium#1404)

* Turn off sbt colour formatting

Signed-off-by: smlambert <slambert@gmail.com>

* Update jdk8 aarch64 git repo

Signed-off-by: smlambert <slambert@gmail.com>

* Fix nested if

Signed-off-by: smlambert <slambert@gmail.com>

Reorganize JCK targets (adoptium#1401)

Signed-off-by: Mesbah_Alam@ca.ibm.com <Mesbah_Alam@ca.ibm.com>

Fix external/system-test apache-ant uri (adoptium#1405)

Add -q option back in git cmd (adoptium#1407)

The full console output is too long to load, so add back quite option in
git cmd

Signed-off-by: lanxia <lan_xia@ca.ibm.com>

Fix MBCS_Tests_CLDR_11 testcase and several minor changes (adoptium#1406)

* Fix CLDR_11 testcase and several minor changes
* Fix MBCS_Tests.unicode testcase

skip B6206527 and B6521014 on macOS (adoptium#1390)

Remove block to re-download openjdk-test repo for debveloper branch (adoptium#1389)

Signed-off-by: Mesbah_Alam@ca.ibm.com <Mesbah_Alam@ca.ibm.com>

Make JAVA_BIN variable usable in MBCS_Tests  (adoptium#1387)

* Fix test environment check scripts and some test shells in MBCS_Tests

* Create set_variable.bat for test2.bat

squashing past 15 commits on jdk-tests

Triage (adoptium#1392)

* Unexcluded RuntimeThreadInheritanceLeak

* Unexclude TestTerminatingThreadLocal

* Added issue for FilterUSRTest

Add Jenkins file to test Linux x86 32 bits platforms (adoptium#1395)

Add groovy script for Linux x86 32 bits test pipeline.

Signed-off-by: Violeta Sebe <vsebe@ca.ibm.com>

skip remaining ipv6 tests on macOS and skip TestZoneInfo310.java on all (adoptium#1398)

Add Lambda Benchmarks From BumbleBench (adoptium#1393)

• Added new benchmarks: DispatchBench, FibBench, GroupingBench & SieveBench
• Enabled new and existing tests for Java 11 along with Java 8

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>

Set correct test repo for aarch64 builds (adoptium#1403)

Fix git repo for jdk8 aarch64 (adoptium#1404)

* Turn off sbt colour formatting

Signed-off-by: smlambert <slambert@gmail.com>

* Update jdk8 aarch64 git repo

Signed-off-by: smlambert <slambert@gmail.com>

* Fix nested if

Signed-off-by: smlambert <slambert@gmail.com>

Reorganize JCK targets (adoptium#1401)

Signed-off-by: Mesbah_Alam@ca.ibm.com <Mesbah_Alam@ca.ibm.com>

Fix external/system-test apache-ant uri (adoptium#1405)

Add -q option back in git cmd (adoptium#1407)

The full console output is too long to load, so add back quite option in
git cmd

Signed-off-by: lanxia <lan_xia@ca.ibm.com>

Fix MBCS_Tests_CLDR_11 testcase and several minor changes (adoptium#1406)

* Fix CLDR_11 testcase and several minor changes
* Fix MBCS_Tests.unicode testcase

conflicts cropped up again
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 25, 2019
@piyush286
Add GPU Benchmarks From BumbleBench
4527e38 
• Added new CPU and GPULambda benchmarks: BitonicBench, DoitgenBench, KmeansBench & MatMultBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Oct 25, 2019
Merged
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Oct 25, 2019
@piyush286
Add GPU Benchmarks From BumbleBench
add0f9c 
• Added new CPU and GPULambda benchmarks: BitonicBench, DoitgenBench, KmeansBench & MatMultBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Oct 30, 2019
Closed
smlambert pushed a commit that referenced this issue Oct 30, 2019
@piyush286
Add GPU Benchmarks From BumbleBench (#1413)
73fe55c 
• Added new CPU and GPULambda benchmarks: BitonicBench, DoitgenBench, KmeansBench & MatMultBench

Issue: #1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
This was referenced Nov 1, 2019
Open
Open
Open
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Nov 1, 2019
@piyush286
Add Crypto Benchmarks
2c52949 
• Added Crypto benchmarks from BumbleBench: CipherBench, DigestBench, EllipticCurveBench, GCMBench, HMACBench, KeyExchangeBench, RSABench, SignatureBench & SSLSocketBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Nov 1, 2019
Merged
smlambert pushed a commit that referenced this issue Nov 8, 2019
@piyush286
Add Crypto Benchmarks (#1424)
a05c307 
• Added Crypto benchmarks from BumbleBench: CipherBench, DigestBench, EllipticCurveBench, GCMBench, HMACBench, KeyExchangeBench, RSABench, SignatureBench & SSLSocketBench

Issue: #1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Nov 13, 2019
@piyush286
Add Micro-Benchmarks for Collections
aa9fd00 
• Added benchmarks for ArrayList
	○ forEach: ArrayListForEachConsumerAnonymousBench, ArrayListForEachConsumerFinalBench, ArrayListForEachLambdaBench, ArrayListForEachTradBench
	○ remove: ArrayListRemoveIfLambdaBench, ArrayListRemoveIfTradBench
	○ sort: ArrayListSortCollectionsBench, ArrayListSortComparatorBench, ArrayListSortLambdaBench
• Added benchmarks for HashMap
	○ forEach: HashMapForEachBiConsumerAnonymousBench, HashMapForEachBiConsumerFinalBench, HashMapForEachLambdaBench, HashMapForEachTradBench
	○ replaceAll: HashMapReplaceAllLambdaBench, HashMapReplaceAllTradBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Nov 13, 2019
@piyush286
Add Micro-Benchmarks for Collections
18eb1bb 
• Added benchmarks for ArrayList
	○ forEach: ArrayListForEachConsumerAnonymousBench, ArrayListForEachConsumerFinalBench, ArrayListForEachLambdaBench, ArrayListForEachTradBench
	○ remove: ArrayListRemoveIfLambdaBench, ArrayListRemoveIfTradBench
	○ sort: ArrayListSortCollectionsBench, ArrayListSortComparatorBench, ArrayListSortLambdaBench
• Added benchmarks for HashMap
	○ forEach: HashMapForEachBiConsumerAnonymousBench, HashMapForEachBiConsumerFinalBench, HashMapForEachLambdaBench, HashMapForEachTradBench
	○ replaceAll: HashMapReplaceAllLambdaBench, HashMapReplaceAllTradBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Nov 13, 2019
@piyush286
Add Micro-Benchmarks for Collections
b372ca2 
• Added benchmarks for ArrayList
	○ forEach: ArrayListForEachConsumerAnonymousBench, ArrayListForEachConsumerFinalBench, ArrayListForEachLambdaBench, ArrayListForEachTradBench
	○ remove: ArrayListRemoveIfLambdaBench, ArrayListRemoveIfTradBench
	○ sort: ArrayListSortCollectionsBench, ArrayListSortComparatorBench, ArrayListSortLambdaBench
• Added benchmarks for HashMap
	○ forEach: HashMapForEachBiConsumerAnonymousBench, HashMapForEachBiConsumerFinalBench, HashMapForEachLambdaBench, HashMapForEachTradBench
	○ replaceAll: HashMapReplaceAllLambdaBench, HashMapReplaceAllTradBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Nov 13, 2019
Merged
smlambert pushed a commit that referenced this issue Nov 18, 2019
@piyush286
Add Micro-Benchmarks for Collections (#1442)
9e982af 
• Added benchmarks for ArrayList
	○ forEach: ArrayListForEachConsumerAnonymousBench, ArrayListForEachConsumerFinalBench, ArrayListForEachLambdaBench, ArrayListForEachTradBench
	○ remove: ArrayListRemoveIfLambdaBench, ArrayListRemoveIfTradBench
	○ sort: ArrayListSortCollectionsBench, ArrayListSortComparatorBench, ArrayListSortLambdaBench
• Added benchmarks for HashMap
	○ forEach: HashMapForEachBiConsumerAnonymousBench, HashMapForEachBiConsumerFinalBench, HashMapForEachLambdaBench, HashMapForEachTradBench
	○ replaceAll: HashMapReplaceAllLambdaBench, HashMapReplaceAllTradBench

Issue: #1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Jan 10, 2020
@piyush286
Remove Java Version Constraint for BumbleBench
d64bf3e 
•  Enabled BumbleBench to run for all Java versions:
	○ Earlier, BB could not build for Java 12 and up due to adoptium/bumblebench#17

Related Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Jan 11, 2020
@piyush286
Remove Java Version Constraint for BumbleBench
ff6902f 
•  Enabled BumbleBench to run for all Java versions:
	○ Earlier, BB could not build for Java 12 and up due to adoptium/bumblebench#17

Related Issues:
adoptium#1379
adoptium#1541

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Jan 11, 2020
Merged
smlambert pushed a commit that referenced this issue Jan 12, 2020
@piyush286
Remove Java Version Constraint for BumbleBench (#1546)
ac6521b 
•  Enabled BumbleBench to run for all Java versions:
	○ Earlier, BB could not build for Java 12 and up due to adoptium/bumblebench#17

Related Issues:
#1379
#1541

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
llxia pushed a commit to llxia/openjdk-tests that referenced this issue Jan 22, 2020
@piyush286 @llxia
Remove Java Version Constraint for BumbleBench (adoptium#1546)
84432b8 
•  Enabled BumbleBench to run for all Java versions:
	○ Earlier, BB could not build for Java 12 and up due to adoptium/bumblebench#17

Related Issues:
adoptium#1379
adoptium#1541

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Apr 13, 2020
Open
piyush286 added a commit to piyush286/openjdk-tests that referenced this issue Apr 13, 2020
@piyush286
Add Humble Benchmarks
e6bae29 
• Added Crypto benchmarks from BumbleBench: DistinctStringsEqualsBench, NewStringBufferWithCapacityBench, NewStringBuilderWithCapacityBench & SameStringsEqualsBench

Issue: adoptium#1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Apr 13, 2020
Merged
smlambert pushed a commit that referenced this issue Apr 23, 2020
@piyush286
Add Humble Benchmarks (#1727)
d215ee0 
• Added Crypto benchmarks from BumbleBench: DistinctStringsEqualsBench, NewStringBufferWithCapacityBench, NewStringBuilderWithCapacityBench & SameStringsEqualsBench

Issue: #1379

Signed-off-by: Piyush Gupta <piyush286@gmail.com>
@piyush286 piyush286 mentioned this issue Oct 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
No open projects
aqa-tests
  
Done
Milestone
October 2019
Development

No branches or pull requests
2 participants
@karianna @piyush286