Lincheck benchmarks #250

eupp · 2023-11-20T18:57:26Z

This PR adds benchmarking infrastructure to the Lincheck. It includes:

API to track progress of lincheck test run: LincheckRunTracker interface.
Statistics gathering infrastructure based on LincheckRunTracker API.
New gradle tasks to run benchmarks (tests and benchmarks are put into separate source sets).
Several benchmarks for standard concurrent data structures and synchronization primitives.
Script to draw various plots based on collected benchmark statistics (using Lets-Plot Kotlin library).

ndkoval · 2024-01-19T18:29:13Z

...m/test/org/jetbrains/kotlinx/lincheck_test/verifier/linearizability/RendezvousChannelTest.kt

Why do you need to rename it? It causes a lot of unrelated changes.

I can revert this.

ndkoval · 2024-01-19T18:30:06Z

src/jvm/test/org/jetbrains/kotlinx/lincheck_test/representation/ThreadDumpTest.kt

@@ -25,8 +25,8 @@ class ThreadDumpTest {
        repeat(iterations) {
            val options = StressOptions()
                .minimizeFailedScenario(false)
-                .iterations(100_000)
-                .invocationsPerIteration(1)
+                .iterations(10)


Why do you need this change?

For some reason, without this the CI do not pass, but fails with OOM.
With this change, the memory usage (according to TC) is roughly the same as on develop branch.

benchmarks branch perf report

develop branch perf report

ndkoval · 2024-01-19T18:31:23Z

src/jvm/main/org/jetbrains/kotlinx/lincheck/CTestConfiguration.kt

@@ -38,8 +38,10 @@ abstract class CTestConfiguration(
    val customScenarios: List<ExecutionScenario>
 ) {
    abstract fun createStrategy(
-        testClass: Class<*>, scenario: ExecutionScenario, validationFunction: Actor?,
-        stateRepresentationMethod: Method?, verifier: Verifier


What is the purpose of not passing a verifier instance in the constructor?

The code for checking invocation result was duplicated in stress and managed strategies.
Once I refactored to be the extension function of Strategy class, there is no longer a need to pass the verifier in the constructor, since it effectively becomes unused by the concrete Strategy implementers.

ndkoval · 2024-01-19T18:31:46Z

...jvm/benchmark/org/jetbrains/kotlinx/lincheck_benchmark/running/RendezvousChannelBenchmark.kt

+
+@ExperimentalCoroutinesApi
+@InternalCoroutinesApi
+@Param(name = "value", gen = IntGen::class, conf = "1:5")


Do you need this?

What line exactly do you mean?
The @Param annotation?

I think @Param could be removed, yes.

ndkoval · 2024-01-19T18:32:32Z

src/jvm/benchmark/org/jetbrains/kotlinx/lincheck_benchmark/running/LocksBenchmarks.kt

+    fun get(): Int = lock.withLock { counter }
+}
+
+class IntrinsicLockBenchmark : AbstractLincheckBenchmark() {


SynchronizedBenchmark?

Ok, will rename

ndkoval · 2024-01-19T18:34:36Z

...jvm/benchmark/org/jetbrains/kotlinx/lincheck_benchmark/running/ConcurrentHashMapBenchmark.kt

+import org.jetbrains.kotlinx.lincheck_benchmark.AbstractLincheckBenchmark
+import java.util.concurrent.ConcurrentHashMap
+
+@Param(name = "key", gen = IntGen::class, conf = "1:5")


Do you need it?

ndkoval · 2024-01-19T18:34:44Z

src/jvm/benchmark/org/jetbrains/kotlinx/lincheck_benchmark/running/ConcurrentDequeBenchmark.kt

+import org.jetbrains.kotlinx.lincheck_benchmark.AbstractLincheckBenchmark
+import java.util.concurrent.*
+
+@Param(name = "value", gen = IntGen::class, conf = "1:5")


Do you need it?

ndkoval · 2024-01-19T18:34:47Z

src/jvm/benchmark/org/jetbrains/kotlinx/lincheck_benchmark/running/BufferedChannelBenchmark.kt

+import org.jetbrains.kotlinx.lincheck_benchmark.AbstractLincheckBenchmark
+
+@InternalCoroutinesApi
+@Param(name = "value", gen = IntGen::class, conf = "1:5")


Do you need it?

ndkoval · 2024-01-19T18:36:10Z

src/jvm/benchmark/org/jetbrains/kotlinx/lincheck_benchmark/running/LocksBenchmarks.kt

+    }
+}
+
+class ReentrantLockBenchmark : AbstractLincheckBenchmark() {


Do we need this benchmark? It does not look realistic.

I took it from lincheck tests.
There were tests for several kinds of locks: spin locks, intrinsic locks (synchronized), and re-entrant lock objects.

I just changed the lock-protected data structure from set to counter. Because I think the exact data structure used under lock is not relevant for these benchmarks, so I picked the simplest one possible. If you have arguments why we should use sets as an underlying data structure, I will be ready to revert it to sets.

ndkoval · 2024-01-19T18:40:47Z

src/jvm/main/org/jetbrains/kotlinx/lincheck/strategy/stress/StressStrategy.kt

@@ -44,19 +42,11 @@ class StressStrategy(
        }
    }

-    override fun run(): LincheckFailure? {


Please make this refactoring (and the one in ManagedStrategy) in a separate PR

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

…shboards Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

* rewrite .txt report file on each run * compute average invocation time in case of per-iteration granularity * do not draw error-bars if standard-error is not computed Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

- do not reuse verifier for minimization Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

…r tests Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

eupp requested a review from ndkoval November 20, 2023 18:57

ndkoval linked an issue Jan 18, 2024 that may be closed by this pull request

Create real-world benchmarks #252

Open

ndkoval requested changes Jan 19, 2024

View reviewed changes

eupp mentioned this pull request Jan 22, 2024

Strategy run API refactoring #262

Open

eupp added 22 commits May 30, 2024 13:46

refactor the strategy API to expose the single invocation run method

9156cd3

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

fix minor review comments

e4a35d6

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

fixes after rebase

2a7dd77

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

run tracker API

198ab50

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

statistics API

296ebfe

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

add benchmark infra and several basic data structure benchmarks

46b1df9

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

slightly improved trackers API

c223d2d

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

add the ability to pass parameters to the benchmark suite Gradle task

40e33ef

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

minor renaming

0a3d68b

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

add simple .txt report format for testing integration with ij-perf da…

d83e5e7

…shboards Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

add lock benchmarks

14be4f3

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

add channel benchmarks

07c8145

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

basic CLI for producing benchmark plots

f3f1ead

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

use Clikt library for CLI

9590d11

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

various fixes in reports generation

5e37a91

* rewrite .txt report file on each run * compute average invocation time in case of per-iteration granularity * do not draw error-bars if standard-error is not computed Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

moved benchmarks into separate folder

10490b0

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

add README.md for benchmarks

3f9f877

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

minor

88321f5

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

minor

6d51336

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

try to fix memory consumption bug

f565e42

- do not reuse verifier for minimization Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

debugging OOM on build server: try to disable statistics gathering fo…

b51bcd3

…r tests Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

rollback statistics gathering in tests

23ea160

Signed-off-by: Evgenii Moiseenko <evg.moiseenko94@gmail.com>

eupp force-pushed the lincheck-benchmarks branch from f6d693d to 23ea160 Compare June 10, 2024 10:37

eupp added 3 commits June 10, 2024 13:05

fixes after rebase

0b96846

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

fixes after rebase

27870e2

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

fix lincheck/jvmBenchmark resolve in IDEA

c0e8ce9

Signed-off-by: Evgeniy Moiseenko <evg.moiseenko94@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lincheck benchmarks #250

Lincheck benchmarks #250

eupp commented Nov 20, 2023

ndkoval Jan 19, 2024

eupp Jan 22, 2024

ndkoval Jan 19, 2024

eupp Jan 22, 2024

ndkoval Jan 19, 2024

eupp Jan 22, 2024

ndkoval Jan 19, 2024

eupp Jan 22, 2024

eupp Jan 22, 2024

ndkoval Jan 19, 2024

eupp Jan 22, 2024

ndkoval Jan 19, 2024

ndkoval Jan 19, 2024

ndkoval Jan 19, 2024

ndkoval Jan 19, 2024

eupp Jan 22, 2024

ndkoval Jan 19, 2024

eupp Jan 22, 2024

eupp Jan 22, 2024

Lincheck benchmarks #250

Are you sure you want to change the base?

Lincheck benchmarks #250

Conversation

eupp commented Nov 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment