Module rearange and add kotlin benchmark #950

jinia91 · 2024-03-16T05:30:23Z

Summary

I suggest some modifications to the current structure to facilitate benchmark testing of the fixture-monkey-kotlinmodule using JMH

To ensure consistent results, use the same class and use the java-test-fixtures plugin for code reuse.
make new module fixture-monkey-benchmarks for aggregating sub-benchmark-module.
Due to some local compile issues when defining Java classes within the Kotlin module, move the ExpressionGeneratorJavaTestSpecs.java to a Java package in fixture-monkey-kotlin.

Add default benchmark samples to the fixture-monkey-kotlin module.

 For a proper comparison, the sample test cases are planned as follows:
 
 1. Benchmark for initializing Java classes in Kotlin
 2. Benchmark for initializing Kotlin classes in Kotlin
 3. Benchmark for initializing composite classes with Kotlin-Java internal references (todo)

Modify the GitHub Actions workflow for benchmark reporting.

How Has This Been Tested?

Local Build and Ci Pipline

Is the Document updated?

no need

…nchmark sample

seongahjo · 2024-03-16T06:58:28Z

How about adding a new module to benchmark?

jinia91 · 2024-03-16T07:05:06Z

How about adding a new module to benchmark?

Im considering two approaches

first, make new module for benchmark integration testing
second, performing benchmark testing for each module separately and collect just reporting.

From the perspective of ~~scalability~~ Flexibility , choice second approach.

However, I'm not entirely confident due to the downside of increased dependency complexity.

jinia91 · 2024-03-16T07:15:41Z

fixture-monkey Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit
ManipulationBenchmark.fixed	avgt	1	10	576.165762	2.457355	ms/op
ManipulationBenchmark.thenApply	avgt	1	10	1301.278260	5.921343	ms/op
ObjectGenerationBenchmark.beanGenerateOrderSheetWithFixtureMonkey	avgt	1	10	596.718850	5.575784	ms/op
ObjectGenerationBenchmark.builderGenerateOrderSheetWithFixtureMonkey	avgt	1	10	505.365829	3.092013	ms/op
ObjectGenerationBenchmark.fieldReflectionGenerateOrderSheetWithFixtureMonkey	avgt	1	10	572.670919	6.588839	ms/op
ObjectGenerationBenchmark.jacksonGenerateOrderSheetWithFixtureMonkey	avgt	1	10	607.734411	4.826412	ms/op

fixture-monkey-kotlin Module Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit
KotlinObjectGenerationBenchMark.beanGenerateJavaOrderSheetWithFixtureMonkey	avgt	1	10	1029.619206	21.428195	ms/op
KotlinObjectGenerationBenchMark.beanGenerateKotlinOrderSheetWithFixtureMonkey	avgt	1	10	980.861308	14.237564	ms/op

seongahjo · 2024-03-16T07:19:35Z

I'm sure you've given it a lot of thought.
Can you tell me about a specific situation where flexibility is critical?
I have no idea so far. I'm not sure about it is worth than complex dependency.

jinia91 · 2024-03-16T07:29:56Z

Currently, the execution speed of performance bench check feels too slow, leading to a decrease in productivity.

With the current PR, as adding just two of kotlin benchmarking, there has been an increase of over 4 minutes.

As fixture-monkey continues to expand with more tests being added, i expect to take more times in future.

so, in near future, try to various pipelines such as:

Running only some benchmarks based on module changes
Executing each benchmark in parallel
and so on...etc..

are expected to be considered.

In such cases, big one module for benchmark might lead to break changes.

Of course, this is just speculative in my opinion, and the problems of increased dependency complexity and management points cannot be ignored.

seongahjo · 2024-03-16T07:35:10Z

Oh, I see, thank you for your detailed explanation.
Yes, I agree that a large module leads to bad for performance, which is bad for productivity.
Then how about a new module that contains sub benchmark modules like fixture-monkey-tests?

jinia91 · 2024-03-16T07:39:57Z

Oh, that sounds good.

There seems to be no need to place the benchmark testing module under the basic module structure unnecessarily.
I will change the direction to manage under the fixture-monkey-benchmark each sub benchmark modules.

Thank you for the great idea.

seongahjo · 2024-03-18T01:47:11Z

...rks/fixture-monkey-benchmark/src/jmh/java/com/navercorp/fixturemonkey/BuilderOrderSheet.java

@@ -174,4 +174,3 @@ public enum BundleType {
 		IDENTICAL_PRODUCT
 	}
 }
-


EOL is missing now.

seongahjo · 2024-03-18T01:47:14Z

...s/fixture-monkey-benchmark/src/testFixtures/java/com/navercorp/fixturemonkey/OrderSheet.java

@@ -180,4 +180,3 @@ public enum BundleType {
 		IDENTICAL_PRODUCT
 	}
 }
-


EOL is missing now.

seongahjo · 2024-03-18T01:48:31Z

fixture-monkey-kotlin/build.gradle.kts

@@ -10,6 +10,7 @@ dependencies {
    implementation("org.jetbrains.kotlin:kotlin-reflect:${Versions.KOTLIN}")

    testImplementation(project(":fixture-monkey-jackson"))
+    testImplementation(project(":fixture-monkey-javax-validation"))


Is this dependency necessary?

In the ExpressionGeneratorJavaTestSpecs, javax validation was previously being used.

Even on the main branch, compile was not successful in my local environment, the exact reason is unclear.

After changing modules, ci compile fails in the absence of the dependency, so it necessary in the current situation.

seongahjo · 2024-03-18T01:49:07Z

.../src/test/java/com/navercorp/fixturemonkey/kotlin/spec/ExpressionGeneratorJavaTestSpecs.java

@@ -16,7 +16,7 @@
 * limitations under the License.
 */

-package com.navercorp.fixturemonkey.kotlin;
+package com.navercorp.fixturemonkey.kotlin.spec;


I think this change has nothing to do with this PR.

seongahjo · 2024-03-18T01:49:11Z

...monkey-kotlin/src/test/kotlin/com/navercorp/fixturemonkey/kotlin/ExpressionGeneratorsTest.kt

+import com.navercorp.fixturemonkey.kotlin.spec.ExpressionGeneratorJavaTestSpecs.DogJava
+import com.navercorp.fixturemonkey.kotlin.spec.ExpressionGeneratorJavaTestSpecs.PersonJava


I think this change has nothing to do with this PR.

seongahjo

Only minor convention changes are required.
Nice work 👍

jinia91 added 2 commits March 16, 2024 14:21

change some module dependency for kotlin benchmark testing and add be…

669356a

…nchmark sample

change benchmark github workflow

b8c0b9d

jinia91 closed this Mar 16, 2024

jinia91 reopened this Mar 16, 2024

jinia91 added 4 commits March 16, 2024 14:53

change benchmark github workflow

6f49f26

change benchmark github workflow

6766a99

change benchmark github workflow and sample benchmark test

fe9b2e8

enhance sample kotlin test

94bfe7c

jinia91 changed the title ~~Module reArange for kotlin benchmark~~ Module rearange and add kotlin benchmark Mar 16, 2024

formatting

453ed67

jinia91 marked this pull request as ready for review March 16, 2024 07:19

github-actions bot requested review from acktsap, benelog, jwChung, mhyeon-lee, seongahjo and SooKim1110 March 16, 2024 07:19

github-actions bot assigned jinia91 Mar 16, 2024

jinia91 marked this pull request as draft March 16, 2024 08:45

jinia91 added 4 commits March 16, 2024 17:47

change module strategy to benchmarks and sub benchmark

f67d20f

doc roll back

a6f8e20

adjust jmh module setting

902a33f

temp clean cache

1b0b32b

jinia91 added 2 commits March 16, 2024 18:03

ci retry

4a27832

add task jar for ci

b9dc3dc

jinia91 marked this pull request as ready for review March 16, 2024 09:45

jinia91 mentioned this pull request Mar 16, 2024

Add KotlinAndJavaCompositeArbitraryIntrospector for supporting instantiating kotlin class with java reference Type #948

Merged

seongahjo reviewed Mar 18, 2024

View reviewed changes

seongahjo approved these changes Mar 18, 2024

View reviewed changes

jinia91 added 3 commits March 18, 2024 13:11

line feed?

31f72a4

del and test unnecessary dependency

5fb5630

rollback dependency

421759a

seongahjo merged commit f670cf8 into naver:main Mar 18, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Module rearange and add kotlin benchmark #950

Module rearange and add kotlin benchmark #950

jinia91 commented Mar 16, 2024 •

edited

seongahjo commented Mar 16, 2024

jinia91 commented Mar 16, 2024 •

edited

jinia91 commented Mar 16, 2024 •

edited

seongahjo commented Mar 16, 2024

jinia91 commented Mar 16, 2024

seongahjo commented Mar 16, 2024 •

edited

jinia91 commented Mar 16, 2024 •

edited

seongahjo Mar 18, 2024

seongahjo Mar 18, 2024

seongahjo Mar 18, 2024

jinia91 Mar 18, 2024 •

edited

seongahjo Mar 18, 2024

seongahjo Mar 18, 2024

seongahjo left a comment

		import com.navercorp.fixturemonkey.kotlin.spec.ExpressionGeneratorJavaTestSpecs.DogJava
		import com.navercorp.fixturemonkey.kotlin.spec.ExpressionGeneratorJavaTestSpecs.PersonJava

Module rearange and add kotlin benchmark #950

Module rearange and add kotlin benchmark #950

Conversation

jinia91 commented Mar 16, 2024 • edited

Summary

How Has This Been Tested?

Is the Document updated?

seongahjo commented Mar 16, 2024

jinia91 commented Mar 16, 2024 • edited

jinia91 commented Mar 16, 2024 • edited

seongahjo commented Mar 16, 2024

jinia91 commented Mar 16, 2024

seongahjo commented Mar 16, 2024 • edited

jinia91 commented Mar 16, 2024 • edited

seongahjo Mar 18, 2024

Choose a reason for hiding this comment

seongahjo Mar 18, 2024

Choose a reason for hiding this comment

seongahjo Mar 18, 2024

Choose a reason for hiding this comment

jinia91 Mar 18, 2024 • edited

Choose a reason for hiding this comment

seongahjo Mar 18, 2024

Choose a reason for hiding this comment

seongahjo Mar 18, 2024

Choose a reason for hiding this comment

seongahjo left a comment

Choose a reason for hiding this comment

jinia91 commented Mar 16, 2024 •

edited

jinia91 commented Mar 16, 2024 •

edited

jinia91 commented Mar 16, 2024 •

edited

seongahjo commented Mar 16, 2024 •

edited

jinia91 commented Mar 16, 2024 •

edited

jinia91 Mar 18, 2024 •

edited