including benchmarking code for QFJ within another module in the code base

[busy-spin]

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Provide additional maven module to include all performance regression.
Any code changes would additionally have to pass the performance regression.

Example , having this benchmark code from @chrjohn, within code base is useful.
#559

Describe alternatives you've considered
N/A

Additional context
N/A

[chrjohn]

Sounds like a sensible idea. Are you aware of anything that could also trigger a github action in case there is a regression? Or will the build just fail?

[jvirtanen]

Sounds like a sensible idea. Are you aware of anything that could also trigger a github action in case there is a regression? Or will the build just fail?

Running microbenchmarks reliably in GitHub Actions is probably hard, as they are very sensitive to their runtime environment. However, just having the microbenchmarks in an executable Maven module is itself a good goal, even if it is not part of the CI.

For Philadelphia, I run the microbenchmarks manually towards the end of the release cycle. JMH provides machine-readable output formats, such as JSON, if you pass it the -rf and -rff options. In Philadelphia, I use this to generate a human-readable performance regression test report (sample). (Incidentally, the sample shows how susceptible these things are; these are from an otherwise idle laptop, which itself should be quite a bit more stable than the standard GitHub Actions runners. Earlier I used to spin up a c4.xlarge Amazon EC2 instance just for this purpose.)

[busy-spin]

Sounds like a sensible idea. Are you aware of anything that could also trigger a github action in case there is a regression? Or will the build just fail?

Running microbenchmarks reliably in GitHub Actions is probably hard, as they are very sensitive to their runtime environment. However, just having the microbenchmarks in an executable Maven module is itself a good goal, even if it is not part of the CI.

For Philadelphia, I run the microbenchmarks manually towards the end of the release cycle. JMH provides machine-readable output formats, such as JSON, if you pass it the -rf and -rff options. In Philadelphia, I use this to generate a human-readable performance regression test report (sample). (Incidentally, the sample shows how susceptible these things are; these are from an otherwise idle laptop, which itself should be quite a bit more stable than the standard GitHub Actions runners. Earlier I used to spin up a c4.xlarge Amazon EC2 instance just for this purpose.)

@chrjohn now I think about it it might be difficult to come up performance regression as we need to come up with sensible assertion for the benchmark. But I totally agree with @jvirtanen , if we have executable maven module based on JMH we can provide way for contributors to manually benchmark them.

Can we take this as a feature, I'd like to make a PR on this.

[chrjohn]

The assertion probably would be that after each commit the performance should not decrease.
But as @jvirtanen said (thank you for your thoughts on this) it could be problematic.
Please go ahead and create a PR. Many thanks in advance. 👍

[busy-spin]

hi @chrjohn / @jvirtanen

I made PR for quickfixj-per-test , kindly provide your feedback

#582