Skip to content

Commit

Permalink
Update the benchmarks.md docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Akirathan committed Feb 7, 2024
1 parent dc4676b commit 0997cf2
Showing 1 changed file with 30 additions and 18 deletions.
48 changes: 30 additions & 18 deletions docs/infrastructure/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,26 +13,38 @@ To track the performance of the engine, we use
benchmarks:

- [micro benchmarks](#engine-jmh-microbenchmarks) located directly in the
`runtime` SBT project. These benchmarks are written in Java, and are used to
measure the performance of specific parts of the engine.
`runtime-benchmarks` SBT project. These benchmarks are written in Java, and
are used to measure the performance of specific parts of the engine.
- [standard library benchmarks](#standard-library-benchmarks) located in the
`test/Benchmarks` Enso project. These benchmarks are entirelly written in
Enso, along with the harness code.
`test/Benchmarks` Enso project. These benchmarks are entirely written in Enso,
along with the harness code.

## Engine JMH microbenchmarks

These benchmarks are written in Java and are used to measure the performance of
specific parts of the engine. The sources are located in the `runtime` SBT
project, under `src/bench` source directory.
specific parts of the engine. The sources are located in the
`runtime-benchmarks` SBT project, under `engine/runtime-benchmarks` directory.

### Running the benchmarks

To run the benchmarks, use `bench` or `benchOnly` command - `bench` runs all the
benchmarks and `benchOnly` runs only one benchmark specified with the fully
qualified name. The parameters for these benchmarks are hard-coded inside the
JMH annotations in the source files. In order to change, e.g., the number of
measurement iterations, you need to modify the parameter to the `@Measurement`
annotation.
To run the benchmarks, use `bench` or `benchOnly` command in the
`runtime-benchmarks` project - `bench` runs all the benchmarks and `benchOnly`
runs only one benchmark specified with the fully qualified name. The
aforementioned commands are mere shortcuts to the
[standard JMH launcher](https://github.com/openjdk/jmh/blob/master/jmh-core/src/main/java/org/openjdk/jmh/Main.java).
To get the full power of the JMH launcher, invoke simply `run` with cmdline
options passed to the launcher. For the full options summary, see the
[JMH source code](https://github.com/openjdk/jmh/blob/master/jmh-core/src/main/java/org/openjdk/jmh/runner/options/CommandLineOptions.java),
or invoke `run -h`.

You can change the parameters to the benchmarks either by modifying the
annotations directly in the source code, or by passing the parameters to the JMH
runner. For example, to run the benchmarks with 3 warmup iterations and 2
measurement iterations, use:

```
sbt:runtime-benchmarks> run -w 3 -i 2 <bench-name>
```

### Debugging the benchmarks

Expand Down Expand Up @@ -94,12 +106,12 @@ the full options summary, either see the
or run the launcher with `-h` option.

The `std-benchmarks` SBT project supports `bench` and `benchOnly` commands, that
work the same as in the `runtime` project, with the exception that the benchmark
name does not have to be specified as a fully qualified name, but as a regular
expression. To access the full flexibility of the JMH launcher, run it via
`Bench/run` - for example, to see the help message: `Bench/run -h`. For example,
you can run all the benchmarks that have "New_Vector" in their name with just 3
seconds for warmup iterations and 2 measurement iterations with
work the same as in the `runtime-benchmarks` project, with the exception that
the benchmark name does not have to be specified as a fully qualified name, but
as a regular expression. To access the full flexibility of the JMH launcher, run
it via `Bench/run` - for example, to see the help message: `Bench/run -h`. For
example, you can run all the benchmarks that have "New_Vector" in their name
with just 3 seconds for warmup iterations and 2 measurement iterations with
`Bench/run -w 3 -i 2 New_Vector`.

Whenever you add or delete any benchmarks from `test/Benchmarks` project, the
Expand Down

0 comments on commit 0997cf2

Please sign in to comment.