Benchmark workflow

[jonatas]

The objective here is to bring a wide discussion about the ideal model to run
multiple benchmarks and allow us to correlate them later.

Discussing the idea with @ryanbooz and @zseta , generally, we start with a simple plan, like:

Let's test database A against database B with scenarios X and Z.
Let's also run the same benchmark in different machine sizes to compare throughput and
efficiency. Plus, let's see how the performance goes with different parallelization levels and so on.

And then we start our journey in a few steps:

1. Provide the machines - set up the tsbs machine that will send data and
   queries to the targets with different configuration
2. Setup the target machines installing OS/database and providing a common-auth to connect
3. Setup the initial configuration we want to benchmark: how many rows of data, how
   dense is the time-series (rows/day), and the type of data IoT or DevOps.
4. Later we adapt the initial config with the derived configuration targetting different
   machines and parallelization levels.

Machines ready, we're done to start using tsbs_load with --config and
synchronize each run to not mess with parallel benchmarks in the master machine
that could affect the performance.

To run every command, we should be in a screen session to keep it detached
from our ssh connection.

Later, we need to collect all text reports and manually capture and move the
data to a spreadsheet to allow us to correlate the benchmarks. Allowing us to
better understand what params work best in which context.

The most important info we collect is the throughput of rows/sec and metrics/sec of
each scenario: IoT/DevOps.

This is the first part, that covers load but we also have run_queries which also allows using
different sets of configs like parallelization and different types of queries in the context.

From the queries, we have an identifier per query and we also can run several
queries in parallel with variants of the initial configuration, like before and
after compressing data.

After getting metadata from all the queries, we need to manually capture the
performance information of each configuration set and move to a spreadsheet
to later correlate the data.

So, several of these steps are done manually and we also need to keep an eye in
the pipeline to reuse the same tsbs machine to push the data.

We don't have a specific issue here but an open space to see as a community how we can approach the problem and
improve the way we work to have a better flow and also allow us to reuse previous benchmarks without needing to rerun everything again.

[jonatas]

Configuration ideas

One idea I have about running multiple configurations is to expand our regular config file to allow us to run some options that can multiply the configuration and serialize it.

Example of config:

data-source:
  type: SIMULATOR
  ... # skipping details here
loader:
  db-specific:
    # ...
    host: "some-ip-here"
    partition-index: true
    partitions: 1
    time-partition-index: false
    use-hypertable: true
    use-jsonb-tags: false
    # ... a lot more configs here
  runner:
    batch-size: 10000
    channel-capacity: "0"
    db-name: benchmark
    do-abort-on-exist: false
    do-create-db: true
    do-load: true
    flow-control: false
    hash-workers: true
    limit: 2016000000
    reporting-period: 30s
    seed: 135
    workers: 24

Now let's imagine we would like to test the following scenarios:

• 3 machines
• different set of workers
• different batch sizes

I could do the following changes, only transforming the configs into an array of values:

data-source:
  type: SIMULATOR
  ... # skipping details here
loader:
  db-specific:
    # ...
    host:
      - "some-ip-here"
      - "second-machine-ip-here"
      - "third-machine-ip-here"
    # ... a lot more configs here
  runner:
    batch-size: [1000, 5000, 10000]
    workers: [8, 16, 24]

And then, the tsbs_load could prepare the plan of configs and run sequentially in the target machines.

Collecting and storing results

From the storage perspective, I'd love if we set a TSBS server where people around the world could push different benchmarks to our server. Dumping different time-series related to benchmark, even tracking how CPU and IO behave while executing some specific benchmark task.

It can be the first step to have a tsbs website with all database benchmarks exposed. It can be a rich source to learn what technology works better in what scenario.