Salesforce Hyper: Hot runs are measuring cold times

[caetanosauer]

After recent refactorings to the benchmark scripts, the hot runs reported by Hyper are actually cold. This is because the HyperProcess object in our query script actually restarts the server from scratch in every iteration:

with HyperProcess(telemetry=Telemetry.DO_NOT_SEND_USAGE_DATA_TO_TABLEAU) as hyper:
    with Connection(hyper.endpoint, 'hits.hyper', CreateMode.NONE) as connection:
        start = timeit.default_timer()
        rows = connection.execute_list_query(query)
        end = timeit.default_timer()

To properly measure a hot run, the repetitions must happen inside that with block, as we had them in our original script.

This is the second time this happens. The first one was when our scripts were modified to "prevent cheating" when there was no cheating taking place at all.

What is the ClickBench official guideline on how systems should report their hot runs? Should we instead provide scripts that work on a client-server fashion, so that they can be explicitly started and stopped? The new script structure does not seem to allow this embedded mode anymore, since query has to be called per repetition of a query now instead of per query (with repetitions inside it).