Data frame

[dantleech]

Introduces the concept of a data frame.

The data frame can be used programatically in reports and can be accessed through the expression language. The data frame is introduced as a new parameter suite in the expression report.

This addition will enable any value from the suite to be referenced, the following expression f.e. will show a % comparison to the named benchmark for each row:

percent_diff(mode(result_time_avg), mode(suite[subject_name = "benchFoo"]["result_time_avg"]))

TODO:

• Create DataFrame and associated classes
• Support expression access for DataFrames
• Improve tests
• Regression test perforamance
• Update docs
• Split out unrelated improvements/features

Performance

Initially report generation performance was degraded due to the way the evaluation cache worked (it serialized the parameters as a cache key), removing the cache actually results in a significant gain for report generation at the expense of assertions and progtress loggers:

+------------------------------+---------------+-------------+------+-----+------------------+-------------------+----------------+
| benchmark                    | subject       | set         | revs | its | mem_peak         | mode              | rstdev         |
+------------------------------+---------------+-------------+------+-----+------------------+-------------------+----------------+
| ParserBench                  | benchEvaluate | comp. w/tol | 10   | 10  | 1.883mb -2.83%   | 74.146μs +1.40%   | ±1.18% -62.76% |
| ParserBench                  | benchEvaluate | comp.       | 10   | 10  | 1.883mb -2.83%   | 63.677μs +0.55%   | ±2.23% -6.48%  |
| AssertionProcessorBench      | benchAssert   | 0           | 10   | 10  | 3.093mb -3.53%   | 412.578μs +29.59% | ±1.16% -24.57% |
| ExpressionGeneratorBench     | benchGenerate | 0           | 10   | 10  | 12.214mb -52.33% | 8.260ms -35.83%   | ±1.51% -20.55% |
| RunCommandBench              | benchDefault  | 0           | 1    | 10  | 5.881mb -1.02%   | 73.072ms -20.05%  | ±2.25% -3.37%  |
| RunCommandBench              | benchInBand   | 0           | 1    | 10  | 5.881mb -1.02%   | 74.573ms -25.33%  | ±1.45% -50.43% |
| RunCommandBench              | benchNoEnv    | 0           | 1    | 10  | 5.501mb -1.26%   | 21.621ms -3.46%   | ±2.16% -10.81% |
| VariantSummaryFormatterBench | benchFormat   | 0           | 10   | 10  | 3.269mb +1.98%   | 622.163μs +9.24%  | ±2.13% +6.92%  |
+------------------------------+---------------+-------------+------+-----+------------------+-------------------+----------------+

As the regressions are at are sub-millisecond it seems like an acceptable trade-off with the added benefit of removing a cache.

[dantleech]

worth prefixing _? the columns are are not extensible however

[dantleech]

this is a BC break potentially - add another method and deprecate the array one

[tillkruss]

This is fantastic!

[dantleech]

Nice! I'm not sure percent_diff works here though - red is "bad", while negative is "good". You could also change the expresion to use the x format or do that maths for percentage rather than using percent_diff to remove the colors. Or just leave it if it works :)

[tillkruss]

Agreed. The red is a little aggressive. Could you explain how to do the Nx notation?

[dantleech]

Math and concatenation: (mode(suite[subjectName="foo"]) / mode(result_time_avg)) ~ "x" or something ...