You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the runner creates a complicated directory hierarchy to organize benchmarks. This both good and bad...
good: To send results for only one benchmark suite, all you have to do is send the one directory. (But then again, if you want to send results for the one benchmark suite but for all environments, then you have to send three directories and do some glob that's annoying to write)
bad: To aggregate results, we have to know about this directory structure in the aggregator
bad: To aggregate manually run results with existing aggregator configs, we have to create directories
We should rethink this and other parts of the runner-aggregator interface to simplify what's going on and make it easier to do things like manually run the aggregator on a new collection of files that aren't organized in the complicated way.
Of course, adding a command-line option to the aggregator telling it to load files from a particular directory would work to solve this issue... but how would it work when config files reference each other? Would paths mentioned in aggregator recipes be relative to the specified path instead?
There's probably more considerations for this issue...
What we need to pass from runner to aggregator
Actual data given by the benchmarks
Parameters used to launch the benchmarks. Can include command line
Environment and machine information
Possibly even the "raw" aggregator config used to read raw results into a pandas dataframe
The text was updated successfully, but these errors were encountered:
Currently, the runner creates a complicated directory hierarchy to organize benchmarks. This both good and bad...
We should rethink this and other parts of the runner-aggregator interface to simplify what's going on and make it easier to do things like manually run the aggregator on a new collection of files that aren't organized in the complicated way.
Of course, adding a command-line option to the aggregator telling it to load files from a particular directory would work to solve this issue... but how would it work when config files reference each other? Would paths mentioned in aggregator recipes be relative to the specified path instead?
There's probably more considerations for this issue...
What we need to pass from runner to aggregator
The text was updated successfully, but these errors were encountered: