Skip to content

aggregate has no --workload-path, so runs made with --workload-path can't be aggregated #57

Description

@serhiy-bzhezytskyy

Is your feature request related to a problem? Please describe

run can take its workload either from a repository or from a local path — add_workload_source() in solrorbit/benchmark.py adds a mutually-exclusive --workload-repository / --workload-path pair, and it is applied to run, list and info.

aggregate isn't in that list. It defines only --workload-repository itself, so there is no way to point it at a local workload directory, even though aggregate() does need to load the workload (it reads the test procedure's schedule to get per-task iteration counts, via workload.load_workload and repository.name).

The effect is that a workflow which is fully supported for run stops working at aggregate time:

$ solr-orbit run --workload-path=/path/to/my-workload ...        # works
$ solr-orbit aggregate --test-runs=<id1>,<id2> --workload-path=/path/to/my-workload
solr-orbit: error: unrecognized arguments: --workload-path=/path/to/my-workload

So if I develop a workload locally and benchmark it with --workload-path — the "create your own workload" path in the docs — I can run it but I can't aggregate the results. Cloning the default workloads repository works fine and is what I ended up doing, so this is an inconvenience rather than a blocker.

Describe the solution you'd like

Call add_workload_source(aggregate_parser) and drop the hand-rolled --workload-repository, since the shared helper already provides it plus --workload-revision. That makes aggregate accept a workload source the same way run does, keeps the two options mutually exclusive, and reuses configure_workload_params for path normalisation and validation.

The one behavioural detail: a path sets workload.path, which is what makes the loader pick SimpleWorkloadRepository, so aggregate should skip setting repository.name when a path was given — otherwise the loader goes looking for the workload inside that repository instead.

Describe alternatives you've considered

Going further: a test run already records the workload it used, so aggregate could resolve the spec from the stored run and not need a workload source argument at all. That's arguably the better shape, but it is a larger change and it would need a decision about what to do when the stored workload is no longer available. I'd rather start with the parity fix, which matches the existing pattern.

Additional context

Came up while running a multi-configuration geonames campaign; I hit it before working out that the documented default repository clone satisfies aggregate on its own.

Minor and related, in the other repo: solr-orbit-workloads' .gitignore only lists .idea, so running a workload leaves untracked geonames/__pycache__/ etc. in the clone. That is enough to make the git checkout inside GitWorkloadRepository.update() fail with "Could not checkout [main]. Do you have uncommitted changes?", including when running --offline. Adding __pycache__/ there would avoid it — I can open that as a separate issue on solr-orbit-workloads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions