[Bug]: aggregate always fails — update_config_object reads a TestRun attribute that does not exist

[serhiy-bzhezytskyy]

Describe the bug

solr-orbit aggregate fails for any input:

[ERROR] ❌ Cannot aggregate. 'TestRun' object has no attribute 'cluster_config_instance_params'.

update_config_object (solrorbit/aggregator.py:169) reads test_run.cluster_config_instance_params. TestRun defines cluster_config_params (solrorbit/metrics.py:890). The name being read exists nowhere else in the tree:

$ grep -rn 'cluster_config_instance_params' . --include='*.py'
solrorbit/aggregator.py:169        # the only occurrence, and it is a read

This looks like a half-applied rename. 2c85d505 ("Rebase renamed-components onto latest developments") renamed the attribute on TestRun from provision_config_instance_params to cluster_config_params, but renamed this read site to cluster_config_instance_params. e10342d1 ("fixes for aggregate command") later corrected the config key on the same line (cluster_config_instance.params → cluster_config.params) and left the attribute name. Both commits came in with the port, so their PR numbers are OpenSearch Benchmark's rather than this repo's — hence the SHAs.

Unlike the null-metrics crash reported separately, this one does not depend on the data. It is on the plain aggregate → build_aggregated_results → update_config_object path, so it fires every time.

A side effect worth noting: because the read never succeeds, the cluster-config params never reach the aggregated result at all.

To reproduce

solr-orbit aggregate --test-runs=<id1>,<id2>

Any two compatible test runs will do.

If the runs contain an operation with null metrics (which is the case for the default geonames workload, because of optimize), the null-metrics crash happens first during metric reduction — that is why this one only becomes visible once that is fixed.

Expected behavior

aggregate reads test_run.cluster_config_params and completes, and the cluster-config params reach the aggregated result. With that one name corrected, the aggregate for my runs carries cluster-config-instance-params: {"heap_size": "6g"} as it was meant to.

Host / Environment

• solr-orbit edb3d03e (main, up to date at time of report), installed from source, Python 3.12
• Reproducible with any test runs; found with geonames via --pipeline=docker on Solr 10.0.0
• macOS 15 (arm64), Docker Desktop

Additional context

Why the test suite doesn't catch it, since that seems worth flagging for the project generally: tests/aggregator_test.py passes bare Mock() test runs, and a Mock creates whatever attribute is asked of it, so the read succeeds regardless of the name. The full suite passes on main while aggregate cannot run once.

Mock(spec=TestRun) doesn't help either — spec inspects the class, which knows nothing about attributes assigned in __init__, so it raises on cluster_config first and points at the wrong line. A regression test for this has to construct a real TestRun.

The same code is present in OpenSearch Benchmark, which this is a port of, so it looks inherited rather than introduced here. I couldn't find an existing report in either project.

Relevant log output

  File ".../solrorbit/aggregator.py", line 193, in build_aggregated_results
    self.update_config_object(test_run)
  File ".../solrorbit/aggregator.py", line 169, in update_config_object
    "cluster_config.params", test_run.cluster_config_instance_params)
AttributeError: 'TestRun' object has no attribute 'cluster_config_instance_params'.
  Did you mean: 'cluster_config_instance'?