[Bug]: aggregate fails with TypeError when an operation reports null throughput (e.g. optimize)

[serhiy-bzhezytskyy]

Describe the bug

solr-orbit aggregate crashes for any set of test runs in which an operation has null values in its min/max metric fields:

[ERROR] ❌ Cannot aggregate. '<' not supported between instances of 'NoneType' and 'NoneType'.

calculate_weighted_average in solrorbit/aggregator.py reduces min/max with value.get(metric_field, 0). The 0 default only applies when the key is absent — it does not help when the key is present with value None, which is what the stored results contain. min() then compares None to None and raises.

This is not an edge case in practice: the optimize operation in the geonames workload reports error_rate: 1.0 with all-null throughput on every run I have, and optimize is part of the default workload — so every aggregation of a geonames campaign fails. The max branch has the same problem, and the percentile/median branch below it (value * iterations after a .get(..., 0)) would raise on None for the same reason.

There is a second, independent occurrence in calculate_rsd, on the path from build_aggregated_results_dict that passes the per-run mean values. I only found that one by running on real data after unit tests for the first site were already green.

Aggregating N runs is the documented way to get a trustworthy result out of a noisy environment, so in practice this blocks multi-run campaigns on the default workload.

To reproduce

1. Run the geonames workload more than once (repeat with a different --user-tag):

   solr-orbit run --pipeline=docker --distribution-version=10.0.0 \
     --workload-path=<path>/solr-orbit-workloads/geonames \
     --cluster-config-params="heap_size:6g" -k
   

2. Aggregate the resulting test runs:

   solr-orbit aggregate --test-runs=<id1>,<id2>,<id3>,<id4>,<id5>
   

3. It fails immediately — ❌ FAILURE (took 0 seconds), nothing written.

The stored results contain, for every run:

{ "task": "optimize",
  "error_rate": 1.0,
  "throughput": { "min": null, "mean": null, "median": null, "max": null, "unit": "ops/s" } }

Minimal reproduction against the real method, if it helps — the code under test is not mocked, only the cfg/store __init__ is bypassed:

from solrorbit.aggregator import Aggregator
agg = Aggregator.__new__(Aggregator)
agg.test_runs = {"run1": None, "run2": None}
agg.accumulated_iterations = {"run1": {"optimize": 1}, "run2": {"optimize": 1}}
bad = {"min": None, "mean": None, "median": None, "max": None, "unit": "ops/s"}
agg.calculate_weighted_average({"throughput": [bad, bad]}, "optimize")   # TypeError

A healthy operation with numeric values aggregates fine, which is what points at None as the trigger rather than anything about the harness.

Expected behavior

aggregate completes and produces aggregated results. For an operation with no valid samples I would expect the aggregate to carry null through for that metric — honest, it says "not measured" — rather than substituting 0, which would read as a real measurement of zero throughput.

Note that aggregate_json_by_key in the same file already skips nulls deliberately (next((obj for obj in json_elements if obj is not None), None)), so handling null metrics is an established pattern here; calculate_weighted_average just doesn't do it.

Host / Environment

• solr-orbit edb3d03e (main, up to date at time of report), installed from source, Python 3.12
• Solr 10.0.0 (official Docker image) via --pipeline=docker
• Workload: geonames, full corpus, 6g heap, 5 runs per configuration
• macOS 15 (arm64), Docker Desktop

Additional context

Found while running a 5-runs-per-configuration geonames campaign to compare Solr on Lucene 10.4 against Lucene 11 — aggregate is what computes the weighted means and the per-metric RSD that make a multi-run result readable, so this was a hard stop.

git log -L225,225:solrorbit/aggregator.py traces these lines to the OpenSearch Benchmark commits the port inherited (c6ec0a11 "Add aggregate command" and 2d14ed40 "Change min/max to overall_min/overall_max"), and the same code is present upstream, so this looks inherited rather than introduced by the Solr port. I couldn't find an existing report in either project.

Separately (can open a dedicated issue if that's useful): optimize reporting error_rate: 1.0 and null throughput on every run suggests the force-merge step isn't being measured at all. It's equal on both sides of a comparison so it doesn't skew compare, but it is the direct cause of this crash. I haven't root-caused that one.

Relevant log output

Traceback (most recent call last):
  File ".../solrorbit/benchmark.py", line 1218, in dispatch_sub_command
    aggregator_instance.aggregate()
  File ".../solrorbit/aggregator.py", line 284, in aggregate
    aggregated_results = self.build_aggregated_results()
  File ".../solrorbit/aggregator.py", line 176, in build_aggregated_results
    aggregated_results = self.build_aggregated_results_dict()
  File ".../solrorbit/aggregator.py", line 129, in build_aggregated_results_dict
    aggregated_task_metrics = self.calculate_weighted_average(task_metrics, task)
  File ".../solrorbit/aggregator.py", line 225, in calculate_weighted_average
    weighted_metrics[metric]['overall_min'] = min(value.get(metric_field, 0) for value in values)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'