[BUG] Test whether Babylon registry-clean results depend on registry run order

[lukekarrys]

Problem

During the investigation, Babylon registry-clean showed vlt at about 1.13× npm even though request-log analysis excluded registry-side causes:

• no packument exceeded one second across roughly 7,600 fetches;
• vlt tarballs were faster in aggregate;
• vlt packument p99 was about half npmjs’s.

The remaining suspect is harness order. npm always executes first, Babylon extracts about 1,858 tarballs, and vlt showed a 14-second run-to-run spread versus npm’s 3 seconds, consistent with environmental drift.

Experiment

Run Babylon registry-clean with registry order reversed (ideally alternate/randomize order across repetitions). scripts/registry/common.sh currently uses hard-coded command order even when BENCH_INCLUDE_REGISTRY lists a different order, so either honor list order or use a controlled experiment branch.

Acceptance criteria

• Capture a baseline and reversed/alternating-order runs under comparable conditions.
• Record raw per-run times and relevant machine/system context.
• Decide whether the visible deficit is a registry effect or harness order/drift.
• If order matters, make ordering configurable or randomized and document the methodology.
• Publish the experiment result in the issue.

Source

Performance investigation: item 10 — Babylon registry-clean anomaly