Flaky: test_emissions_tracker_flush tests fail intermittently on master (order/shared-state dependent)

[davidberenstein1957]

Two independent investigations hit this while working on unrelated branches, so recording it rather than letting it keep costing people time.

Symptom

tests/test_emissions_tracker_flush.py fails intermittently when the full suite runs, most often:

• test_carbon_tracker_offline_flush
• test_carbon_tracker_offline_logging_output

Failures are on row counts — 0 == 1, 2 == 3 — i.e. an expected measurement did not land before the assertion.

What is established

• It reproduces on master, not only on feature branches: 1–2 of the 3 tests fail per full-suite run, and the same rate appears on unrelated branch tips.
• Every one of them passes in isolation. So it is order- or shared-state-dependent, not a broken assertion.
• Both investigations independently concluded the same thing while trying to attribute a failure to their own change, and neither could.

Likely cause

Scheduler timing plus state leaking between tests. PeriodicScheduler chains a threading.Timer per tick, so a tracker that a test does not stop keeps firing into whatever test runs next — 24 leaked, still-armed timers were measured across the suite in one investigation. A flush test asserting "N rows by now" is exactly the kind of assertion that a stray tick or a stolen scheduler slot perturbs.

Notes for whoever picks this up

• The re-arming Timer is removed by the single-daemon-thread scheduler change, which may make this disappear on its own — worth re-measuring after that lands rather than fixing twice.
• If it survives that, the fix is probably to stop asserting on wall-clock-dependent row counts, or to give these tests an injected clock rather than real sleeps.
• A tests/conftest.py autouse fixture that cancels leftover timers exists on one branch as a safety net. It should stay a safety net — tests should stop their own trackers.

Cheap first step: run pytest tests/ -q on master five times and record which tests fail, to confirm the set and the rate before changing anything.

[davidberenstein1957]

Additional finding from a separate investigation, which narrows this considerably.

tests/test_emissions_tracker_flush.py and tests/test_logging_output.py are order/state-sensitive around a shared emissions.csv in the repository root. They fail on origin/master under some orderings and pass after rm emissions.csv.

So the first thing to try is not a scheduler fix but making these tests write to a per-test temporary directory instead of the repo root — and checking whether a stray emissions.csv is being left behind by an earlier test in the run.

That likely also explains why the failures are on row counts (0 == 1, 2 == 3): the assertions are counting rows in a file that a previous test already wrote to.

The leaked-timer theory above may still be a contributing factor, but this one is cheaper to check and easier to fix.