Skip to content

Fix flaky test_support_bundle_lifecycle#10181

Open
smklein wants to merge 2 commits intomainfrom
bundle-flake
Open

Fix flaky test_support_bundle_lifecycle#10181
smklein wants to merge 2 commits intomainfrom
bundle-flake

Conversation

@smklein
Copy link
Copy Markdown
Collaborator

@smklein smklein commented Mar 27, 2026

The sp_ereport_ingester background task runs on a 30-second timer and ingests ereports from simulated SPs into the database. The support bundle collection test asserts that zero ereports are found, but if the ingester fires before the bundle collector runs, it populates the DB with 11 ereports from the simulated SPs (configured in sp_sim_config.test.toml), causing the assertion to fail.

Fix this by disabling the sp_ereport_ingester in the test config. The ingester has its own dedicated tests that construct it directly, so no test coverage is lost. Comments are added at both assertion sites explaining the dependency on the ingester being disabled.

Fixes #10179

The sp_ereport_ingester background task runs on a 30-second timer and
ingests ereports from simulated SPs into the database. The support
bundle collection test asserts that zero ereports are found, but if the
ingester fires before the bundle collector runs, it populates the DB
with 11 ereports from the simulated SPs (configured in
sp_sim_config.test.toml), causing the assertion to fail
nondeterministically.

Fix this by disabling the sp_ereport_ingester in the test config. The
ingester has its own dedicated tests that construct it directly, so no
test coverage is lost. Comments are added at both assertion sites
explaining the dependency on the ingester being disabled.
@smklein smklein requested a review from hawkw March 27, 2026 22:07
@hawkw
Copy link
Copy Markdown
Member

hawkw commented Mar 27, 2026

The sp_ereport_ingester background task runs on a 30-second timer and ingests ereports from simulated SPs into the database. The support bundle collection test asserts that zero ereports are found, but if the ingester fires before the bundle collector runs, it populates the DB with 11 ereports from the simulated SPs (configured in sp_sim_config.test.toml), causing the assertion to fail.

Gah, sorry. This one's my fault. 😬

Copy link
Copy Markdown
Member

@hawkw hawkw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, the change to lose the ereport ingester task's output in test_omdb_success_cases is a bit of a bummer, I kinda wonder if we could have those tests edit the config file to turn the task back on or something? But if that's too annoying, I'd definitely rather fix the flake than test the actual output...

Comment on lines -905 to -910
total ereports received: <TOTAL_EREPORTS_RECEIVED_REDACTED>
new ereports ingested: <NEW_EREPORTS_INGESTED_REDACTED>
total HTTP requests sent: <TOTAL_HTTP_REQUESTS_SENT_REDACTED>
total collection errors: <TOTAL_COLLECTION_ERRORS_REDACTED>
reporters with ereports: <REPORTERS_WITH_EREPORTS_REDACTED>
reporters with collection errors: <REPORTERS_WITH_COLLECTION_ERRORS_REDACTED>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, losing this is a bit of a shame, I'd really much rather see the more realistic output in the test output here...

webhook_deliverator.second_retry_backoff_secs = 20
read_only_region_replacement_start.period_secs = 999999
sp_ereport_ingester.period_secs = 30
sp_ereport_ingester.disable = true
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would maybe like to see a comment here explaining why this task is disabled for tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test failed in CI: test_support_bundle_lifecycle

2 participants