Fix flakiness of test_measure_sliding_window #115322
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed change
I noticed this test failure on #115146.
It appears that the
async_block_till_done()
call after inserting the entity states is not always sufficient to update the sensor values in time for the proceeding assertions. Removing the earlier (apparently redundant) call toasync_block_till_done()
seems to fix the issue.I have no knowledge in the HA async architecture, but here's some speculation: Perhaps there are a cascade of tasks, some of which are not visible
async_block_till_done
, which continue executing beyond that synchronisation barrier. These could lead to nondeterministic results. By delaying the start of this hypothetical cascade of tasks until the test case's entity data is in place, there is no chance of stale state appearing later in the test.This change appears to solve the flakiness in this particular test - but does not address any deeper root cause that may exist. It also does not attempt to identify and fix any other tests which may be suffering from this issue.
Before change: n=10,000 trials, 20 failures$\implies p_{fail} \ge 0.0008492$ with a 99% CI$\alpha = (1 - 0.0008492)^{10,000} \approx 0.002 \le 0.01$ suggesting the flakiness is resolved
After change: n=10,000 trials, 0 failures,
Type of change
Additional information
Checklist
ruff format homeassistant tests
)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest
.requirements_all.txt
.Updated by running
python3 -m script.gen_requirements_all
..coveragerc
.To help with the load of incoming pull requests: