Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flakiness of test_measure_sliding_window #115322

Merged

Conversation

cpfair
Copy link
Contributor

@cpfair cpfair commented Apr 9, 2024

Proposed change

I noticed this test failure on #115146.

It appears that the async_block_till_done() call after inserting the entity states is not always sufficient to update the sensor values in time for the proceeding assertions. Removing the earlier (apparently redundant) call to async_block_till_done() seems to fix the issue.

I have no knowledge in the HA async architecture, but here's some speculation: Perhaps there are a cascade of tasks, some of which are not visible async_block_till_done, which continue executing beyond that synchronisation barrier. These could lead to nondeterministic results. By delaying the start of this hypothetical cascade of tasks until the test case's entity data is in place, there is no chance of stale state appearing later in the test.

This change appears to solve the flakiness in this particular test - but does not address any deeper root cause that may exist. It also does not attempt to identify and fix any other tests which may be suffering from this issue.

Before change: n=10,000 trials, 20 failures $\implies p_{fail} \ge 0.0008492$ with a 99% CI
After change: n=10,000 trials, 0 failures, $\alpha = (1 - 0.0008492)^{10,000} \approx 0.002 \le 0.01$ suggesting the flakiness is resolved

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@cpfair cpfair force-pushed the test_measure_sliding_window-flakiness branch from d787bcc to 8a9090a Compare April 10, 2024 02:52
Copy link
Member

@bdraco bdraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cpfair

@bdraco bdraco merged commit 0636ba3 into home-assistant:dev Apr 10, 2024
24 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants