Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

appx test failure on windows: test_denial_of_service_prevented_str_to_int #96710

Closed
sobolevn opened this issue Sep 9, 2022 · 3 comments
Closed

appx test failure on windows: test_denial_of_service_prevented_str_to_int #96710

sobolevn opened this issue Sep 9, 2022 · 3 comments
Assignees
@gpshead
Labels
OS-windows type-bug An unexpected behavior, bug, or error

Comments

@sobolevn
Copy link
Member

sobolevn commented Sep 9, 2022

Looks like this test is flaky.
Example: https://dev.azure.com/Python/cpython/_build/results?buildId=111460&view=logs&j=c8a71634-e5ec-54a0-3958-760f4148b765&t=599737bc-ad72-560d-1530-0f89b05729e4

======================================================================
FAIL: test_denial_of_service_prevented_str_to_int (test.test_int.IntStrDigitLimitsTests.test_denial_of_service_prevented_str_to_int)
Regression test: ensure we fail before performing O(N**2) work.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\b\layout-appx-amd64\Lib\test\test_int.py", line 715, in test_denial_of_service_prevented_str_to_int
    self.assertLess(seconds_to_fail_extra_huge, seconds_to_convert/8)
AssertionError: 0.015625 not less than 0.013671875

Related #96684
CC @zooba
@sobolevn sobolevn added type-bug An unexpected behavior, bug, or error OS-windows labels Sep 9, 2022
@zooba
Copy link
Member

zooba commented Sep 9, 2022

Probably just needs the expected value raised. I expect this test should take multiple seconds if it fails, so having a threshold of ~10ms seems very low.

@gpshead, do you agree?

@zooba
Copy link
Member

zooba commented Sep 9, 2022

Also, I doubt #96684 is at all related. This test has nothing to do with the WMI tests or that module.

@bedevere-bot bedevere-bot mentioned this issue Sep 9, 2022
Merged
@gpshead
Copy link
Member

gpshead commented Sep 9, 2022

Yes I'll just raise the expected value to < 1/2 instead of < 1/8 and see if this ever comes up again. it's a regression test, there was nothing exact about the chosen values. false test passes due to system load or CI timing hiccups are fine.

@gpshead gpshead self-assigned this Sep 9, 2022
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 9, 2022
@gpshead @miss-islington
pythongh-96710: Make the test timing more lenient for the int/str DoS…
64f460d 
… regression test. (pythonGH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
gpshead added a commit that referenced this issue Sep 9, 2022
@gpshead
gh-96710: Make the test timing more lenient for the int/str DoS regre…
11e3548 
…ssion test. (#96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
This was referenced Sep 9, 2022
Merged
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 9, 2022
@gpshead @miss-islington
pythongh-96710: Make the test timing more lenient for the int/str DoS…
5fba3b1 
… regression test. (pythonGH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
@gpshead gpshead closed this as completed Sep 9, 2022
miss-islington added a commit that referenced this issue Sep 9, 2022
@miss-islington @gpshead
gh-96710: Make the test timing more lenient for the int/str DoS regre…
5a17200 
…ssion test. (GH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
miss-islington added a commit that referenced this issue Sep 9, 2022
@miss-islington @gpshead
gh-96710: Make the test timing more lenient for the int/str DoS regre…
fecda02 
…ssion test. (GH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
@lazka lazka mentioned this issue Sep 11, 2022
Merged
@bedevere-bot bedevere-bot mentioned this issue Oct 11, 2022
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 11, 2022
@gpshead @miss-islington
pythongh-96710: Make the test timing more lenient for the int/str DoS…
9a7b032 
… regression test. (pythonGH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
@bedevere-bot bedevere-bot mentioned this issue Oct 11, 2022
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 11, 2022
@gpshead @miss-islington
pythongh-96710: Make the test timing more lenient for the int/str DoS…
92d51b9 
… regression test. (pythonGH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 11, 2022
@gpshead @miss-islington
pythongh-96710: Make the test timing more lenient for the int/str DoS…
256b24b 
… regression test. (pythonGH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
@bedevere-bot bedevere-bot mentioned this issue Oct 11, 2022
Merged
ambv pushed a commit that referenced this issue Oct 11, 2022
@miss-islington @gpshead
[3.8] gh-96710: Make the test timing more lenient for the int/str DoS…
a44cc0a 
… regression test. (GH-96717) (#98197)

gh-96710: Make the test timing more lenient for the int/str DoS regression test. (GH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
ambv pushed a commit that referenced this issue Oct 11, 2022
@miss-islington @gpshead
[3.9] gh-96710: Make the test timing more lenient for the int/str DoS…
157a8b8 
… regression test. (GH-96717) (#98196)

gh-96710: Make the test timing more lenient for the int/str DoS regression test. (GH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
ambv pushed a commit that referenced this issue Oct 11, 2022
@miss-islington @gpshead
[3.7] gh-96710: Make the test timing more lenient for the int/str DoS…
e7fe111 
… regression test. (GH-96717) (#98195)

gh-96710: Make the test timing more lenient for the int/str DoS regression test. (GH-96717)

A regression would still absolutely fail and even a flaky pass isn't
harmful as it'd fail most of the time across our N system test runs.

Windows has a low resolution timer and CI systems are prone to odd
timing so this just gives more leeway to avoid flakiness.
(cherry picked from commit 11e3548)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gpshead gpshead

Labels
OS-windows type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gpshead @zooba @sobolevn