TestWindow_Distribution is flaky #1586
Labels
effort/hours
Estimated to take one or several hours
exp/beginner
Can be confidently tackled by newcomers
good first issue
Good issue for new contributors
kind/bug
A bug in existing code (including security flaws)
P1
High: Likely tackled by core team if no one steps up
Milestone
Additional information:
Describe the bug:
I tried updating the version of ipfs-cluster in Nixpkgs from 0.14.4 to 0.14.5: NixOS/nixpkgs#160460. A bot built the new version and ran the test suite but the test TestWindow_Distribution failed four times: 1, 2, 3, 4
Sometimes the test
TestWindow_Distribution/increasing_latency_distribution
failed and sometimesTestWindow_Distribution/random_latency_distribution
failed.Here are the important error messages from all test failures in an arbitrary order:
window_test.go:400: want: [4 3 3 2 2 1 1], got: [4 3 3 3 3 1 1]
window_test.go:400: want: [4 3 3 2 2 1 1], got: [5 4 3 2 2 1 1]
window_test.go:400: want: [18 11 8 7 9 3 1 4], got: [19 12 9 8 9 4 1 5]
window_test.go:400: want: [18 11 8 7 9 3 1 4], got: [19 12 9 8 9 4 1 5]
window_test.go:400: want: [18 11 8 7 9 3 1 4], got: [18 11 8 7 9 4 1 4]
The test failures only occurred on macOS (both x86-64 and aarch64), while the tests did not fail on Linux, but I don't think this has anything to do with the OS. Maybe the macOS build bots are just slower or more overloaded than the Linux bots.
As you can see above, the values in the
got
array are often one higher than the values in thewant
array. I think the tests depend on the precise timing of the scheduler and general execution speed. A simple fix would be to sleep for 100s of milliseconds by multiplying with 100 instead of sleeping for 10s of milliseconds here:https://github.com/ipfs/ipfs-cluster/blob/9e35b7bc9ba7fa3ad6153531ebd6d6a8c985d7d4/monitor/metrics/window_test.go#L372-L379
Alternatively, allowing the values in the
got
array to be one higher than the values in thewant
array would also work.The text was updated successfully, but these errors were encountered: