Failing when testing under heavy system-load

[vanHoesel]

When building a huge application layer and with tons of Perl modules that need to be installed, when the system is quite busy, the test may occasionally fail.

It does even tell you in the when.t test:

These tests may occationally fail due to small timing differences.

But failing these test randomly is not an option in huge CI/CD environments where you require reliable operation.

Is it possible to make these test pass when the values are approximately right? I'm thinking about making the sleep time twice as long, or even longer, and then allowing the results to be 95% accurate.

[choroba]

I fear this isn't possible. Under very heavy load once the machine starts swapping, the times can be 20 times longer.

[choroba]

Even this change doesn't always work :-(

modified   t/when.t
@@ -23,13 +23,13 @@ scale(2);
 is &scale, 2;
 my $now = &time;
 sleep 2;
-is(&time - $now, 4);
+cmp_ok(&time - $now, '>=', 4);
 
 to(CORE::time);
-is(&time - CORE::time, 0);
+cmp_ok(&time - CORE::time, '>=', 0);
 
 scale(scale() * 2);
-is(&time - CORE::time, 0);
+cmp_ok(&time - CORE::time, '>=', 0);
 
 Time::Warp::reset(); to(&time + 5);
-is(&time - CORE::time, 5);
+cmp_ok(&time - CORE::time, '>=', 5);

When running 600 instances of the test in parallel, one of them failed with

not ok 8
#   Failed test at t/when.t line 35.
#     '4'
#         >=
#     '5'