Hypothesis health checks frequently fail

[Hexcles]

Example: https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=45217&view=logs&j=47b03e2b-d31e-537f-c8cf-66c80d147ca1&t=67b19343-5bc4-57b8-71e9-6c31e5bc256c&l=64

E hypothesis.errors.FailedHealthCheck: Data generation is extremely slow: Only produced 8 valid examples in 1.02 seconds (5 invalid ones and 4 exceeded maximum size). Try decreasing size of the data you're generating (with e.g.max_size or max_leaves parameters).
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.too_slow to the suppress_health_check settings for this test.

According to Azure: "This task has failed 12 times across 1042 pipeline runs in the last 14 days."

[Hexcles]

I'm going to temporarily disable this health check on Windows.

[Hexcles]

This fails on Mac, as well. In fact, it fails frequently only on Azure. Perhaps a performance issue?

[Hexcles]

Actually, it also failed on Taskcluster (Linux)...

I've merged the workaround to disable the health check in Python 3, so I'm downgrading this to roadmap. @svillar could you investigate why Python 3 is slower when running these hypothesis tests?

[Hexcles]

It might be possible that the data generation has always been slow, and Python 3 implementation only pushed us a tad over the threshold. It'd be good if we can get the generation rates and compare them.

[svillar]

It might be possible that the data generation has always been slow, and Python 3 implementation only pushed us a tad over the threshold. It'd be good if we can get the generation rates and compare them.

I've run the test_manifest_to_json* tests with health checks on (--hypothesis-show-statistics) measuring the time required to complete the execution and it turns out that I'm consistently getting lower figures for python3 executions (7.2s vs 4.6s) so not sure what's going on.

Truth is, that for those tests data generation implies >98% of the time.