[QA] Fix flaky data_manager unit tests

[petar-qb]

Description

See issue for more.

Changes introduced in this PR:

• deterministically solve the problem of unstable data_manager tests
• speed-up data_manager unit tests a lot

Screenshot

Notice

• I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

  • I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.
  • I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorized to submit this contribution on behalf of the original creator(s) or their licensees.
  • I certify that the use of this contribution as authorized by the Apache 2.0 license does not violate the intellectual property rights of anyone else.
  • I have not referenced individuals, products or companies in any commits, directly or indirectly.
  • I have not added data or restricted code in any commits, directly or indirectly.

[antonymilne]

@petar-qb this is reallyawesome work 💯 Not only have you fixed the flakiness, the data manager tests have gone from taking ~35s (which was always a source of mild irritation for me) to around 3.5s! I've tested the latest version I pushed by running pytest test_data_manager.py 100 times. Before the change, that would fail ~10% of the time, and now there's not a single failure 🥳

I made a couple of commits, see what you think. My reasoning is:

• we should try to keep tests as close to real/unmocked behaviour as possible, and only tests in data manager use time.sleep
• hence let's not make it autouse=True or increase the fixture scope outside where it's strictly needed and instead keep it as explicitly opt-in
• set tick=True for similar reasons to ensure time is only frozen for the purpose of fast-forwarding time.sleep and does not artificially make subsequent calls to .load seem simultaneous
• monkeypatching the time.sleep was super clever but I would again prefer to be explicit, especially given that freezegun comes with the .tick method for exactly this, and monkeypatching is always quite fragile (e.g. from time import sleep; sleep() isn't the same as import time; time.sleep())
• the downside is that any future tests that use time.sleep will not automatically benefit, but I think this is ok because I don't anticipate there being many such things (just tests of caching really), and I'd rather be explicit here rather than it being too magical

[petar-qb]

@antonymilne thanks for taking the PR to a new level. I really like how the tests cases are now much more realistic (with timeouts of 300s and 100s). I agree with all your bullets from the comment above, but unfortunately I can't approve the PR I created 😄.

[huong-li-nguyen]

LGTM! Thank you guys 💪