Tests flakiness and some test seems to be locking up in CI #1266
Comments
|
Apart from that it looks like we are seeing test flakiness when the CI runs the tests twice for the same commit (one for the tests and one for the executable builds) and the tests fail in one of the runs:
However, due to the issues in the first comment we haven't run the full CI tests in the last few commits, so it is possible this could be fixed in the latest master. We'll have to look into it. |
|
The latest commit was failing due to formatting, so I've fixed that up and pushed (30ea2f8). Let's see what results we get now. |
|
Okay, I can confirm that the test flakiness is still there in So far all the examples of this test failing have been in Python 3.8 in macOS, so something to consider. |
|
(Off the top of my head) I think this is reason that I didn't merge the 3.8 "upgrade" PR at the time -- there don't seem to be suitable wheels for that combination of Python, pygame & MacOS. |
|
(... and thanks for trawling through this, by the way, @carlosperate ...) |
|
While the issue in some of the CI jobs I posted definitely was due to missing macOS wheels on 3.8. I'm still seeing this issue in other runs. For example:
All the rest stuck in |
|
Also had a similar issue in Travis for Linux and Python 3.5: In this case travis stops the job after 10min without stdout, but it also looks stock at |
|
Added another entry to this comment: #1266 (comment) |
|
Adding |
|
@carlosperate top digital archaeology there... ;-) I wonder why the Python interpreter isn't copied over. :-/ |
|
I think I found the issue in my specific case. That venv was created from an app bundle, so the python executable in the venv was an alias to the interpreter inside that Mu installation. When I deleted that (testing) app bundle it it left an alias pointing to a file that didn't exist any more. My error traces are from my environment, which might be a different issue from the CI error, even if it looks similar (getting stuck while running the test_app.py tests). |
|
Got the following from the CI. However this crash doesn't fit the original pattern, where the tests get stuck until cancelled (or GH times out). In this case the error was thrown right away. Ubuntu 20.04 Py3.7: https://github.com/mu-editor/mu/runs/1911461023?check_suite_focus=true |
|
Repeating the CI tests until a "getting stuck" error was encountered. In this case I cancelled the run after an hour, and unfortunately in that case we don't get any additional output: Windows 2016 Py3.7: https://github.com/mu-editor/mu/runs/1911754485?check_suite_focus=true |
|
This run with the extra pytes flags is still stuck after 4h, let's see if tomorrow morning it has timed-out or stopped somehow. I doubt it'll actually include the stdout, but is worth checking: https://github.com/mu-editor/mu/runs/1913382027 Update: After the GH CI timeout we don't get any additional info than we do when it is manually stopped: |
|
Random conjectures, that are probably wrong. Just throwing them out there in case they stick...
🤷 |
|
Do the tests run in parallel? Could we have two processes trying to do the same thing (like creating a venv) and interfering with each other? The Travis output provides a tiny bit more, which could be helpful: |
|
@carlosperate good point.... I guess we could do the following when in CI:
|
|
Sounds like pytest runs test sequentially by default: https://www.tutorialspoint.com/pytest/pytest_run_tests_in_parallel.htm
Yeah, we should definitely try running I've been running the tests in my environment in a loop (and deleting the venv between iterations) and could not replicate the issue. However I've noticed that during these steps the tests do take a few seconds to continue at the same point and the PyQt warnings it prints are interesting: Sounds quite plausible that this is related to the problem. |
|
Agree it's worth getting to grips with. I don't have bandwidth right now, but definitely want to get a better handle on the use of Qt explicit/implicit threads in a testing (non-GUI) context. |
|
I'm getting similar warnings on the macOS console when running Mu normally. Not sure yet of the cause: |
|
Trying to trigger the Poking a bit on the code and it looks like it happens at the ipython kernel installation in the venv: Lines 517 to 529 in 14b24a9 Line 352 in 14b24a9 Lines 84 to 85 in 14b24a9 Lines 75 to 77 in 14b24a9 |
|
Thanks, @carlosperate -- my understanding was that a Qt QProcess would not instantiate a new thread, but would use the underlying OS Process creation (in Windows: We could obviously switch back to using a |
|
A couple of weeks ago the flakiness got worse, from a random job failing sometimes, to several jobs constantly failing on all test runs. Branch https://github.com/mu-editor/mu/tree/test-flakiness was created a while ago, I've rebased and created PR #1364 to collect test runs from the same commit and capture the current failure rate there Correction: We also had this full green CI 5 days ago, so maybe the one from today could have been a fluke as well, but we'll find out soon. |
|
Thanks for tracking, @carlosperate. From where you are, does it seem more likely to be some change in the CI infrastructure, which might have shaken out some latent race condition? Or some change in our codebase, eg around the Splash screen (the most obvious candidate) which started causing the problem? |
|
We can take an old commit and restart the tests, if they start to fail it might GH platform related. Took this random commit, around this time all merges to master were fully green: Pushed as a new branch, let's see what the result is: https://github.com/mu-editor/mu/actions/runs/658293392 To be fair this might be too long ago (31st Jan), if this passes without issues I'll get a little bit closer to today. Edit: Run https://github.com/mu-editor/mu/actions/runs/658293392 3 times and they all passed. |
|
Ahh, the conversation on Gitter reminded me that this issue was also present in TravisCI:
So it was likely introduced by a code change. |
|
Having just had a bunch of test runs hang -- I've just added a timeout of 30s to pytest. My point is simply to make it fail faster if it's going to hang (assuming that the pytest timeout has enough clout) |
|
The 30s timeout doesn't seem to achieve anything. I've applied pytest skip markers to progressively more tests. However, for the first time, just now I saw the [Ubuntu latest 3.8] run stall on "Install Mu dependencies" -- which suggests that it might not be our tests suite as such which is stalling, but the GH infrastructure. |
|
... ok; looks like that last run was a test stall. When I cancelled it, the logs eventually updated and failed after the tests dialogs. https://github.com/mu-editor/mu/runs/2156931209?check_suite_focus=true |
|
After a lot of to-ing and fro-ing and skipping tests: it looks like it's something in test_app which is causing the hangs. I've added a 30s timeout to the checkout & test steps; and a 5 minute timeout overall. That's just to try to get a faster fail if we get one! Trying now to narrow down the individual tests |
|
(Work happening over at #1388) |
|
Ok; merged #1388 into master -- but still more problems inside https://github.com/mu-editor/mu/runs/2157592932?check_suite_focus=true |
|
Correction: in both cases (Ubuntu 16 & Ubuntu latest) |
|
Work continuing in #1389 -- seeing if we can isolate a specific test or pattern of tests which cause the hang |
Yeah, that happens sometimes, often it's really stalled in the pytest step, but the log show that until it times out or gets manually cancelled. Sometimes the pip install or apt-install steps look like they get stuck, but in the end they always (as far as I've noticed) finish, it's just than instead taking less than a minute, they might take over 5min.
Yeah, the bast majority of the logs collected here and in #1364 all get stuck in test_app.
So is the current conclusion or theory that the issue is in |
Basically, yes. At least, leaving that one test skipped seems to give us a (nearly) flawless set of runs. The only failure I've had since wasn't a timeout but a segfault. (I've got the logs for that one stored locally somewhere). |
|
I think the original issue causing the issues listed here has been resolved. |
The GitHub Actions jobs have been queuing since yesterday.
The grey icons are test runs I had to cancel as they were stuck there for 4h+:
Other CI runs with a similar lock up:
Update
There was originally 2 issues, one to do with dependencies in Python 3.8 (already fixed) and another problem not yet identified where some runs get stuck: #1266 (comment)
The text was updated successfully, but these errors were encountered: