libgcc_s.so.1 must be installed for pthread_cancel to work during SCT's post-install validation

[gaspardcereza]

Description

When submitting our Shimming-Toolbox Pull Request to the CI, we get an error that seems to be related to the installation of SCT. These tests were passing a few days ago and we have not modified our PR since then, so it seems to come from SCT's end.

This happens with this PR and this one.

The bug happens when testing under Ubuntu-18.04 3.8.

See error output

Spinal Cord Toolbox (git-master-666237aeb31246ff163905c7f3cf6cbe8d696e44)

sct_check_dependencies 
--

SCT info:
- version: git-master-666237aeb31246ff163905c7f3cf6cbe8d696e44
- path: /home/runner/work/spinalcordtoolbox
OS: linux (Linux-5.4.0-1056-azure-x86_64-with-debian-buster-sid)
CPU cores: Available: 2, Used by ITK functions: 2
RAM: Total: 6954MB, Used: 495MB, Available: 6128MB
Check Python executable.............................[OK]
  Using bundled python 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59) 
[GCC 7.5.0] at /home/runner/work/spinalcordtoolbox/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if colored is installed.......................[OK] (1.4.2)
Check if dipy is installed..........................[OK] (1.4.1)
Check if futures is installed.......................[OK]
Check if h5py is installed..........................[OK] (2.10.0)
Check if Keras (2.1.5) is installed.................[OK] (2.1.5)
Check if ivadomed is installed......................[OK] (2.8.0)
Check if matplotlib is installed....................[OK] (3.3.4)
Check if nibabel is installed.......................[OK] (3.2.1)
Check if numpy is installed.........................[OK] (1.19.5)
Check if onnxruntime (1.4.0) is installed...........[OK] (1.4.0)
Check if pandas is installed........................[OK] (1.1.5)
Check if psutil is installed........................[OK] (5.8.0)
Check if pyqt5 (5.11.3) is installed................[OK] (5.11.3)
Check if pytest is installed........................[OK] (6.2.5)
Check if pytest-cov is installed....................[OK] (2.12.1)
Check if raven is installed.........................[OK]
Check if requests is installed......................[OK] (2.26.0)
Check if requirements-parser is installed...........[OK] (0.2.0)
Check if scipy is installed.........................[OK] (1.5.4)
Check if scikit-image is installed..................[OK] (0.17.2)
Check if scikit-learn is installed..................[OK] (0.24.2)
Check if tensorflow (1.5.0) is installed............[OK] (1.5.0)
Check if torch (1.5.0+cpu) is installed.............[OK] (1.5.0+cpu)
Check if torchvision (0.6.0+cpu) is installed.......[OK] (0.6.0+cpu)
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.62.2)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3 is installed.......................[OK] (1.26.6)
Check if pytest_console_scripts is installed........[OK]
Check if wquantiles is installed....................[OK] (0.4)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if figure can be opened with matplotlib.......Generating new fontManager, this may take some time...
libgcc_s.so.1 must be installed for pthread_cancel to work
./install_sct: line 738:  3219 Aborted                 (core dumped) sct_check_dependencies

Installation validation Failed!


Installation failed!

Please copy the output of this Terminal (starting with the command install_sct) and upload it as a .txt attachment in a new topic on SCT's forum:

--> http://forum.spinalcordmri.org/c/sct

Error: Process completed with exit code 1.

Steps to Reproduce

Follow this link and hit the Re-run jobs button.

Expected behavior:

All the tests should pass.

Actual behavior:

The Ubuntu-18.04 3.8 test fails and cancels the execution of all the other tests.

[joshuacwnewton]

Thanks for bringing this to our attention! It looks like this is affecting SCT's main test suite, too. Notably, only some GitHub Actions CI runners are affected, though: Debian 9, Debian 10, Ubuntu 18.04, and CentOS 7.

This also looks likes a recent change, as SCT's recurring cron job was fine prior to today: https://github.com/spinalcordtoolbox/spinalcordtoolbox/actions

Investigating further...

[joshuacwnewton]

Diffing the install logs, I found that these were the changes to the installed packages. The biggest change I notice is an update from ivadomed 2.7.4 to ivadomed 2.8.0. I presume this also lead to updates from ivadomed's dependencies, as well. This doesn't exactly get us closer to figuring out the precise cause... but at least I can try quickly running the test suite with ivadomed==2.7.4 pinned just to see if that isolates the problem.

EDIT: Pinning ivadomed==2.7.4 indeed causes the test suite to pass. So, somewhere in those changed packages is something that affects how matplotlib finds libgcc_s.so.1. 🤔

[joshuacwnewton]

It seems to me that the problem might be in some way connected to the way we import matplotlib in sct_check_dependencies.

Aha! Currently, in sct_check_dependencies, we...

1. Import PyQt first, then we...
2. Import matplotlib.pyplot afterwards.

This is the exact import order that was reported to cause this error in ContinuumIO/anaconda-issues#9190.

So, I tried importing matplotlib.pyplot before PyQt in 8334247. As a result, the error disappears completely. So, in the short-term, hooray! We have a workaround. 🎉

[joshuacwnewton]

What I still don't understand:

• Why did these specific package changes trigger the issue in the first place?
• Why were the Debian 9, Debian 10, Ubuntu 18.04, and CentOS 7 GitHub Actions runners affected, but not the other runners?

Still, hopefully this provides some new data points to help others reproduce the issue.

[gaspardcereza]

Thanks for the investigation @joshuacwnewton ! I hope your PR will fix the problem because our CI is currently not very useful... Let me know if you need any help with this matter.