Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky CI failures #3784

Closed
Remi-Gau opened this issue Jun 28, 2023 · 1 comment · Fixed by #3785
Closed

Flaky CI failures #3784

Remi-Gau opened this issue Jun 28, 2023 · 1 comment · Fixed by #3785
Labels
Infrastructure This issue describes a problem with the project's infrastructure (CI/CD...). Maintenance This issue is related to maintenance work.

Comments

@Remi-Gau
Copy link
Collaborator

Those usually look like this:

On windows

https://github.com/nilearn/nilearn/actions/runs/5397326834/jobs/9801905802#step:8:4804

CI LOG details 
Windows fatal exception: access violation

Thread 0x00000bb0 (most recent call first):

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\kaleido\scopes\base.py", line 149 in _collect_standard_error

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 975 in run

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 1038 in _bootstrap_inner

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 995 in _bootstrap


Thread 0x00001898 (most recent call first):

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 320 in wait

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\externals\loky\backend\queues.py", line 113 in _feed

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 975 in run

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 1038 in _bootstrap_inner

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 995 in _bootstrap


Thread 0x00001144 (most recent call first):

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\multiprocessing\connection.py", line 810 in _exhaustive_wait

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\multiprocessing\connection.py", line 878 in wait

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\externals\loky\process_executor.py", line 611 in wait_result_broken_or_wakeup

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\externals\loky\process_executor.py", line 557 in run

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 1038 in _bootstrap_inner

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\threading.py", line 995 in _bootstrap


Current thread 0x00001b48 (most recent call first):

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\scipy\linalg_decomp_lu.py", line 313 in lu

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\sklearn\utils\extmath.py", line 274 in randomized_range_finder

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\sklearn\utils\extmath.py", line 446 in randomized_svd

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\memory.py", line 349 in call

File "D:\a\nilearn\nilearn\nilearn\decomposition_multi_pca.py", line 192 in _raw_fit

File "D:\a\nilearn\nilearn\nilearn\regions\parcellations.py", line 354 in _raw_fit

File "D:\a\nilearn\nilearn\nilearn\decomposition_base.py", line 452 in fit

File "D:\a\nilearn\nilearn\nilearn\regions\tests\test_parcellations.py", line 171 in test_parcellations_transform_multi_nifti_images

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\python.py", line 194 in pytest_pyfunc_call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_callers.py", line 80 in _multicall

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_manager.py", line 112 in _hookexec

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_hooks.py", line 433 in call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\python.py", line 1788 in runtest

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 169 in pytest_runtest_call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_callers.py", line 80 in _multicall

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_manager.py", line 112 in _hookexec

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_hooks.py", line 433 in call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 262 in 

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 341 in from_call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 261 in call_runtest_hook

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 222 in call_and_report

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 133 in runtestprotocol

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\runner.py", line 114 in pytest_runtest_protocol

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_callers.py", line 80 in _multicall

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_manager.py", line 112 in _hookexec

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_hooks.py", line 433 in call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\main.py", line 349 in pytest_runtestloop

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_callers.py", line 80 in _multicall

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_manager.py", line 112 in _hookexec

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_hooks.py", line 433 in call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\main.py", line 324 in main

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\main.py", line 270 in wrap_session

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\main.py", line 317 in pytest_cmdline_main

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_callers.py", line 80 in multicall

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_manager.py", line 112 in hookexec

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pluggy_hooks.py", line 433 in call

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\config_init.py", line 166 in main

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages_pytest\config_init.py", line 189 in console_main

File "C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\pytest_main.py", line 5 in 

File "", line 88 in _run_code

File "", line 198 in _run_module_as_main

C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\externals\loky\backend\resource_tracker.py:310: UserWarning: resource_tracker: There appear to be 2 leaked folder objects to clean up at shutdown

warnings.warn(

C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\externals\loky\backend\resource_tracker.py:326: UserWarning: resource_tracker: C:\Users\RUNNER1\AppData\Local\Temp\joblib_memmapping_folder_5404_994950f8cdb349039ec2d2cdb9671787_c363107d49404926b23b4dc2726c7bea: FileNotFoundError(2, 'The system cannot find the path specified')

warnings.warn(f'resource_tracker: {name}: {e!r}')

C:\hostedtoolcache\windows\Python\3.11.4\x64\Lib\site-packages\joblib\externals\loky\backend\resource_tracker.py:326: UserWarning: resource_tracker: C:\Users\RUNNER1\AppData\Local\Temp\joblib_memmapping_folder_5404_994950f8cdb349039ec2d2cdb9671787_3dd7ffe9a40d4d63902a1c80026a0cc1: FileNotFoundError(2, 'The system cannot find the path specified')

warnings.warn(f'resource_tracker: {name}: {e!r}')

./build_tools/github/test.sh: line 3:  2000 Segmentation fault      python -m pytest --pyargs nilearn --cov=nilearn

On Mac

https://github.com/nilearn/nilearn/actions/runs/5394020534/jobs/9794597362#step:8:4596

CI LOG details 
Fatal Python error: Segmentation fault


Thread 0x0000700006359000 (most recent call first):

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/kaleido/scopes/base.py", line 149 in _collect_standard_error

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 917 in run

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 980 in _bootstrap_inner

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 937 in _bootstrap


Thread 0x0000700004350000 (most recent call first):

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 312 in wait

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/backend/queues.py", line 113 in _feed

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 917 in run

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 980 in _bootstrap_inner

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 937 in _bootstrap


Thread 0x000070000334d000 (most recent call first):

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/selectors.py", line 416 in select

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/multiprocessing/connection.py", line 931 in wait

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line 611 in wait_result_broken_or_wakeup

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line 557 in run

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 980 in _bootstrap_inner

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/threading.py", line 937 in _bootstrap


Current thread 0x000000011543a600 (most recent call first):

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/scipy/linalg/_decomp_lu.py", line 313 in lu

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/sklearn/utils/extmath.py", line 274 in randomized_range_finder

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/sklearn/utils/extmath.py", line 446 in randomized_svd

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/memory.py", line 349 in call

File "/Users/runner/work/nilearn/nilearn/nilearn/decomposition/_multi_pca.py", line 192 in _raw_fit

File "/Users/runner/work/nilearn/nilearn/nilearn/regions/parcellations.py", line 354 in _raw_fit

File "/Users/runner/work/nilearn/nilearn/nilearn/decomposition/_base.py", line 452 in fit

File "/Users/runner/work/nilearn/nilearn/nilearn/regions/tests/test_parcellations.py", line 143 in test_parcellations_transform_single_nifti_image

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_callers.py", line 80 in _multicall

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_manager.py", line 112 in _hookexec

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_hooks.py", line 433 in call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/python.py", line 1788 in runtest

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_callers.py", line 80 in _multicall

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_manager.py", line 112 in _hookexec

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_hooks.py", line 433 in call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 262 in 

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 341 in from_call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 261 in call_runtest_hook

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 222 in call_and_report

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 133 in runtestprotocol

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_callers.py", line 80 in _multicall

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_manager.py", line 112 in _hookexec

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_hooks.py", line 433 in call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_runtestloop

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_callers.py", line 80 in _multicall

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_manager.py", line 112 in _hookexec

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_hooks.py", line 433 in call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/main.py", line 324 in _main

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/main.py", line 270 in wrap_session

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_callers.py", line 80 in _multicall

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_manager.py", line 112 in _hookexec

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pluggy/_hooks.py", line 433 in call

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/config/init.py", line 166 in main

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/_pytest/config/init.py", line 189 in console_main

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pytest/main.py", line 5 in 

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/runpy.py", line 87 in _run_code

File "/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/runpy.py", line 197 in _run_module_as_main

./build_tools/github/test.sh: line 3:  6267 Segmentation fault: 11  python -m pytest --pyargs nilearn --cov=nilearn

/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/backend/resource_tracker.py:310: UserWarning: resource_tracker: There appear to be 6 leaked semlock objects to clean up at shutdown

warnings.warn(

/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/backend/resource_tracker.py:310: UserWarning: resource_tracker: There appear to be 2 leaked folder objects to clean up at shutdown

warnings.warn(

/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/backend/resource_tracker.py:326: UserWarning: resource_tracker: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/joblib_memmapping_folder_6267_25153dc464584420b21761453a3d8c03_ef28899160f84d26b3974c3daa98f33c: FileNotFoundError(2, 'No such file or directory')

warnings.warn(f'resource_tracker: {name}: {e!r}')

/Users/runner/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/joblib/externals/loky/backend/resource_tracker.py:326: UserWarning: resource_tracker: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/joblib_memmapping_folder_6267_25153dc464584420b21761453a3d8c03_2e58b608bfd643aa88ca9f9987b4c271: FileNotFoundError(2, 'No such file or directory')

warnings.warn(f'resource_tracker: {name}: {e!r}')

Not made a systematic check but they seem to happen to only happen for python > 3.8.
@Remi-Gau
Copy link
Collaborator Author

seems to be occuring around those tests:

  • nilearn/regions/tests/test_parcellations.py::test_parcellations_fit_on_multi_nifti_images[hierarchical_kmeans]
  • nilearn/regions/tests/test_parcellations.py::test_parcellations_transform_multi_nifti_images[5-kmeans]

@Remi-Gau Remi-Gau mentioned this issue Jun 28, 2023
Merged
@Remi-Gau Remi-Gau added Maintenance This issue is related to maintenance work. Infrastructure This issue describes a problem with the project's infrastructure (CI/CD...). labels Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Infrastructure This issue describes a problem with the project's infrastructure (CI/CD...). Maintenance This issue is related to maintenance work.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[FIX] address flaky tests in CI Remi-Gau/nilearn
1 participant
@Remi-Gau