CI: Rebase NumPy compiled extension test modules on Cygwin by DWesl · Pull Request #23073 · numpy/numpy

DWesl · 2023-01-23T12:01:46Z

This seems to be causing fork failures on Cygwin. Let's see if this reduces the number of failures.

DWesl · 2023-01-23T16:41:54Z

Four hours to see if all extension modules can find their dependencies. That step really shouldn't take longer than the test that loads all the modules, and definitely not more than twice as long.

--ignore led to more failures, which is not ideal.

DWesl · 2023-01-24T00:23:37Z

Six hundred errors on the last two attempts, which is less than ideal given that the original problem was eight. (EDIT: 8-41 in the last few PRs)

Let's see if this shows why everything's got a fork() failure now.

It should be plenty of time; each of these commands should complete in maybe five seconds per file on a slow day.

Let's see if this fixes the 8-50 fork failures.

seberg · 2023-01-27T11:07:04Z

On the cygwin help it still mentions to check for bad "bloda" software running (mostly anti-virus stuff), no idea if that can help...

But besides that, I am wondering if we have to opt for some hot-fix to get CI green again soon.

DWesl · 2023-01-27T13:38:28Z

On the cygwin help it still mentions to check for bad "bloda" software running (mostly anti-virus stuff), no idea if that can help...

It can, but I don't see anything obvious:
https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md

But besides that, I am wondering if we have to opt for some hot-fix to get CI green again soon.

pytestmark = pytest.mark.xfail(sys.platform == "cygwin", message="Random fork() failures", error=BlockingIOError)

at module level in a couple of places should work to get CI green, but I haven't tested that.

Forgot to check this earlier.

Hopefully this will keep the main memory space clear enough to allow all tests to succeed.

This should help debugging a bit.

DWesl · 2023-01-27T19:47:49Z

The equivalent of -j 4 may separate the extension modules enough to work without the overhead of --forked (almost five hours to run just the F2Py tests seems like too much).

Hopefully this helps, since --forked takes almost five hours.

This reverts commit 6088646.

This reverts commit 33709af.

Let's see if this eliminates the fork failures. It's back to around where it was before I started tinkering, so this approach may not work.

Tests hang, which is less than ideal. Hopefully this one works well.

DWesl · 2023-01-29T16:23:27Z

The tests for every submodule but f2py are working, it's just that last submodule causing problems. We could drop those tests from CI runs, or I could port over the F2Py xfail list from #23114 and see if the combination allows CI to pass.

DWesl · 2023-01-29T16:39:45Z

On the cygwin help it still mentions to check for bad "bloda" software running (mostly anti-virus stuff), no idea if that can help...

It can, but I don't see anything obvious: https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md

I found the image description:
https://github.com/actions/runner-images/blob/win22/20230123.1/images/win/Windows2022-Readme.md
The changes on Jan 24 (vcpkg, Kubectl, and Pulumi are the ones that catch my attention, the others don't look relevant) don't have obvious antivirus or DLL-handling changes.
The changes on Jan 19 (Vcpkg and Pester are the ones I don't know) also doesn't show anything obvious.
The Jan 12 changes are probably too far back to be relevant to this, since it got noticed on Jan 22

But besides that, I am wondering if we have to opt for some hot-fix to get CI green again soon.
pytestmark = pytest.mark.xfail(sys.platform == "cygwin", message="Random fork() failures", error=BlockingIOError)
at module level in a couple of places should work to get CI green, but I haven't tested that.

#23114 now up: it takes a few more places than I thought it would if running all the tests together.

Also adjust CI so they don't immediately collide with NumPy. I forgot to do that last time, which caused problems.

DWesl · 2023-01-29T18:18:40Z

Or write the locations of the NumPy extension modules to the rebase database so the F2Py tests don't put their modules right on top of those, guaranteeing fork() failures.

mattip

LGTM, unbreaks the cygwin build. I am a little concerned about listing the non-f2py directories to test, but maybe this is good enough for now?

mattip · 2023-01-30T04:53:55Z

.github/workflows/cygwin.yml

+        # Not sure if that will run the top-level tests.
+        shell: "C:\\tools\\cygwin\\bin\\bash.exe -o igncr -eo pipefail {0}"
+        run: |
+          for submodule in array_api compat core distutils fft lib linalg ma matrixlib polynomial random tests typing;


I am not sure what is the best path forward here:

a list of directories to use, which will break the next time we add/remove a top-level directory

some ugly path parsing like find numpy -maxdepth 1 -type d | cut -f2 -d/ | grep -v f2py | grep -v numpy | grep -v "^_" to get all the directories except f2py

add a command line switch to runtests.py to allow skipping a subdirectory

move the build to meson, and add a command line option to dev.py to skip a subdirectory

What you have is the simplest, but will be fragile. On the other hand, how often do we change the top-level directories?

Maybe is there a way to verify that your list is complete with a helper script?

I think this might work runtests.py ... -- --deselect f2py

for name in numpy/*; do if [ -d "${name}" -a "${name}" != numpy/f2py -a "${name:6:1}" != "_" ] ; then echo ${name##numpy/}; fi; done

or

find numpy/* -maxdepth 0 -type d -a ! -name f2py -a ! -name _\* | cut -f2 -d/

would also work. I think a helper script to check the list is complete would need the same path parsing.

I don't think this separation is needed anymore, so I'll go back to the simple invocation and see if that works.

All submodule tests run at the same time again; PR title changed to match.

Ideally this works nicely and I can change the PR name. If not, I put the split back, then the parallelization if that still doesn't work.

numpy/f2py/tests/util.py

This assumes NumPy is rebased before tests run, but does not assume the locations are in the database.

seberg

This looks OK to me, not super pretty, but also reasonably little additional code in the test.

Thanks for so valiantly tracking this down!

mattip · 2023-01-30T17:21:58Z

Thanks @DWesl

CI: Split up NumPy compiled extension test modules

acad7e0

This seems to be causing fork failures on Cygwin. Let's see if this reduces the number of failures.

DWesl added 2 commits January 23, 2023 17:15

CI: Use -k to split test cases instead of --ignore

b9b30e8

--ignore led to more failures, which is not ideal.

CI: Split tests by submodule to see if that works.

54879be

DWesl added 2 commits January 23, 2023 19:25

FIX: Make sure CI fails if tests fail.

d7dd060

CI: Print more debug information while rebasing.

37c1a99

Let's see if this shows why everything's got a fork() failure now.

DWesl mentioned this pull request Jan 26, 2023

Cygwin CI job fails with f2py errors #23070

Closed

DWesl added 2 commits January 26, 2023 08:29

CI: Put timeouts on Cygwin dependency checks.

3222962

It should be plenty of time; each of these commands should complete in maybe five seconds per file on a slow day.

TST: Rebase F2Py test modules on Cygwin.

6088646

Let's see if this fixes the 8-50 fork failures.

FIX: Add glob import for test module rebase.

33709af

Forgot to check this earlier.

DWesl mentioned this pull request Jan 27, 2023

TST: Mark F2PY tests as XFail on Cygwin #23114

Closed

DWesl added 2 commits January 27, 2023 12:19

CI: Run each F2Py test in a separate process

5b5b0a8

Hopefully this will keep the main memory space clear enough to allow all tests to succeed.

CI: Split F2Py and non-F2Py tests again

4e29fde

This should help debugging a bit.

DWesl added 4 commits January 27, 2023 19:02

CI: Run F2Py tests in parallel on Cygwin

3437ec4

Hopefully this helps, since --forked takes almost five hours.

Revert "TST: Rebase F2Py test modules on Cygwin."

015ecf6

This reverts commit 6088646.

Revert "FIX: Add glob import for test module rebase."

dbeaf07

This reverts commit 33709af.

CI: Increase number of processes for F2Py tests

38f43f7

Let's see if this eliminates the fork failures. It's back to around where it was before I started tinkering, so this approach may not work.

melissawm mentioned this pull request Jan 28, 2023

Feature request: PR comments with build preview website link napari/docs#98

Closed

CI: Revert increase in parallel test processes.

83f62a9

Tests hang, which is less than ideal. Hopefully this one works well.

TST: Rebase F2Py-built extension modules.

2fa4441

Also adjust CI so they don't immediately collide with NumPy. I forgot to do that last time, which caused problems.

DWesl mentioned this pull request Jan 29, 2023

BUG: fix broken numpy.distutils Fortran handling #23066

Merged

mattip reviewed Jan 30, 2023

View reviewed changes

CI: Unsplit the Cygwin tests.

bb1b212

Ideally this works nicely and I can change the PR name. If not, I put the split back, then the parallelization if that still doesn't work.

DWesl commented Jan 30, 2023

View reviewed changes

numpy/f2py/tests/util.py Outdated Show resolved Hide resolved

CI: Rebase numpy DLLs in runtests.py.

2293a62

This assumes NumPy is rebased before tests run, but does not assume the locations are in the database.

DWesl changed the title ~~CI: Split up NumPy compiled extension test modules~~ CI: Rebase NumPy compiled extension test modules Jan 30, 2023

DWesl changed the title ~~CI: Rebase NumPy compiled extension test modules~~ CI: Rebase NumPy compiled extension test modules on Cygwin Jan 30, 2023

seberg approved these changes Jan 30, 2023

View reviewed changes

mattip merged commit 9d2c019 into numpy:main Jan 30, 2023

DWesl deleted the patch-2 branch January 8, 2025 18:45

Uh oh!

Conversation

DWesl commented Jan 23, 2023

Uh oh!

DWesl commented Jan 23, 2023

Uh oh!

DWesl commented Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Jan 27, 2023

Uh oh!

DWesl commented Jan 27, 2023

Uh oh!

DWesl commented Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DWesl commented Jan 29, 2023

Uh oh!

DWesl commented Jan 29, 2023

Uh oh!

DWesl commented Jan 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip left a comment

Choose a reason for hiding this comment

Uh oh!

mattip Jan 30, 2023

Choose a reason for hiding this comment

Uh oh!

mattip Jan 30, 2023

Choose a reason for hiding this comment

Uh oh!

DWesl Jan 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DWesl Jan 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

mattip commented Jan 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DWesl commented Jan 24, 2023 •

edited

Loading

DWesl commented Jan 27, 2023 •

edited

Loading

DWesl commented Jan 29, 2023 •

edited

Loading

DWesl Jan 30, 2023 •

edited

Loading