Hey @yadudoc, thanks for raising this. It seems like the error source is from the job submission layer (saga/PSI-J).

I am on it and I will try to reproduce it if possible.

since some recent changes that hopefully fixed other CI failures, it looks like this error is now the dominant cause of CI test failures that I am experiencing.

(this has affected every PR I've tried to merge or review today)

@benclifford I am preparing a PR that fixes this in addition to several fixes by today.

@yadudoc , @benclifford, a PR to fix this issue and others is open now #3060.

@AymenFJA 's fix PR #3060 didn't satisfy my "why is this broken / how does this fix it?" curiosity, so I dug in a bit today. I've identified these issues which interact here, I think, in no particular order:

• radical.saga PTYProcess misbehaves when more than FD_SETSIZE (I think) files are open: file descriptors returned by fork cannot always be used with select, resulting in base error message that is reported in this issue Test failures from Radical pilot executor #3029: ValueError: filedescriptor out of range in select() means the given file descriptor is bigger than FD_SETSIZE despite otherwise being valid (I think).

• some part of the pytest stack hangs at exit when the above error occurs during a test, rather than exiting cleanly. I can recreate this hang by putting in a deliberate exception like this:

                     # check if the pty pipe is ready for data
                    raise RuntimeError("BENC: FAIL AT SELECT")
                    _, wlist, _ = select.select ([], [self.parent_in], [],
                                                 _POLLDELAY)

... and then running pytest parsl/tests/test_radical/test_mpi_funcs.py --config local

• parsl tests in --config local mode are accumulating around 10 file descriptors per test: so it's pretty easy for 1024 open FDs to be reached, if the radical test happens to run late in the randomised order. On my laptop I've seen other fd exhaustion errors, in a ulimit constrained environment, distinct from this Test failures from Radical pilot executor #3029 radical hang. I've documented this more in issue --config local tests (and so probably parsl used without immediately exiting) accumulates many open fds. #3077

I did some more hacking for the hang described about as "some part of the pytest stack hangs at exit": the stack is waiting for a process launched in radical.pilot/proxy.py to exit before the whole Python process will exit (as that is how multiprocessing processes work) -- however it looks like that process does not exit in this situation:

I modified my Python runtime so that it reports the name of processes that it waits for at exit, and I modified radical.pilot to add a name= on all processes, and I see this at the end of the hanging test when I'd expect pytest to exit.

        proc = mp.Process(target=self._worker, args=(sid, q, term, self._path), name="BENC: proxy")

and then:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================ 2 warnings, 1 error in 8.06s ================================================
calling join() for process BENC: proxy

If I set daemon=True on that proxy process, pytest exits without hanging, but I have not tried to understand if that is a safe thing for proxy.py to do - it was just to understand if that was the only hang.

none of the above gives any explanation for what works (or does not work) about Aymen's PR #3060 though... so I'll try to understand that.

adding safe-exit tag because this hang is also going to occur for real users in some situations

hypothesis on why this failure has become worse recently:

this test failure is happening when around a thousand file descriptors are open at the start of the radical test.

this test failure happens in pytest --config local because many tests are run before, all of which leave some fds (and other junk) around (see issue #3077)

this test failure happens non-deterministically because the tests are deliberately run in random order to flush out inter-test dependencies (introduced in massive test rearrangement PR #1210) - if this test runs "early" in the test run, the fd count will be "low" and the test will not fail. if this test runs "late" in the test run, the fd count will be "high" and the test will fail

recently a bunch more tests have been added to the --config local collection, for example driven by recent htex feature development.

adding more tests means that it's more likely that this radical test will run beyond the 1000-ish tests

so while this bug has always existed, it's not a change in the radical stack that has caused it to happen more, but instead a change in the test environment to be more fd-intensive.

Hi Ben,

thanks for deep-diving into this!

As Aymen said earlier, SAGA is on it's way out of RP and we'll likely not attempt to fix this on the SAGA layer. We still use it in some places as PSI/J does not jet support data staging. The PARSL tests triggering these kind of errors only add to the motivation to complete the transition though ;-) Let me have a look over the next few days to see if I can do a hackish SAGA replacement for the code path relevant to this.

PS: Setting daemon on the RP processes will avoid the hangs, but at the same time may result in zombie processes. Getting Python to clean up nicely when subprocesses and threads are mixed has been (an continues to be) a challenge unfortunately.

Quick note that this should be resolved with the next RPC release: the saga removal is completed, the respective code path simply does not exist anymore. Sorry it took so long, but it turned out that some refactoring and code removal ultimately was simpler (and more sensible) than fixing this.