While the original failures are gone, I am now seeing two new failures. As of now I am failing to see how they could be related to this PR, but I checked master in #6568 and failed to trigger them.
2022-03-19T09:42:42.5345180Z ======================================================================
2022-03-19T09:42:42.5345480Z ERROR: datalad.support.tests.test_parallel.test_creatsubdatasets
2022-03-19T09:42:42.5345990Z ----------------------------------------------------------------------
2022-03-19T09:42:42.5346280Z Traceback (most recent call last):
2022-03-19T09:42:42.5348780Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 368, in _iter_threads
2022-03-19T09:42:42.5349170Z raise _FinalShutdown()
2022-03-19T09:42:42.5349420Z datalad.support.parallel._FinalShutdown
2022-03-19T09:42:42.5349590Z
2022-03-19T09:42:42.5349760Z During handling of the above exception, another exception occurred:
2022-03-19T09:42:42.5349950Z
2022-03-19T09:42:42.5350040Z Traceback (most recent call last):
2022-03-19T09:42:42.5350630Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
2022-03-19T09:42:42.5350970Z self.test(*self.arg)
2022-03-19T09:42:42.5351550Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile
2022-03-19T09:42:42.5351910Z return t(*(arg + (filename,)), **kw)
2022-03-19T09:42:42.5352520Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/tests/test_parallel.py", line 127, in test_creatsubdatasets
2022-03-19T09:42:42.5353160Z assert_raises(IncompleteResultsError, list, ProducerConsumer(paths[::-1], create_, jobs=5))
2022-03-19T09:42:42.5353690Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/unittest/case.py", line 756, in assertRaises
2022-03-19T09:42:42.5354270Z return context.handle('assertRaises', args, kwargs)
2022-03-19T09:42:42.5354670Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/unittest/case.py", line 178, in handle
2022-03-19T09:42:42.5355500Z callable_obj(*args, **kwargs)
2022-03-19T09:42:42.5356100Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 265, in __iter__
2022-03-19T09:42:42.5356470Z yield from self._iter_threads(self._jobs)
2022-03-19T09:42:42.5357060Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 417, in _iter_threads
2022-03-19T09:42:42.5357490Z self.shutdown(force=True, exception=self._producer_exception or interrupted_by_exception)
2022-03-19T09:42:42.5358560Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 233, in shutdown
2022-03-19T09:42:42.5358940Z raise exception
2022-03-19T09:42:42.5359580Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 401, in _iter_threads
2022-03-19T09:42:42.5359950Z done_useful |= self._pop_done_futures(lgr)
2022-03-19T09:42:42.5360560Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 463, in _pop_done_futures
2022-03-19T09:42:42.5360890Z raise exception
2022-03-19T09:42:42.5361300Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2022-03-19T09:42:42.5361650Z result = self.fn(*self.args, **self.kwargs)
2022-03-19T09:42:42.5362270Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 329, in consumer_worker
2022-03-19T09:42:42.5362590Z for r in res:
2022-03-19T09:42:42.5363170Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/interface/utils.py", line 369, in generator_func
2022-03-19T09:42:42.5363500Z allkwargs):
2022-03-19T09:42:42.5364080Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/interface/utils.py", line 544, in _process_results
2022-03-19T09:42:42.5364420Z for res in results:
2022-03-19T09:42:42.5364950Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/core/local/create.py", line 376, in __call__
2022-03-19T09:42:42.5365320Z path, initopts, fake_dates, description)
2022-03-19T09:42:42.5365920Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/core/local/create.py", line 530, in _setup_annex_repo
2022-03-19T09:42:42.5366260Z fake_dates=fake_dates
2022-03-19T09:42:42.5366800Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/repo.py", line 152, in __call__
2022-03-19T09:42:42.5367160Z instance = type.__call__(cls, *new_args, **new_kwargs)
2022-03-19T09:42:42.5367740Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 205, in __init__
2022-03-19T09:42:42.5368110Z repo=repo, git_opts=git_opts, fake_dates=fake_dates)
2022-03-19T09:42:42.5368690Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/gitrepo.py", line 904, in __init__
2022-03-19T09:42:42.5369160Z init_options=from_cmdline + to_options(**git_opts),
2022-03-19T09:42:42.5369750Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/gitrepo.py", line 545, in init
2022-03-19T09:42:42.5370100Z pathobj.mkdir(parents=True)
2022-03-19T09:42:42.5370440Z File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/pathlib.py", line 1273, in mkdir
2022-03-19T09:42:42.5370800Z self._accessor.mkdir(self, mode)
2022-03-19T09:42:42.5371380Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsdb8n_dqd/subds1'
This failure reproduced on rerunning that build a couple of times and then stopped showing up when I switched the lexists(str) to use pathlib. However, no usage of pathlib (exists() or is_symlink(), try: lstat() except: ... and os.path.lexists(pathlib.Path)) seems to be a plausible fix (but none of them seemed to trigger the issue).
There's a flaky test failure, I first observed in PR #6550. I'm pretty sure it's unrelated to that PR and it doesn't easily reproduce. However, similar things happen elsewhere so I want to have it on record here, as I suspect there's a general issue with respect to using
ProducerConsumer(at east for some usecases).Extracted from #6550 (comment):
While the original failures are gone, I am now seeing two new failures. As of now I am failing to see how they could be related to this PR, but I checked
masterin #6568 and failed to trigger them.One exclusively on github action macOS(brew) only:
Which seems wild, since the
mkdircall apparently causing theFileExistsErroris is done right after testing forlexiststo beFalse.This failure reproduced on rerunning that build a couple of times and then stopped showing up when I switched the
lexists(str)to usepathlib. However, no usage ofpathlib(exists() or is_symlink(),try: lstat() except: ...andos.path.lexists(pathlib.Path)) seems to be a plausible fix (but none of them seemed to trigger the issue).