Skip to content

Possible race condition in datalad.support.tests.test_parallel.test_creatsubdatasets #6598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bpoldrack opened this issue Mar 30, 2022 · 3 comments · Fixed by #7075
Closed

Possible race condition in datalad.support.tests.test_parallel.test_creatsubdatasets #6598

bpoldrack opened this issue Mar 30, 2022 · 3 comments · Fixed by #7075
Labels
fix-implemented A fix is available, but has not been merged or released, yet.

Comments

@bpoldrack
Copy link
Member

There's a flaky test failure, I first observed in PR #6550. I'm pretty sure it's unrelated to that PR and it doesn't easily reproduce. However, similar things happen elsewhere so I want to have it on record here, as I suspect there's a general issue with respect to using ProducerConsumer (at east for some usecases).

Extracted from #6550 (comment):

While the original failures are gone, I am now seeing two new failures. As of now I am failing to see how they could be related to this PR, but I checked master in #6568 and failed to trigger them.

One exclusively on github action macOS(brew) only:

2022-03-19T09:42:42.5345180Z ======================================================================
2022-03-19T09:42:42.5345480Z ERROR: datalad.support.tests.test_parallel.test_creatsubdatasets
2022-03-19T09:42:42.5345990Z ----------------------------------------------------------------------
2022-03-19T09:42:42.5346280Z Traceback (most recent call last):
2022-03-19T09:42:42.5348780Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 368, in _iter_threads
2022-03-19T09:42:42.5349170Z     raise _FinalShutdown()
2022-03-19T09:42:42.5349420Z datalad.support.parallel._FinalShutdown
2022-03-19T09:42:42.5349590Z 
2022-03-19T09:42:42.5349760Z During handling of the above exception, another exception occurred:
2022-03-19T09:42:42.5349950Z 
2022-03-19T09:42:42.5350040Z Traceback (most recent call last):
2022-03-19T09:42:42.5350630Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
2022-03-19T09:42:42.5350970Z     self.test(*self.arg)
2022-03-19T09:42:42.5351550Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile
2022-03-19T09:42:42.5351910Z     return t(*(arg + (filename,)), **kw)
2022-03-19T09:42:42.5352520Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/tests/test_parallel.py", line 127, in test_creatsubdatasets
2022-03-19T09:42:42.5353160Z     assert_raises(IncompleteResultsError, list, ProducerConsumer(paths[::-1], create_, jobs=5))
2022-03-19T09:42:42.5353690Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/unittest/case.py", line 756, in assertRaises
2022-03-19T09:42:42.5354270Z     return context.handle('assertRaises', args, kwargs)
2022-03-19T09:42:42.5354670Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/unittest/case.py", line 178, in handle
2022-03-19T09:42:42.5355500Z     callable_obj(*args, **kwargs)
2022-03-19T09:42:42.5356100Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 265, in __iter__
2022-03-19T09:42:42.5356470Z     yield from self._iter_threads(self._jobs)
2022-03-19T09:42:42.5357060Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 417, in _iter_threads
2022-03-19T09:42:42.5357490Z     self.shutdown(force=True, exception=self._producer_exception or interrupted_by_exception)
2022-03-19T09:42:42.5358560Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 233, in shutdown
2022-03-19T09:42:42.5358940Z     raise exception
2022-03-19T09:42:42.5359580Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 401, in _iter_threads
2022-03-19T09:42:42.5359950Z     done_useful |= self._pop_done_futures(lgr)
2022-03-19T09:42:42.5360560Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 463, in _pop_done_futures
2022-03-19T09:42:42.5360890Z     raise exception
2022-03-19T09:42:42.5361300Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2022-03-19T09:42:42.5361650Z     result = self.fn(*self.args, **self.kwargs)
2022-03-19T09:42:42.5362270Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/parallel.py", line 329, in consumer_worker
2022-03-19T09:42:42.5362590Z     for r in res:
2022-03-19T09:42:42.5363170Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/interface/utils.py", line 369, in generator_func
2022-03-19T09:42:42.5363500Z     allkwargs):
2022-03-19T09:42:42.5364080Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/interface/utils.py", line 544, in _process_results
2022-03-19T09:42:42.5364420Z     for res in results:
2022-03-19T09:42:42.5364950Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/core/local/create.py", line 376, in __call__
2022-03-19T09:42:42.5365320Z     path, initopts, fake_dates, description)
2022-03-19T09:42:42.5365920Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/core/local/create.py", line 530, in _setup_annex_repo
2022-03-19T09:42:42.5366260Z     fake_dates=fake_dates
2022-03-19T09:42:42.5366800Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/repo.py", line 152, in __call__
2022-03-19T09:42:42.5367160Z     instance = type.__call__(cls, *new_args, **new_kwargs)
2022-03-19T09:42:42.5367740Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/annexrepo.py", line 205, in __init__
2022-03-19T09:42:42.5368110Z     repo=repo, git_opts=git_opts, fake_dates=fake_dates)
2022-03-19T09:42:42.5368690Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/support/gitrepo.py", line 904, in __init__
2022-03-19T09:42:42.5369160Z     init_options=from_cmdline + to_options(**git_opts),
2022-03-19T09:42:42.5369750Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/gitrepo.py", line 545, in init
2022-03-19T09:42:42.5370100Z     pathobj.mkdir(parents=True)
2022-03-19T09:42:42.5370440Z   File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/pathlib.py", line 1273, in mkdir
2022-03-19T09:42:42.5370800Z     self._accessor.mkdir(self, mode)
2022-03-19T09:42:42.5371380Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsdb8n_dqd/subds1'

Which seems wild, since the mkdir call apparently causing the FileExistsError is is done right after testing for lexists to be False.

This failure reproduced on rerunning that build a couple of times and then stopped showing up when I switched the lexists(str) to use pathlib. However, no usage of pathlib (exists() or is_symlink(), try: lstat() except: ... and os.path.lexists(pathlib.Path)) seems to be a plausible fix (but none of them seemed to trigger the issue).
@bpoldrack bpoldrack mentioned this issue Mar 30, 2022
Merged
1 task
@yarikoptic
Copy link
Member

yarikoptic commented Oct 7, 2022 •
edited
Loading

another issue I searched up is #5402 which was "let it RiP until we awaken it from the dead" so here it came ;)

but it is quite common indeed with first seems failing in #6291

$> datalad foreach-dataset --o-s relpath -J 3 -r git grep 'FileExistsError: .*File exists: .*/datalad_temp_test_creatsubdatasets'
2022/01/27/pr/6291/45dd580/github-Test on macOS-2660-failed/1_test (brew).txt:2022-01-27T14:32:24.8163230Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsrjny10lo/subds1'
2022/01/27/pr/6291/45dd580/github-Test on macOS-2660-failed/test (brew)/8_Run tests.txt:2022-01-27T14:32:24.8163220Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsrjny10lo/subds1'
2022/01/27/pr/UNK/cd80cf5/github-Test on macOS-2662-failed/1_test (brew).txt:2022-01-27T15:01:27.6833020Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsloi2x71j/subds1'
2022/01/27/pr/UNK/cd80cf5/github-Test on macOS-2662-failed/test (brew)/8_Run tests.txt:2022-01-27T15:01:27.6833010Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsloi2x71j/subds1'
2022/01/28/pr/6380/0a135a2/github-Test on macOS-2677-failed/1_test (brew).txt:2022-01-28T09:39:52.6425850Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsshzg6oi3/subds1'
2022/01/28/pr/6380/0a135a2/github-Test on macOS-2677-failed/test (brew)/8_Run tests.txt:2022-01-28T09:39:52.6425840Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsshzg6oi3/subds1'
2022/01/28/pr/6380/f372d76/github-Test on macOS-2676-failed/2_test (snapshot).txt:2022-01-28T09:47:33.3196010Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets88a6a4et/subds1'
2022/01/28/pr/6380/f372d76/github-Test on macOS-2676-failed/test (snapshot)/8_Run tests.txt:2022-01-28T09:47:33.3195820Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets88a6a4et/subds1'
2022/01/28/pr/6387/7d24340/github-Test on macOS-2696-failed/1_test (brew).txt:2022-01-28T15:10:02.6588460Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetskv7t0clb/subds1'
2022/01/28/pr/6387/7d24340/github-Test on macOS-2696-failed/test (brew)/8_Run tests.txt:2022-01-28T15:10:02.6588450Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetskv7t0clb/subds1'
2022/02/01/push/master/c2183b3/github-Test on macOS-2710-failed/1_test (brew).txt:2022-02-01T06:53:11.8338810Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets5_zepzff/subds1'
2022/02/01/push/master/c2183b3/github-Test on macOS-2710-failed/test (brew)/8_Run tests.txt:2022-02-01T06:53:11.8338810Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets5_zepzff/subds1'
2022/02/10/push/master/dfdc184/github-Test on macOS-2799-failed/1_test (brew).txt:2022-02-10T15:54:15.6263540Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetskfr825j8/subds1'
2022/02/10/push/master/dfdc184/github-Test on macOS-2799-failed/test (brew)/8_Run tests.txt:2022-02-10T15:54:15.6263530Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetskfr825j8/subds1'
2022/02/11/push/bf-askpass/d3fcf7b/github-Test on macOS-2805-failed/2_test (snapshot).txt:2022-02-11T20:18:48.3712460Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsem0dxfrs/subds1'
2022/02/11/push/bf-askpass/d3fcf7b/github-Test on macOS-2805-failed/test (snapshot)/8_Run tests.txt:2022-02-11T20:18:48.3712450Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsem0dxfrs/subds1'
2022/02/15/push/master/5e8cf86/github-Test on macOS-2817-failed/1_test (brew).txt:2022-02-15T22:22:55.6041840Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets8c_p4v9r/subds1'
2022/02/15/push/master/5e8cf86/github-Test on macOS-2817-failed/test (brew)/8_Run tests.txt:2022-02-15T22:22:55.6041830Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets8c_p4v9r/subds1'
2022/02/18/pr/6462/d72db8d/github-Test on macOS-2827-failed/1_test (brew).txt:2022-02-18T07:42:54.1562420Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsgbjz77qd/subds1'
2022/02/18/pr/6462/d72db8d/github-Test on macOS-2827-failed/test (brew)/8_Run tests.txt:2022-02-18T07:42:54.1562410Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsgbjz77qd/subds1'
2022/02/18/pr/6472/ab4d3a6/github-Test on macOS-2840-failed/1_test (brew).txt:2022-02-18T14:46:01.1101560Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsptt13ho0/subds1'
2022/02/18/pr/6472/ab4d3a6/github-Test on macOS-2840-failed/test (brew)/8_Run tests.txt:2022-02-18T14:46:01.1101550Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsptt13ho0/subds1'
2022/02/23/pr/6493/5c23d95/github-Test on macOS-2877-failed/2_test (snapshot).txt:2022-02-23T09:49:12.3803680Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsyxu3u278/subds1'
2022/02/23/pr/6493/5c23d95/github-Test on macOS-2877-failed/test (snapshot)/8_Run tests.txt:2022-02-23T09:49:12.3803670Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsyxu3u278/subds1'
2022/02/23/pr/6495/c9dec43/github-Test on macOS-2883-failed/1_test (brew).txt:2022-02-23T17:07:49.9502920Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsp9poahs6/subds1'
2022/02/23/pr/6495/c9dec43/github-Test on macOS-2883-failed/test (brew)/8_Run tests.txt:2022-02-23T17:07:49.9502910Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsp9poahs6/subds1'
2022/02/23/pr/UNK/b14a902/github-Test on macOS-2875-failed/1_test (brew).txt:2022-02-23T07:31:15.7727620Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets4kduquif/subds1'
2022/02/23/pr/UNK/b14a902/github-Test on macOS-2875-failed/test (brew)/8_Run tests.txt:2022-02-23T07:31:15.7727610Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets4kduquif/subds1'
2022/03/15/pr/UNK/bc34a9d/github-Test on macOS-2987-failed/1_test (brew).txt:2022-03-15T13:30:07.7718300Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetswfrh_a_v/subds1'
2022/03/15/pr/UNK/bc34a9d/github-Test on macOS-2987-failed/test (brew)/8_Run tests.txt:2022-03-15T13:30:07.7718290Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetswfrh_a_v/subds1'
2022/03/19/pr/6550/140afc0/github-Test on macOS-3032-failed/1_test (brew).txt:2022-03-19T09:42:42.5371380Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsdb8n_dqd/subds1'
2022/03/19/pr/6550/140afc0/github-Test on macOS-3032-failed/test (brew)/8_Run tests.txt:2022-03-19T09:42:42.5371370Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsdb8n_dqd/subds1'
2022/03/21/pr/UNK/0fadd7c/github-Test on macOS-3040-failed/1_test (brew).txt:2022-03-21T10:06:09.2381310Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsa37e6w_k/subds1'
2022/03/21/pr/UNK/0fadd7c/github-Test on macOS-3040-failed/test (brew)/8_Run tests.txt:2022-03-21T10:06:09.2381300Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsa37e6w_k/subds1'
2022/03/27/pr/6535/acf586f/github-Test on macOS-3073-failed/1_test (brew).txt:2022-03-27T15:34:26.6373730Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetshlccskjh/subds1'
2022/03/27/pr/6535/acf586f/github-Test on macOS-3073-failed/test (brew)/8_Run tests.txt:2022-03-27T15:34:26.6373730Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetshlccskjh/subds1'
2022/03/30/pr/6600/f2befc9/github-Test on macOS-3087-failed/1_test (brew).txt:2022-03-30T12:07:55.3167420Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets4_5q0dw0/subds1'
2022/03/30/pr/6600/f2befc9/github-Test on macOS-3087-failed/test (brew)/8_Run tests.txt:2022-03-30T12:07:55.3167410Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets4_5q0dw0/subds1'
2022/03/30/pr/UNK/3c56897/github-Test on macOS-3084-failed/1_test (brew).txt:2022-03-30T10:04:54.2541500Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsbyms0rrm/subds1'
2022/03/30/pr/UNK/3c56897/github-Test on macOS-3084-failed/test (brew)/8_Run tests.txt:2022-03-30T10:04:54.2541500Z FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsbyms0rrm/subds1'
2022/05/09/pr/6672/d923216/github-Test on macOS-3255-failed/1_test (brew).txt:2022-05-09T09:13:08.4264810Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets5lb33534/subds1'
2022/05/09/pr/6672/d923216/github-Test on macOS-3255-failed/test (brew)/8_Run tests.txt:2022-05-09T09:13:08.4264800Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets5lb33534/subds1'
2022/05/10/pr/6690/80a85d7/github-Test on macOS-3264-failed/1_test (brew).txt:2022-05-10T12:25:35.3165400Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsh_qe7byo/subds1'
2022/05/10/pr/6690/80a85d7/github-Test on macOS-3264-failed/test (brew)/8_Run tests.txt:2022-05-10T12:25:35.3165390Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsh_qe7byo/subds1'
2022/05/18/pr/6690/891956e/github-Test on macOS-3271-failed/1_test (brew).txt:2022-05-18T10:38:23.8710990Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets5zh5rrgf/subds1'
2022/05/18/pr/6690/891956e/github-Test on macOS-3271-failed/test (brew)/8_Run tests.txt:2022-05-18T10:38:23.8710980Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets5zh5rrgf/subds1'
2022/05/26/push/master/310fd9f/github-Test on macOS-3292-failed/1_test (brew).txt:2022-05-27T00:07:00.9596860Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsh4hipt0_/subds1'
2022/05/26/push/master/310fd9f/github-Test on macOS-3292-failed/test (brew)/8_Run tests.txt:2022-05-27T00:07:00.9596850Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsh4hipt0_/subds1'
2022/05/31/pr/UNK/3bfa84d/github-Test on macOS-3300-failed/1_test (brew).txt:2022-05-31T09:34:59.7659000Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetse8vd93ly/subds1'
2022/05/31/pr/UNK/3bfa84d/github-Test on macOS-3300-failed/test (brew)/8_Run tests.txt:2022-05-31T09:34:59.7659000Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetse8vd93ly/subds1'
2022/04/28/pr/6273/c275bb8/appveyor-7919-failed/6454nxngm6cc266r.txt:[00:24:51] E           FileExistsError: [Errno 17] File exists: '/home/appveyor/DLTMP/datalad_temp_test_creatsubdatasetsuhgq2cyl/subds1'
2022/04/28/pr/6273/c275bb8/appveyor-7919-failed/6454nxngm6cc266r.txt:[00:24:51] FAILED ../datalad/support/tests/test_parallel.py::test_creatsubdatasets - FileExistsError: [Errno 17] File exists: '/home/appveyor/DLTMP/datalad_temp_test_creatsubdatasetsuhgq2cyl/subds1'
2022/04/28/pr/UNK/5182b79/github-Test on macOS-3187-failed/1_test (brew).txt:2022-04-28T20:26:51.4854940Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsoddmgt8t/subds1'
2022/04/28/pr/UNK/5182b79/github-Test on macOS-3187-failed/test (brew)/8_Run tests.txt:2022-04-28T20:26:51.4854930Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsoddmgt8t/subds1'
2022/04/29/pr/6273/ee77a1d/github-Test on macOS-3200-failed/1_test (brew).txt:2022-04-29T13:44:13.1267760Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsaeo0rz6e/subds1'
2022/04/29/pr/6273/ee77a1d/github-Test on macOS-3200-failed/test (brew)/8_Run tests.txt:2022-04-29T13:44:13.1267750Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsaeo0rz6e/subds1'
2021/11/05/push/master/54d1600/travis-12877-failed/5.txt:FileExistsError: [Errno 17] File exists: '/tmp/datalad_temp_test_creatsubdatasetsj8sbl87p/subds1'
2022/07/08/pr/6820/86f5df6/github-Test on macOS-3464-failed/1_test (brew).txt:2022-07-08T21:12:02.0118900Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsm0ismy03/subds1'
2022/07/08/pr/6820/86f5df6/github-Test on macOS-3464-failed/test (brew)/8_Run tests.txt:2022-07-08T21:12:02.0118900Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsm0ismy03/subds1'
2022/07/21/push/master/1c8944c/github-Test on macOS-3511-failed/1_test (brew).txt:2022-07-21T16:36:48.6520110Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets06q79b6m/subds1'
2022/07/21/push/master/1c8944c/github-Test on macOS-3511-failed/test (brew)/8_Run tests.txt:2022-07-21T16:36:48.6520110Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets06q79b6m/subds1'
2022/07/26/pr/6888/09bd37c/github-Test on macOS-3531-failed/2_test (snapshot).txt:2022-07-26T15:19:27.6787610Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsdsxz3eim/subds1'
2022/07/26/pr/6888/09bd37c/github-Test on macOS-3531-failed/test (snapshot)/8_Run tests.txt:2022-07-26T15:19:27.6787610Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasetsdsxz3eim/subds1'
2022/07/28/pr/6893/895becc/github-Test on macOS-3559-failed/1_test (brew).txt:2022-07-28T15:55:45.1297640Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets4k3wg4_g/subds1'
2022/07/28/pr/6893/895becc/github-Test on macOS-3559-failed/test (brew)/8_Run tests.txt:2022-07-28T15:55:45.1297640Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets4k3wg4_g/subds1'
2022/08/01/push/doc-6800/c6fc9ea/github-Test on macOS-3566-failed/1_test (brew).txt:2022-08-01T13:25:57.6429030Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets8emitnyg/subds1'
2022/08/01/push/doc-6800/c6fc9ea/github-Test on macOS-3566-failed/test (brew)/8_Run tests.txt:2022-08-01T13:25:57.6429020Z E           FileExistsError: [Errno 17] File exists: '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_test_creatsubdatasets8emitnyg/subds1'
2022/08/17/pr/6944/15b2794/appveyor-8370-failed/u0xvbqf6yr7qoajn.txt:[00:50:17] E           FileExistsError: [Errno 17] File exists: '/Users/appveyor/DLTMP/datalad_temp_test_creatsubdatasetsu3fdciqg/subds1'
2022/08/17/pr/6944/15b2794/appveyor-8370-failed/u0xvbqf6yr7qoajn.txt:[00:50:17] FAILED ../datalad/support/tests/test_parallel.py::test_creatsubdatasets - FileExistsError: [Errno 17] File exists: '/Users/appveyor/DLTMP/datalad_temp_test_creatsubdatasetsu3fdciqg/subds1'

so indeed OSX only! (but could be snapshot) and we should expect it to come about now -- it is a rare month without it, unless it was somehow resolved.

@yarikoptic yarikoptic added the fix-implemented A fix is available, but has not been merged or released, yet. label Oct 7, 2022
yarikoptic added a commit to yarikoptic/datalad that referenced this issue Oct 7, 2022
@yarikoptic
BF: lock across threads check/instantiation of Flyweight instances
8cdc19b 
As commented added to the comment says we might want (eventually) make
locking smarter and per "id" since in principle we should be able to
instantiate multiple separate instances in parallel.  Might be useful or
even critical for parallel "create" and alike.  But I think such use-cases
are rare so decided to KISS.  test added, so disable locking to see how things
could go wrong (hint: there is more than 1 way ;) ).

Fixes datalad#6598
@yarikoptic yarikoptic mentioned this issue Oct 7, 2022
Merged
yarikoptic added a commit to yarikoptic/datalad that referenced this issue Oct 7, 2022
@yarikoptic
BF: lock across threads check/instantiation of Flyweight instances
e22230f 
As commented added to the comment says we might want (eventually) make
locking smarter and per "id" since in principle we should be able to
instantiate multiple separate instances in parallel.  Might be useful or
even critical for parallel "create" and alike.  But I think such use-cases
are rare so decided to KISS.  test added, so disable locking to see how things
could go wrong (hint: there is more than 1 way ;) ).

Fixes datalad#6598
@yarikoptic
Copy link
Member

I believe it is addressed by #7075 merged into maint.

@yarikoptic-gitmate
Copy link
Collaborator

Issue fixed in 0.17.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fix-implemented A fix is available, but has not been merged or released, yet.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BF: lock across threads check/instantiation of Flyweight instances yarikoptic/datalad
3 participants
@yarikoptic @bpoldrack @yarikoptic-gitmate