Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UX: Allow create-sibling-ria to only act on existing store #6045

Merged
merged 6 commits into from Oct 12, 2021
Merged

Conversation

adswa
Copy link
Member

@adswa adswa commented Oct 6, 2021

This implements a basic solution to #5982.
The issue is a feature request for the ability to only act on
create-sibling-ria calls if the specified store already
exists. This change adds a parameter to create-sibling-ria,
'--construct-store' (False by default), which, if set, creates a new
store if it doesn't exist under the specified url, and otherwise fails to
create a RIA sibling, warning the user that there is no store at the
specified path. This is an API change to the current release, which
will create a new store if it doesn't find one.

The parameter is intentially not named create-store because this would
lead to a name conflict with the create_store method of ria_utils.

@adswa adswa linked an issue Oct 6, 2021 that may be closed by this pull request
@adswa adswa added the semver-minor Increment the minor version when merged label Oct 6, 2021
@adswa adswa requested a review from bpoldrack October 6, 2021 14:25
Copy link
Member

@bpoldrack bpoldrack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless there are strong objections to my suggestion I would prefer it to be the other way around. Only the first dataset per store would need to actually create it. Hence, in most cases creation isn't needed. I would go with a --create-store flag rather than expect-store.
For maint its default should probably be true, but switch to false in master. (or not have it in maint at all)

@@ -394,7 +400,18 @@ def __call__(url,
# - more generally consider store creation a dedicated command or
# option

create_store(SSHRemoteIO(ssh_host) if ssh_host else LocalIO(),
io = SSHRemoteIO(ssh_host) if ssh_host else LocalIO()
if not io.exists(Path(base_path)) and expect_store:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with how the ORA remote asses the existence of a store, a valid ria-layout-version file at that path is needed, too. Mere existence of the path itself doesn't make it a useable store. Also: base_path could be a file and would pass this test.

So, I think change condition to "io.read_file(Path(base_path) / 'ria-layout-version') doesn't raise anything".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have implemented this now. But I had to take a detour via a try/except clause, because io.read_file(Path(base_path) / 'ria-layout-version') will raise FileNotFound if there is no layout. If you know a more elegant way, I'd be all ears :-)

@adswa
Copy link
Member Author

adswa commented Oct 6, 2021

Thanks for the review, @bpoldrack!

Only the first dataset per store would need to actually create it. Hence, in most cases creation isn't needed. I would go with a --create-store flag rather than expect-store.

Sure, also makes sense. In your opinion, what should happen in case of --create-store at a path of an existing store? Shall we error out, shall we ignore the flag and continue placing a sibling in it, or a "mix" between completely ignoring and erroring with some form of logging?

For maint its default should probably be true, but switch to false in master. (or not have it in maint at all)

I'd personally prefer to not have it in maint at all

@codecov
Copy link

codecov bot commented Oct 6, 2021

Codecov Report

Merging #6045 (7a18896) into master (c8e7ae9) will decrease coverage by 53.94%.
The diff coverage is 21.42%.

❗ Current head 7a18896 differs from pull request most recent head ce6050c. Consider uploading reports for the commit ce6050c to get more accurate results
Impacted file tree graph

@@             Coverage Diff             @@
##           master    #6045       +/-   ##
===========================================
- Coverage   89.83%   35.89%   -53.95%     
===========================================
  Files         319      319               
  Lines       42414    42424       +10     
===========================================
- Hits        38102    15226    -22876     
- Misses       4312    27198    +22886     
Impacted Files Coverage Δ
datalad/core/distributed/tests/test_push.py 98.72% <ø> (-0.22%) ⬇️
...talad/distributed/tests/test_create_sibling_ria.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/distributed/tests/test_ria.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/distributed/tests/test_ria_basics.py 0.00% <ø> (-97.87%) ⬇️
datalad/distribution/tests/test_get.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/interface/common_opts.py 100.00% <ø> (ø)
datalad/support/archives.py 0.00% <0.00%> (-86.35%) ⬇️
datalad/tests/test_utils.py 0.00% <0.00%> (-97.20%) ⬇️
datalad/distributed/create_sibling_ria.py 66.48% <50.00%> (-16.18%) ⬇️
datalad/core/distributed/tests/test_clone.py 97.43% <100.00%> (ø)
... and 278 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c8e7ae9...ce6050c. Read the comment docs.

@mih
Copy link
Member

mih commented Oct 7, 2021

I also see this not targeting maint at all.

@bpoldrack
Copy link
Member

@adswa

In your opinion, what should happen in case of --create-store at a path of an existing store? Shall we error out, shall we ignore the flag and continue placing a sibling in it, or a "mix" between completely ignoring and erroring with some form of logging?

Continue placing a sibling in it. It's create-sibling after all. The purpose of --create-store, I think, would be to create one if it doesn't exist. If there's a store already then there's nothing to do, but also no conflict.

Comment on lines +422 to +425
log_progress(
lgr.info, 'create-sibling-ria',
'Creating a new RIA store at %s', Path(base_path),
)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a question: I couldn't figure out where this logging ends up. No log level unearthed it. Where does it go, @bpoldrack?

Copy link
Member

@bpoldrack bpoldrack Oct 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depends. Into a progress bar if possible. But it may end up in a log message, if there's no progress bar for some reason. See datalad.log.log_progress

@adswa adswa requested a review from bpoldrack October 7, 2021 10:04
@adswa adswa force-pushed the ux-5982 branch 2 times, most recently from 3654f38 to 781b3f0 Compare October 7, 2021 11:03
Copy link
Member

@bpoldrack bpoldrack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor suggestion to make the doc string a little more clear from my POV, otherwise: LGTM!

datalad/distributed/create_sibling_ria.py Outdated Show resolved Hide resolved
@adswa
Copy link
Member Author

adswa commented Oct 7, 2021

mhh, there is a test failure on macOS and crippledFS that looks like it could be related...
yep that was me 🙃

 ======================================================================
D: /a/b/c
ERROR: datalad.core.distributed.tests.test_push.test_nested_pushclone_cycle_allplatforms
D: /a~
----------------------------------------------------------------------
D: /a b/name
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/core/distributed/tests/test_push.py", line 949, in test_nested_pushclone_cycle_allplatforms
    '--construct-store'])
  File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/runner/runner.py", line 149, in run
    **results,
datalad.runner.exception.CommandError: CommandError: 'datalad -C super get --recursive . --construct-store' failed with exitcode 1

@@ -945,7 +945,8 @@ def test_nested_pushclone_cycle_allplatforms(origpath, storepath, clonepath):
cloneurl = 'ria+' + get_local_file_url(str(storepath), compatibility='git')
with chpwd(clonepath):
run(['datalad', 'clone', cloneurl + '#' + orig_super.id, 'super'])
run(['datalad', '-C', 'super', 'get', '--recursive', '.'])
run(['datalad', '-C', 'super', 'get', '--recursive', '.',
'--construct-store'])
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stupid me, this is wrong

@adswa
Copy link
Member Author

adswa commented Oct 7, 2021

mhhh quite a potpourri of failures... I don't see how the PR is responsible for (all of) them

``` ====================================================================== ERROR: datalad.core.distributed.tests.test_clone.test_clone_datasets_root ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network return func(*args, **kwargs) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/vcr/cassette.py", line 100, in __call__ return type(self)(self.cls, args_getter)._execute_function(function, args, kwargs) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/vcr/cassette.py", line 114, in _execute_function return self._handle_function(fn=handle_function) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/vcr/cassette.py", line 138, in _handle_function return fn(cassette) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/vcr/cassette.py", line 107, in handle_function return function(*args, **kwargs) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/tests/utils.py", line 563, in _wrap_with_tree return t(*(arg + (d,)), **kw) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/core/distributed/tests/test_clone.py", line 151, in test_clone_datasets_root ds = clone("///") File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/interface/utils.py", line 483, in eval_func return return_func(generator_func)(*args, **kwargs) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/interface/utils.py", line 476, in return_func results = list(results) File "/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/datalad/interface/utils.py", line 463, in generator_func msg="Command did not complete successfully") datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed: [{'action': 'install', 'message': ('Failed to clone from any candidate source URL. Encountered ' 'errors per each url were:\n' '- %s', 'https://datasets-tests.datalad.org/\n' " CommandError: 'git -c diff.ignoreSubmodules=none clone " '--progress https://datasets-tests.datalad.org/ ' "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_tree_test_clone_datasets_rootfi1riio8/datasets-tests.datalad.org' " "failed with exitcode 128 [err: 'Cloning into " "'/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_tree_test_clone_datasets_rootfi1riio8/datasets-tests.datalad.org'...\n" "fatal: repository 'https://datasets-tests.datalad.org/' not " "found']\n" '- https://datasets-tests.datalad.org/.git\n' " CommandError: 'git -c diff.ignoreSubmodules=none clone " '--progress https://datasets-tests.datalad.org/.git ' "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_tree_test_clone_datasets_rootfi1riio8/datasets-tests.datalad.org' " "failed with exitcode 128 [err: 'Cloning into " "'/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_tree_test_clone_datasets_rootfi1riio8/datasets-tests.datalad.org'...\n" 'Fetching objects: 5\r' 'error: Unable to get pack index ' 'https://datasets-tests.datalad.org/.git/objects/pack/pack-10e4c61ee138e3fd701f7fab9e68f45b20b8cff5.idx\n' 'error: Unable to find af926ef0c359556ac1d36d71f7e173d97b893ff2 ' 'under https://datasets-tests.datalad.org/.git\n' 'Fetching objects: 8, done.\n' 'Cannot obtain needed blob ' 'af926ef0c359556ac1d36d71f7e173d97b893ff2\n' 'while processing commit ' 'd71c855ef55032962bfe205169721c781f36ae21.\n' "error: fetch failed.']"), 'path': '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/datalad_temp_tree_test_clone_datasets_rootfi1riio8/datasets-tests.datalad.org', 'source_url': '///', 'status': 'error', 'type': 'dataset'}] ```
======================================================================
FAIL: datalad.tests.test_installed.test_run_datalad_help
----------------------------------------------------------------------
Traceback (most recent call last):

  File "/tmp/dl-miniconda-71v5gu1i/lib/python3.9/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/tmp/dl-miniconda-71v5gu1i/lib/python3.9/site-packages/datalad/tests/utils.py", line 1168, in _wrap_assert_cwd_unchanged
    raise exc_info[1]
  File "/tmp/dl-miniconda-71v5gu1i/lib/python3.9/site-packages/datalad/tests/utils.py", line 1140, in _wrap_assert_cwd_unchanged
    ret = func(*args, **kwargs)
  File "/tmp/dl-miniconda-71v5gu1i/lib/python3.9/site-packages/datalad/tests/test_installed.py", line 44, in test_run_datalad_help
    eq_(err, "")

AssertionError: '/tmp/dl-miniconda-71v5gu1i/lib/python3.9/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected)\n  self._warn("No data was collected.", slug="no-data-collected")\n' != ''
======================================================================
ERROR: datalad.core.distributed.tests.test_clone.test_gin_cloning
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/core/distributed/tests/test_clone.py", line 1504, in test_gin_cloning
    ds = clone('https://gin.g-node.org/datalad/datalad-ci-target', path)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 461, in generator_func
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'install',
  'message': ('Failed to clone from any candidate source URL. Encountered '
              'errors per each url were:\n'
              '- %s',
              'https://gin.g-node.org/datalad/datalad-ci-target\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress https://gin.g-node.org/datalad/datalad-ci-target '
              "/home/appveyor/DLTMP/datalad_temp_test_gin_cloning1uucak6y' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_test_gin_cloning1uucak6y'...\n"
              'fatal: unable to access '
              "'https://gin.g-node.org/datalad/datalad-ci-target/': server "
              "certificate verification failed. CAfile: none CRLfile: none']\n"
              '- https://gin.g-node.org/datalad/datalad-ci-target/.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress '
              'https://gin.g-node.org/datalad/datalad-ci-target/.git '
              "/home/appveyor/DLTMP/datalad_temp_test_gin_cloning1uucak6y' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_test_gin_cloning1uucak6y'...\n"
              'fatal: unable to access '
              "'https://gin.g-node.org/datalad/datalad-ci-target/.git/': "
              'server certificate verification failed. CAfile: none CRLfile: '
              "none']"),
  'path': '/home/appveyor/DLTMP/datalad_temp_test_gin_cloning1uucak6y',
  'source_url': 'https://gin.g-node.org/datalad/datalad-ci-target',
  'status': 'error',
  'type': 'dataset'}]
======================================================================
ERROR: datalad.distributed.tests.test_create_sibling_ria.test_create_simple('datalad-test',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 210, in _wrap_skip_if_on_windows
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 284, in _wrap_skip_ssh
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 210, in _wrap_skip_if_on_windows
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/tests/test_create_sibling_ria.py", line 57, in _wrap_with_store_insteadof
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 563, in _wrap_with_tree
    return t(*(arg + (d,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/tests/test_create_sibling_ria.py", line 104, in _test_create_store
    res = ds.create_sibling_ria("ria+ssh://test-store:", "datastore",
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/dataset.py", line 503, in apply_func
    return f(**kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 396, in generator_func
    for r in _process_results(
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 574, in _process_results
    for res in results:
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/create_sibling_ria.py", line 409, in __call__
    io.read_file(Path(base_path) / 'ria-layout-version')
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/ora_remote.py", line 579, in read_file
    raise RIARemoteError("Could not read {}".format(str(file_path)))
datalad.distributed.ora_remote.RIARemoteError: Could not read /home/appveyor/DLTMP/datalad_temp__test_create_store7n8m_dkg/ria-layout-version
-------------------- >> begin captured logging << --------------------
fasteners.process_lock: Level 5: Acquired file lock `b'/home/appveyor/DLTMP/datalad_temp_x6n71g_n/.cache/datalad/sockets/353e4258.lck'` after waiting 0.000s [1 attempts were required]
fasteners.process_lock: Level 5: Unlocked and closed file lock open on `b'/home/appveyor/DLTMP/datalad_temp_x6n71g_n/.cache/datalad/sockets/353e4258.lck'`
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: datalad.distributed.tests.test_create_sibling_ria.test_create_push_url
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 284, in _wrap_skip_ssh
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 210, in _wrap_skip_if_on_windows
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 563, in _wrap_with_tree
    return t(*(arg + (d,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/tests/test_create_sibling_ria.py", line 233, in test_create_push_url
    ds.create_sibling_ria(url, "datastore", push_url=push_url,
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/dataset.py", line 503, in apply_func
    return f(**kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 396, in generator_func
    for r in _process_results(
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 574, in _process_results
    for res in results:
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/create_sibling_ria.py", line 409, in __call__
    io.read_file(Path(base_path) / 'ria-layout-version')
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distributed/ora_remote.py", line 579, in read_file
    raise RIARemoteError("Could not read {}".format(str(file_path)))
datalad.distributed.ora_remote.RIARemoteError: Could not read /home/appveyor/DLTMP/datalad_temp_test_create_push_urlub4w4f4_/ria-layout-version
-------------------- >> begin captured logging << --------------------
fasteners.process_lock: Level 5: Acquired file lock `b'/home/appveyor/DLTMP/datalad_temp_x6n71g_n/.cache/datalad/sockets/353e4258.lck'` after waiting 0.000s [1 attempts were required]
fasteners.process_lock: Level 5: Unlocked and closed file lock open on `b'/home/appveyor/DLTMP/datalad_temp_x6n71g_n/.cache/datalad/sockets/353e4258.lck'`
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: datalad.distribution.tests.test_install.test_install_datasets_root
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 100, in __call__
    return type(self)(self.cls, args_getter)._execute_function(function, args, kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 114, in _execute_function
    return self._handle_function(fn=handle_function)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 138, in _handle_function
    return fn(cassette)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 107, in handle_function
    return function(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 563, in _wrap_with_tree
    return t(*(arg + (d,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/tests/test_install.py", line 182, in test_install_datasets_root
    ds = install("///")
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 461, in generator_func
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'install',
  'message': ('Failed to clone from any candidate source URL. Encountered '
              'errors per each url were:\n'
              '- %s',
              'https://datasets-tests.datalad.org/\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress https://datasets-tests.datalad.org/ '
              "/home/appveyor/DLTMP/datalad_temp_tree_test_install_datasets_root9vvtwmpu/datasets-tests.datalad.org' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_tree_test_install_datasets_root9vvtwmpu/datasets-tests.datalad.org'...\n"
              "fatal: unable to access 'https://datasets-tests.datalad.org/': "
              'server certificate verification failed. CAfile: none CRLfile: '
              "none']\n"
              '- https://datasets-tests.datalad.org/.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress https://datasets-tests.datalad.org/.git '
              "/home/appveyor/DLTMP/datalad_temp_tree_test_install_datasets_root9vvtwmpu/datasets-tests.datalad.org' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_tree_test_install_datasets_root9vvtwmpu/datasets-tests.datalad.org'...\n"
              'fatal: unable to access '
              "'https://datasets-tests.datalad.org/.git/': server certificate "
              "verification failed. CAfile: none CRLfile: none']"),
  'path': '/home/appveyor/DLTMP/datalad_temp_tree_test_install_datasets_root9vvtwmpu/datasets-tests.datalad.org',
  'refds': None,
  'source_url': '///',
  'status': 'error',
  'type': 'dataset'}]
======================================================================
ERROR: datalad.distribution.tests.test_install.test_datasets_datalad_org('',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/tests/test_install.py", line 894, in check_datasets_datalad_org
    ds = install(tdir, source='///dicoms/dartmouth-phantoms/bids_test6-PD+T2w' + suffix)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 461, in generator_func
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'install',
  'message': ('Failed to clone from any candidate source URL. Encountered '
              'errors per each url were:\n'
              '- %s',
              'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress '
              'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w '
              "/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_orgl4w_wdwq' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_orgl4w_wdwq'...\n"
              'fatal: unable to access '
              "'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/': "
              'server certificate verification failed. CAfile: none CRLfile: '
              "none']\n"
              '- '
              'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress '
              'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/.git '
              "/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_orgl4w_wdwq' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_orgl4w_wdwq'...\n"
              'fatal: unable to access '
              "'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/.git/': "
              'server certificate verification failed. CAfile: none CRLfile: '
              "none']"),
  'path': '/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_orgl4w_wdwq',
  'refds': None,
  'source_url': '///dicoms/dartmouth-phantoms/bids_test6-PD+T2w',
  'status': 'error',
  'type': 'dataset'}]
======================================================================
ERROR: datalad.distribution.tests.test_install.test_datasets_datalad_org('/.git',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/tests/test_install.py", line 894, in check_datasets_datalad_org
    ds = install(tdir, source='///dicoms/dartmouth-phantoms/bids_test6-PD+T2w' + suffix)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 461, in generator_func
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'install',
  'message': ('Failed to clone from any candidate source URL. Encountered '
              'errors per each url were:\n'
              '- %s',
              'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress '
              'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/.git '
              "/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_org49my27qd' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_org49my27qd'...\n"
              'fatal: unable to access '
              "'https://datasets-tests.datalad.org/dicoms/dartmouth-phantoms/bids_test6-PD%2BT2w/.git/': "
              'server certificate verification failed. CAfile: none CRLfile: '
              "none']"),
  'path': '/home/appveyor/DLTMP/datalad_temp_check_datasets_datalad_org49my27qd',
  'refds': None,
  'source_url': '///dicoms/dartmouth-phantoms/bids_test6-PD+T2w/.git',
  'status': 'error',
  'type': 'dataset'}]
======================================================================
ERROR: datalad.distribution.tests.test_uninstall.test_remove_recursive_2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 100, in __call__
    return type(self)(self.cls, args_getter)._execute_function(function, args, kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 114, in _execute_function
    return self._handle_function(fn=handle_function)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 138, in _handle_function
    return fn(cassette)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 107, in handle_function
    return function(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/tests/test_uninstall.py", line 482, in test_remove_recursive_2
    install('///labs')
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 483, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 476, in return_func
    results = list(results)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/interface/utils.py", line 461, in generator_func
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'install',
  'message': ('Failed to clone from any candidate source URL. Encountered '
              'errors per each url were:\n'
              '- %s',
              'https://datasets-tests.datalad.org/labs\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress https://datasets-tests.datalad.org/labs '
              "/home/appveyor/DLTMP/datalad_temp_test_remove_recursive_2h05ztsdq/labs' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_test_remove_recursive_2h05ztsdq/labs'...\n"
              'fatal: unable to access '
              "'https://datasets-tests.datalad.org/labs/': server certificate "
              "verification failed. CAfile: none CRLfile: none']\n"
              '- https://datasets-tests.datalad.org/labs/.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress https://datasets-tests.datalad.org/labs/.git '
              "/home/appveyor/DLTMP/datalad_temp_test_remove_recursive_2h05ztsdq/labs' "
              "failed with exitcode 128 [err: 'Cloning into "
              "'/home/appveyor/DLTMP/datalad_temp_test_remove_recursive_2h05ztsdq/labs'...\n"
              'fatal: unable to access '
              "'https://datasets-tests.datalad.org/labs/.git/': server "
              "certificate verification failed. CAfile: none CRLfile: none']"),
  'path': '/home/appveyor/DLTMP/datalad_temp_test_remove_recursive_2h05ztsdq/labs',
  'refds': None,
  'source_url': '///labs',
  'status': 'error',
  'type': 'dataset'}]
======================================================================
FAIL: datalad.core.distributed.tests.test_clone.test_clone_crcns
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 1266, in assert_status
    assert_in(r['status'], label)
  File "/usr/lib/python3.8/unittest/case.py", line 1179, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python3.8/unittest/case.py", line 753, in fail
    raise self.failureException(msg)
AssertionError: 'error' not found in ['ok']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 100, in __call__
    return type(self)(self.cls, args_getter)._execute_function(function, args, kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 114, in _execute_function
    return self._handle_function(fn=handle_function)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 138, in _handle_function
    return fn(cassette)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 107, in handle_function
    return function(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/core/distributed/tests/test_clone.py", line 134, in test_clone_crcns
    assert_status('ok', res)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 1268, in assert_status
    raise AssertionError('Test {}/{}: expected status {} not found in:\n{}'.format(
AssertionError: Test 1/1: expected status ['ok'] not found in:
[
 {
  "action": "install",
  "path": "/home/appveyor/DLTMP/datalad_temp_test_clone_crcnsje5tcl8_/all-nonrecursive",
  "type": "dataset",
  "status": "error",
  "message": [
   "Failed to clone from any candidate source URL. Encountered errors per each url were:\n- %s",
   "https://datasets-tests.datalad.org/\n  CommandError: 'git -c diff.ignoreSubmodules=none clone --progress https://datasets-tests.datalad.org/ /home/appveyor/DLTMP/datalad_temp_test_clone_crcnsje5tcl8_/all-nonrecursive' failed with exitcode 128 [err: 'Cloning into '/home/appveyor/DLTMP/datalad_temp_test_clone_crcnsje5tcl8_/all-nonrecursive'...\nfatal: unable to access 'https://datasets-tests.datalad.org/': server certificate verification failed. CAfile: none CRLfile: none']\n- https://datasets-tests.datalad.org/.git\n  CommandError: 'git -c diff.ignoreSubmodules=none clone --progress https://datasets-tests.datalad.org/.git /home/appveyor/DLTMP/datalad_temp_test_clone_crcnsje5tcl8_/all-nonrecursive' failed with exitcode 128 [err: 'Cloning into '/home/appveyor/DLTMP/datalad_temp_test_clone_crcnsje5tcl8_/all-nonrecursive'...\nfatal: unable to access 'https://datasets-tests.datalad.org/.git/': server certificate verification failed. CAfile: none CRLfile: none']"
  ],
  "source_url": "///"
 }
]
======================================================================
FAIL: datalad.distribution.tests.test_install.test_failed_install_multiple
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 190, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 100, in __call__
    return type(self)(self.cls, args_getter)._execute_function(function, args, kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 114, in _execute_function
    return self._handle_function(fn=handle_function)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 138, in _handle_function
    return fn(cassette)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/vcr/cassette.py", line 107, in handle_function
    return function(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 742, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/distribution/tests/test_install.py", line 465, in test_failed_install_multiple
    eq_(ds.subdatasets(result_xfm='relpaths'), ['crcns', 'ds1', 'ds3'])
AssertionError: ['ds1', 'ds3'] != ['crcns', 'ds1', 'ds3']

Copy link
Member

@mih mih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the test failures (all?) are related to the certificate issue we had recently. The PR is based on a relatively old state of master, maybe a rebase improves the situation.

re naming of the new flag: we are not using "construct" anywhere so far. We use "create". Unless there is a good reason for switching terms, we better not. If the Q whether or not a create of an already existing store is a problem, I'd say that we could think about something like --new-store-ok.

@adswa
Copy link
Member Author

adswa commented Oct 8, 2021

re naming of the new flag: we are not using "construct" anywhere so far. We use "create". Unless there is a good reason for switching terms, we better not.

The name create-store conflicts with a non-private method in ria_utils. --new-store-ok is also nice, IMO. I've renamed it, and rebased to current master.

This implements a basic solution to #5982.
The issue is a feature request for the ability to only act on
create-sibling-ria calls if the specified store already
exists. This change adds a parameter to create-sibling-ria,
'--construct-store' (False by default), which, if set, creates a new
store if it doesn't exist under the specified url, and otherwise fails to
create a RIA sibling, warning the user that there is no store at the
specified path.

The parameter is intentially not named 'create-store' because this would
lead to a name conflict with the 'create_store' method of ria_utils.

TMP typos
TST: Adjust other tests to the API change in create-sibling-ria
@bpoldrack
Copy link
Member

The parameter is intentially not named create-store because this would
lead to a name conflict with the create_store method of ria_utils.

FTR: not a conflict really. One thing is an option to a command, the other a function name. Plus: ria_utils isn't really part of public API anyways.

@adswa
Copy link
Member Author

adswa commented Oct 8, 2021

FTR: not a conflict really. One thing is an option to a command, the other a function name. Plus: ria_utils isn't really part of public API anyways.

But parameter and function have a same name and both are used in the call method. Doesn't this add unnecessary confusion?

if create_store:
    create_store()

@bpoldrack
Copy link
Member

bpoldrack commented Oct 8, 2021

But parameter and function have a same name and both are used in the call method. Doesn't this add unnecessary confusion?

Arguably, yes. But then I'd rather rename an internal function, than making it a "burden" for public API names. However, I personally don't find it particularly confusing. Things like function(dothis=dothis) are not a rare thing either.

@adswa
Copy link
Member Author

adswa commented Oct 8, 2021

ok, noted. Just let me know how you would like me to change it.

@bpoldrack
Copy link
Member

make_store or setup_store would be fine, I guess.

@adswa
Copy link
Member Author

adswa commented Oct 8, 2021

make_store or setup_store would be fine, I guess.

sorry for asking again, just to be clear, that's a new name for the method create_store() in ria_utils, right?

@mih mih mentioned this pull request Oct 9, 2021
Copy link
Member

@mih mih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would vote for keeping new_store_ok and keep internal function names as-is.

The test failures suggest that some (two?) new_store_ok flags are still missing in the tests.

datalad/distributed/create_sibling_ria.py Outdated Show resolved Hide resolved
datalad/distributed/create_sibling_ria.py Outdated Show resolved Hide resolved
adswa and others added 2 commits October 11, 2021 07:45
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
Co-authored-by: Michael Hanke <michael.hanke@gmail.com>
When using an SSH url, a non-existent store would raise a RIARemoteError
or RemoteCommandFailedError instead of a FileNotFoundError.
This change catches these cases, and allows
the creation of new stores over SSH urls again.
Copy link
Member

@mih mih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thx again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-minor Increment the minor version when merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Creating a ria sibling only if store exists
3 participants