Merge commit '0.16.3-13-g446d3c0bd' into merge-maint

* commit '0.16.3-13-g446d3c0bd':
  TST: Add minimal testcase for #6712
  BF: Catch KeyErrors from unavailable WTF infos
  BF(TST): do full clone to ensure having most recent tag
  Add annex.private to ephemeral clones
  Replace Zenodo DOI with JOSS for due credit
  ENH: comment out errorneous check
  ENH: convert-git-annex-layout - better condensed feedback via tqdm
  A little helper to jump between git annex objects layouts
  ENH: provide argcomplete version depend at least 1.12.3
  [skip ci] Update RST changelog
  Update CHANGELOG.md [skip ci]
  No change for a PR to trigger release
  TST: test correct content availability of url-key files after url-sanitization
  BF: Sanitize keys before marking content availability
  MNT: Move _sanitize_key helper from ora_remote into annex_utils

 Conflicts:
	datalad/support/tests/test_fileinfo.py - was already CPed into maint, caused conflict on import. fixed

.travis.yml

@@ -1,6 +1,15 @@
 # vim ft=yaml
 # travis-ci.org definition for DataLad build
 language: python
+
+# Do full clone (~10 extra seconds) to fetch all tags.
+# Otherwise we might be missing the tags for maint PRs
+# whenever those maint releases were not yet merged into master.
+# TODO: specify TRAVIS_BRANCH if ever would become possible
+#(support request was sent to Travis)
+git:
+  depth: false
+
 services:
   - docker
 

CHANGELOG.md

@@ -1,3 +1,26 @@
+# 0.16.3 (Thu May 12 2022)
+
+#### 🐛 Bug Fix
+
+- No change for a PR to trigger release [#6692](https://github.com/datalad/datalad/pull/6692) ([@yarikoptic](https://github.com/yarikoptic))
+- Sanitize keys before checking content availability to ensure correct value for keys with URL or custom backend [#6665](https://github.com/datalad/datalad/pull/6665) ([@adswa](https://github.com/adswa) [@yarikoptic](https://github.com/yarikoptic))
+- Change a key-value pair in drop result record [#6625](https://github.com/datalad/datalad/pull/6625) ([@mslw](https://github.com/mslw))
+- Link docs of datalad-next [#6677](https://github.com/datalad/datalad/pull/6677) ([@mih](https://github.com/mih))
+- Fix `GitRepo.get_branch_commits_()` to handle branch names conflicts with paths [#6661](https://github.com/datalad/datalad/pull/6661) ([@mih](https://github.com/mih))
+- OPT: AnnexJsonProtocol - avoid dragging possibly long data around [#6660](https://github.com/datalad/datalad/pull/6660) ([@yarikoptic](https://github.com/yarikoptic))
+- Remove two too prominent create() INFO log message that duplicate DEBUG log and harmonize some other log messages [#6638](https://github.com/datalad/datalad/pull/6638) ([@mih](https://github.com/mih) [@yarikoptic](https://github.com/yarikoptic))
+- Remove unsupported parameter create_sibling_ria(existing=None) [#6637](https://github.com/datalad/datalad/pull/6637) ([@mih](https://github.com/mih))
+- Add released plugin to .autorc to annotate PRs on when released [#6639](https://github.com/datalad/datalad/pull/6639) ([@yarikoptic](https://github.com/yarikoptic))
+
+#### Authors: 4
+
+- Adina Wagner ([@adswa](https://github.com/adswa))
+- Michael Hanke ([@mih](https://github.com/mih))
+- Michał Szczepanik ([@mslw](https://github.com/mslw))
+- Yaroslav Halchenko ([@yarikoptic](https://github.com/yarikoptic))
+
+---
+
 # 0.16.2 (Thu Apr 21 2022)
 
 #### 🐛 Bug Fix

datalad/core/distributed/clone.py

@@ -1137,6 +1137,16 @@ def postclonecfg_annexdataset(ds, reckless, description=None, remote="origin"):
             "sources, if possible (reckless)", ds.path)
         ds.config.set(
             'annex.hardlink', 'true', scope='local', reload=True)
+    elif reckless == 'ephemeral':
+        # In ephemeral clones we set annex.private=true. This would prevent the
+        # location itself being recorded in uuid.log. With a private repo,
+        # declaring dead (see below after annex-init) seems somewhat
+        # superfluous, but on the other hand:
+        # If an older annex that doesn't support private yet touches the
+        # repo, the entire purpose of ephemeral would be sabotaged if we did
+        # not declare dead in addition. Hence, keep it regardless of annex
+        # version.
+        ds.config.set('annex.private', 'true', scope='local')
 
     lgr.debug("Initializing annex repo at %s", ds.path)
     # Note, that we cannot enforce annex-init via AnnexRepo().
@@ -1169,7 +1179,6 @@ def postclonecfg_annexdataset(ds, reckless, description=None, remote="origin"):
         ds.config.set(
             f'remote.{remote}.annex-ignore', 'true',
             scope='local')
-
         ds.repo.set_remote_dead('here')
 
         if check_symlink_capability(ds.repo.dot_git / 'dl_link_test',

datalad/core/distributed/tests/test_clone.py

@@ -32,6 +32,7 @@
 from datalad.distribution.dataset import Dataset
 from datalad.support.annexrepo import AnnexRepo
 from datalad.support.exceptions import IncompleteResultsError
+from datalad.support.external_versions import external_versions
 from datalad.support.gitrepo import GitRepo
 from datalad.support.network import get_local_file_url
 from datalad.tests.utils_pytest import (
@@ -1367,6 +1368,7 @@ def test_ephemeral(origin_path=None, bare_path=None,
     origin.save()
     # 1. clone via path
     clone1 = clone(origin_path, clone1_path, reckless='ephemeral')
+    eq_(clone1.config.get("annex.private"), "true")
 
     can_symlink = has_symlink_capability()
 
@@ -1382,6 +1384,7 @@ def test_ephemeral(origin_path=None, bare_path=None,
     # 2. clone via file-scheme URL
     clone2 = clone('file://' + Path(origin_path).as_posix(), clone2_path,
                    reckless='ephemeral')
+    eq_(clone2.config.get("annex.private"), "true")
 
     if can_symlink:
         clone2_annex = (clone2.repo.dot_git / 'annex')
@@ -1401,8 +1404,16 @@ def test_ephemeral(origin_path=None, bare_path=None,
                       scope="local")
     # Note, that the only thing to test is git-annex-dead here,
     # if we couldn't symlink:
-    clone1.push(to=DEFAULT_REMOTE,
-                   data='nothing' if can_symlink else 'auto')
+    clone1.push(to=DEFAULT_REMOTE, data='nothing' if can_symlink else 'auto')
+
+    if external_versions['cmd:annex'] >= "8.20210428":
+        # ephemeral clones are private (if supported by annex version). Despite
+        # the push, clone1's UUID doesn't show up in origin
+        recorded_locations = origin.repo.call_git(['cat-file', 'blob',
+                                                   'git-annex:uuid.log'],
+                                                  read_only=True)
+        assert_not_in(clone1.config.get("annex.uuid"), recorded_locations)
+
     if not origin.repo.is_managed_branch():
         # test logic cannot handle adjusted branches
         eq_(origin.repo.get_hexsha(), clone1.repo.get_hexsha())

datalad/interface/common_opts.py

@@ -178,10 +178,11 @@
     Supported modes are:
     ['auto']: hard-link files between local clones. In-place
     modification in any clone will alter original annex content.
-    ['ephemeral']: symlink annex to origin's annex and discard local availability
-    info via git-annex-dead 'here'. Shares an annex between origin and clone
-    w/o git-annex being aware of it. In case of a change in origin you need to
-    update the clone before you're able to save new content on your end.
+    ['ephemeral']: symlink annex to origin's annex and discard local
+    availability info via git-annex-dead 'here' and declares this annex private.
+    Shares an annex between origin and clone w/o git-annex being aware of it.
+    In case of a change in origin you need to update the clone before you're
+    able to save new content on your end.
     Alternative to 'auto' when hardlinks are not an option, or number of consumed
     inodes needs to be minimized. Note that this mode can only be used with clones from
     non-bare repositories or a RIA store! Otherwise two different annex object tree

datalad/local/tests/test_wtf.py

@@ -125,6 +125,10 @@ def test_wtf(topdir=None):
         wtf(flavor='short', sections='*')
         assert_greater(len(cmo.out.splitlines()), 10)  #  many more
 
+    # check that wtf of an unavailable section yields impossible result (#6712)
+    res = wtf(sections=['murkie'], on_failure='ignore')
+    eq_(res[0]["status"], "impossible")
+
     # should result only in '# WTF'
     skip_if_no_module('pyperclip')
 

datalad/local/wtf.py

@@ -494,8 +494,18 @@ def __call__(*, dataset=None, sensitive=None, sections=None, flavor="full", deco
                 raise ValueError(flavor)
 
         for s in sections:
-            infos[s] = section_callables[s]()
-
+            try:
+                infos[s] = section_callables[s]()
+            except KeyError:
+                yield get_status_dict(
+                    action='wtf',
+                    path=Path.cwd(),
+                    status='impossible',
+                    message=(
+                      'Requested section <%s> was not found among the '
+                      'available infos. Skipping report.', s),
+                    logger=lgr
+                    )
         if clipboard:
             external_versions.check(
                 'pyperclip', msg="It is needed to be able to use clipboard")

docs/source/changelog.rst

@@ -2,9 +2,60 @@
 
 Change log
 **********
+0.16.3 (Thu May 12 2022)
+========================
+
+Bug Fix
+-------
+
+-  No change for a PR to trigger release
+   `#6692 <https://github.com/datalad/datalad/pull/6692>`__
+   (`@yarikoptic <https://github.com/yarikoptic>`__)
+-  Sanitize keys before checking content availability to ensure correct
+   value for keys with URL or custom backend
+   `#6665 <https://github.com/datalad/datalad/pull/6665>`__
+   (`@adswa <https://github.com/adswa>`__
+   `@yarikoptic <https://github.com/yarikoptic>`__)
+-  Change a key-value pair in drop result record
+   `#6625 <https://github.com/datalad/datalad/pull/6625>`__
+   (`@mslw <https://github.com/mslw>`__)
+-  Link docs of datalad-next
+   `#6677 <https://github.com/datalad/datalad/pull/6677>`__
+   (`@mih <https://github.com/mih>`__)
+-  Fix ``GitRepo.get_branch_commits_()`` to handle branch names
+   conflicts with paths
+   `#6661 <https://github.com/datalad/datalad/pull/6661>`__
+   (`@mih <https://github.com/mih>`__)
+-  OPT: AnnexJsonProtocol - avoid dragging possibly long data around
+   `#6660 <https://github.com/datalad/datalad/pull/6660>`__
+   (`@yarikoptic <https://github.com/yarikoptic>`__)
+-  Remove two too prominent create() INFO log message that duplicate
+   DEBUG log and harmonize some other log messages
+   `#6638 <https://github.com/datalad/datalad/pull/6638>`__
+   (`@mih <https://github.com/mih>`__
+   `@yarikoptic <https://github.com/yarikoptic>`__)
+-  Remove unsupported parameter create_sibling_ria(existing=None)
+   `#6637 <https://github.com/datalad/datalad/pull/6637>`__
+   (`@mih <https://github.com/mih>`__)
+-  Add released plugin to .autorc to annotate PRs on when released
+   `#6639 <https://github.com/datalad/datalad/pull/6639>`__
+   (`@yarikoptic <https://github.com/yarikoptic>`__)
+
+Authors: 4
+----------
+
+-  Adina Wagner (`@adswa <https://github.com/adswa>`__)
+-  Michael Hanke (`@mih <https://github.com/mih>`__)
+-  Micha Szczepanik (`@mslw <https://github.com/mslw>`__)
+-  Yaroslav Halchenko (`@yarikoptic <https://github.com/yarikoptic>`__)
+
+--------------
+
 0.16.2 (Thu Apr 21 2022)
 ========================
 
+.. _bug-fix-1:
+
 Bug Fix
 -------
 
@@ -732,7 +783,7 @@ Authors: 11
 0.15.6 (Sun Feb 27 2022)
 ========================
 
-.. _bug-fix-1:
+.. _bug-fix-2:
 
 Bug Fix
 -------
@@ -767,7 +818,7 @@ Enhancement
    `#6364 <https://github.com/datalad/datalad/pull/6364>`__
    (`@adswa <https://github.com/adswa>`__)
 
-.. _bug-fix-2:
+.. _bug-fix-3:
 
 Bug Fix
 -------
@@ -818,7 +869,7 @@ Authors: 5
 0.15.4 (Thu Dec 16 2021)
 ========================
 
-.. _bug-fix-3:
+.. _bug-fix-4:
 
 Bug Fix
 -------
@@ -903,7 +954,7 @@ Authors: 6
 0.15.3 (Sat Oct 30 2021)
 ========================
 
-.. _bug-fix-4:
+.. _bug-fix-5:
 
 Bug Fix
 -------
@@ -1007,7 +1058,7 @@ Authors: 7
 0.15.2 (Wed Oct 06 2021)
 ========================
 
-.. _bug-fix-5:
+.. _bug-fix-6:
 
 Bug Fix
 -------
@@ -1074,7 +1125,7 @@ Authors: 5
 0.15.1 (Fri Sep 24 2021)
 ========================
 
-.. _bug-fix-6:
+.. _bug-fix-7:
 
 Bug Fix
 -------
@@ -1487,7 +1538,7 @@ Tests
 0.14.8 (Sun Sep 12 2021)
 ========================
 
-.. _bug-fix-7:
+.. _bug-fix-8:
 
 Bug Fix
 -------
@@ -1580,6 +1631,8 @@ Tests
    message `#5894 <https://github.com/datalad/datalad/pull/5894>`__
    (`@yarikoptic <https://github.com/yarikoptic>`__)
 
+.. _authors-4-1:
+
 Authors: 4
 ----------
 
@@ -1593,7 +1646,7 @@ Authors: 4
 0.14.7 (Tue Aug 03 2021)
 ========================
 
-.. _bug-fix-8:
+.. _bug-fix-9:
 
 Bug Fix
 -------
@@ -1657,7 +1710,7 @@ Tests
    `#5786 <https://github.com/datalad/datalad/pull/5786>`__
    (`@yarikoptic <https://github.com/yarikoptic>`__)
 
-.. _authors-4-1:
+.. _authors-4-2:
 
 Authors: 4
 ----------
@@ -1693,7 +1746,7 @@ Authors: 2
 0.14.5 (Mon Jun 21 2021)
 ========================
 
-.. _bug-fix-9:
+.. _bug-fix-10:
 
 Bug Fix
 -------
@@ -1807,7 +1860,7 @@ Tests
    `#5651 <https://github.com/datalad/datalad/pull/5651>`__
    (`@kyleam <https://github.com/kyleam>`__)
 
-.. _authors-4-2:
+.. _authors-4-3:
 
 Authors: 4
 ----------

setup.py

@@ -51,7 +51,7 @@
         'python-gitlab',     # required for create-sibling-gitlab
     ],
     'misc': [
-        'argcomplete',       # optional CLI completion
+        'argcomplete>=1.12.3',  # optional CLI completion
         'pyperclip',         # clipboard manipulations
         'python-dateutil',   # add support for more date formats to check_dates
     ],

tools/convert-git-annex-layout

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -eu
+
+to="$1"
+
+echo "Initial fsck:"
+git annex fsck --fast 2>&1 | python -m tqdm --null
+echo "Going through objects: "
+for f in .git/annex/objects/*/*/*; do
+    key=$(basename $f)
+    keydir=$(dirname $f)
+    newhashdir=$(git annex examinekey --format="\${$to}" "$key")
+    targetdir=".git/annex/objects/$newhashdir"
+    test -n "$newhashdir"
+    if [ "$keydir" = "${targetdir%/}" ]; then
+        continue
+    fi
+    echo "  $f -> $newhashdir"
+    # This was a wrong assumption - there could be multiple
+    # keys in the same directory so we might have it already.
+    # But I still feel we might need some test here
+    #if test -e "$targetdir"; then
+    #    echo "$targetdir already exists"
+    #    exit 1
+    #fi
+    mkdir -p "$(dirname $targetdir)"
+    mv "$keydir" "${targetdir%/}"
+done
+echo "Final fsck:"
+git annex fsck --fast 2>&1 | python -m tqdm --null