BF: annexrepo: Adjust overly selective is_special_annex_remote()

[kyleam]

To decide if a remote is an annex special remote, we check whether
'annex-externaltype' or 'annex-webdav' is configured for the remote.
This mis-classifies nearly all internal special remotes (anything but
webdav) as an ordinary remote because those don't have
'annex-externaltype' configured.

There's no common annex- option for special remotes. As the
definition of annex's findSpecialRemotes shows, each special remote is
expected to have at least an annex-TYPE option. We could keep a list
of known internal special remotes and look for that or externaltype,
but the list of internal special remotes is a moving target.

findSpecialRemotes' documentation mentions that "special remotes don't
have a configured url", so let's instead identify them based on not
having a URL and having some annex- option aside from annex-uuid and
annex-ignore, two particularly common non-special-remote options.

Given that all valid Git remotes have a configured URL, this should be
a reliable classification for correctly configured remotes. It's of
course possible to get mis-classifications if the user either removes
or adds a URL to a remote's section.

Another option would be to rely on get_special_remotes(), which calls
git cat-file git-annex:remote.log to get the special remotes, but
that'd be a change in behavior because it also considers remotes that
aren't enabled. It'd also be slower:

# This patch
% python -m timeit -s "from datalad.support.annexrepo import AnnexRepo; ar = AnnexRepo('.')"
  "ar.is_special_annex_remote('origin')"
10000 loops, best of 3: 143 usec per loop

# Using get_special_remotes() in is_special_annex_remote()
% python -m timeit -s "from datalad.support.annexrepo import AnnexRepo; ar = AnnexRepo('.')"
  "ar.is_special_annex_remote('origin')"
100 loops, best of 3: 8.82 msec per loop

Fixes #3497.

[kyleam]

@leej3, the patch has changed a bit from the one posted at gh-3497, so it'd be great if you could confirm that this PR fixes your issue as well. Thanks!

[leej3]

Confirmed. This works too. Thanks

[yarikoptic]

Thank you! Is that such a frequent call to do that timing is critical? I would have gone for get_special_remotes() + taking intersection with enabled (present in .git/config) remotes so that there is no guess work

[kyleam]

Is that such a frequent call to do that timing is critical?

To my mind the question isn't whether it is critical (it's not, and very few things are), but what the extra cost gains us.

Within datalad itself we call this when looping over remotes and when traversing dataset hierarchies, so by not calling get_special_remotes() within is_special_annex_remote() we're avoiding some number of git cat-file calls that depends on the repository state and dataset layout. (Or alternatively we're avoiding reworking the code and maybe interface to use some kind of cache.)

I would have gone for get_special_remotes() + taking intersection with enabled (present in .git/config) remotes so that there is no guess work

So AFAICS the main benefit there is that, if the config file has an invalid Git remote without a URL and with some annex-* options, we don't mis-classify it as a special remote. A Git operation with the invalid remote would fail anyway, and I don't see an issue with is_special_annex_remote() assuming a valid remote.

Do you think there are any valid remotes where the intersection approach would give a different answer?

I was able to come up with one: gcrypt. The problem there is that the remote is actually a normal remote too. So while technically is_special_annex_remote() should return true, doing so would make the two current callers of is_special_annex_remote() behave in an unintended way because those callers assume a remote is either special or not.

[yarikoptic]

Similar case I guess might happen with a git special remote which we establish when using some ssh+http accessible server as a common data src? (--as-common-datasrc) Indeed semantic there is not clear -- those are "hybrid" remotes. In the long run I wonder if we should RF into get_remotes(..., type=(git|special|hybrid)) or get_remote_types or get_remote_features to make it possible to disambiguate more explicitly?

[kyleam]

Indeed semantic there is not clear -- those are "hybrid" remotes. In the long run I wonder if we should RF into get_remotes(..., type=(git|special|hybrid)) or get_remote_types or get_remote_features to make it possible to disambiguate more explicitly?

I certainly think it should be fixed somehow, but that seems well outside this PR.

[yarikoptic]

Sure, let's make things better incrementally ;-) Cheers!