Fix test failures on macOS

[jwodder]

Fixes #4941.

[yarikoptic]

eh, according to coverage we are not even testing this flag "directly" (well - git-annex would use it if it saw it available at "compile time") -- could you please tune up or extend some test for create_sibling so it tests with preserve_attrs=True?

[jwodder]

There does not appear to be any way to tell create_sibling to use preserve_attrs here. It appears that no code that calls this method ever sets preserve_attrs=True.

[yarikoptic]

oh, indeed. Could you please craft a dedicated to _RunnerAdapter unittest and test put with preserve_attrs? then we would be sure it would work whenever we need to use it.

[yarikoptic]

ha ha, in the hindsight of rare OSX experience, fixes make so much sense ;)

[yarikoptic]

FTR, travis fail is unrelated AFAIK NFS gotcha:

Traceback (most recent call last):
  File "/opt/python/3.6.7/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers
    finalizer()
  File "/opt/python/3.6.7/lib/python3.6/multiprocessing/util.py", line 186, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs00000000001f837d0000001b'
The command "http_proxy= PATH=$PWD/../tools/coverage-bin:$PATH $NOSE_WRAPPER `which nosetests` "${NOSE_OPTS[@]}" -A "$NOSE_SELECTION_OP($NOSE_SELECTION) and not(turtle)" --with-doctest --with-cov --cover-package datalad --logging-level=INFO $TESTS_TO_PERFORM" exited with 1.

[jwodder]

@yarikoptic Current status: The remaining failing tests are the following:

• datalad.distribution.tests.test_get.test_get_recurse_dirs
• datalad.distribution.tests.test_install.test_install_recursive_repeat
• datalad.interface.tests.test_rerun.test_run_inputs_outputs
• datalad.metadata.tests.test_aggregation.test_aggregate_with_unavailable_objects_from_subds
• datalad.metadata.tests.test_base.test_aggregation
• datalad.distribution.tests.test_get.test_get_multiple_files
• datalad.customremotes.tests.test_archives.test_basic_scenario

Except for the last one, they all appear to involve two local datalad repositories in temporary directories, with one repo copied to another with datalad.api.install; they then fail on an invocation of git annex get in the second repo with the messages "Unable to access these remotes: origin" and "/path/to/origin/.git/annex: createDirectory: does not exist (No such file or directory)".

The last test involves a repo with a datalad-archives remote, which it is unable to fetch from.

[yarikoptic]

Thank you @jwodder ! May be tomorrow we could setup some interactive jitsi session (what about 1pm or 2pm?) and get that one resolved. One possibility comes to mind is "symlinks" -- IIRC Mac uses them for either HOME or TMPDIR and may be somehow that is relevant. An approach to debug it is to run via DATALAD_TESTS_TEMP_KEEP=1 python -m nose -s -v --pdb so you would fall into pdb when test fails and still have those temporary directories available so you could just open shell and go "inspect" in real life on WTF it fails to find it (wrong path? wrong information in git-annex?)

[jwodder]

@yarikoptic I'm open to meeting at 1 PM, but given the note at the bottom of this comment, I don't think it'll be as productive as you expect.

One possibility comes to mind is "symlinks" -- IIRC Mac uses them for either HOME or TMPDIR and may be somehow that is relevant.

$HOME doesn't use symlinks. $TMPDIR doesn't directly use them either, but $TMPDIR is usually under /var, which is a symlink to private/var (note relative path).

An approach to debug it is to ...

The failing tests I listed are for when the tests are run under GitHub Actions. When I run the tests on my machine (macOS 10.13.6, Python 3.9.0, git 2.29.2, git-annex 8.20201007) with the fixes so far, I get only one failure: datalad.support.tests.test_locking.test_lock_if_check_fails fails because it's trying to pickle a function nested inside another function (which makes me wonder how that test is passing in other environments).

[yarikoptic]

The failing tests I listed are for when the tests are run under GitHub Actions.

I am a bit confused: macOS runs here are all green. So where are the failures?

I get only one failure: datalad.support.tests.test_locking.test_lock_if_check_fails fails because it's trying to pickle a function nested inside another function (which makes me wonder how that test is passing in other environments).

If you see how to RF to make it pass locally for you, that would be great. Pity that it doesn't trigger anywhere else indeed, but also it might be a bit of a concern -- in particular with #5022 where I do rely on similar closures, I was not expecting anything to be pickled (at least for that multithreaded execution support). Here, I guess for interprocess locking done by fasteners, pickling might be needed, and I was not aware of that, so may be some uses of the tested there helper would cause similar problems :-/?

NFS issue is "known": #4958

[jwodder]

I am a bit confused: macOS runs here are all green. So where are the failures?

The failures show up when all tests are enabled, such as in #5109.

[jwodder]

@yarikoptic I've refactored the local function test so that it passes.

[yarikoptic]

Ah, ok. So let's take this one from draft and merge if no new surprises emerge. The rest coughed be fixed in that other pr

[yarikoptic]

changes were sound and no concerns were raised, so let's proceed