test_ria_basics.py::test_binary_data: wants interaction

[yarikoptic]

Prior issue (closed as addressed) for this test was #4371 , suggests something about Authorization (didn't check in detail) hence might indeed relate to the desire of interaction.

Happened in the course of testing #7126 which AFAIK does not relate, happened on travis in a single run of the heavy debug logging etc:

=========================== short test summary info ============================
FAILED ../datalad/distributed/tests/test_ria_basics.py::test_binary_data - datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'download_url',
  'error_message': "'ConsoleLog' object has no attribute 'yesno'",
  'exception': 'ConsoleLog' object has no attribute 'yesno' [download_url.py:__call__:202,base.py:download:527,base.py:access:219,base.py:_handle_authentication:286,base.py:_enter_credentials:319,__init__.py:__getattribute__:101,base.py:access:174,base.py:_download:427,http.py:get_downloader_session:622,http.py:check_response_status:124],
  'exception_traceback': '[download_url.py:__call__:202,base.py:download:527,base.py:access:219,base.py:_handle_authentication:286,base.py:_enter_credentials:319,__init__.py:__getattribute__:101,base.py:access:174,base.py:_download:427,http.py:get_downloader_session:622,http.py:check_response_status:124]',
  'message': "AttributeError('ConsoleLog' object has no attribute 'yesno')",
  'path': '/tmp/datalad_temp__test_binary_datathev6l40/dicomfile',
  'status': 'error',
  'type': 'file'}]
= 1 failed, 1162 passed, 46 skipped, 3 xfailed, 17 warnings in 1023.67s (0:17:03) =
The command "http_proxy= PATH=$PWD/../tools/coverage-bin:$PATH $PYTEST_WRAPPER python -m pytest "${PYTEST_OPTS[@]}" -c ../tox.ini -n 2 -m "${PYTEST_SELECTION:+$PYTEST_SELECTION_OP($PYTEST_SELECTION) and }not(turtle)" --doctest-modules --cov=datalad --cov-report= --pyargs $TESTS_TO_PERFORM" exited with 1.
0.00s$ cd ..

Seems to be relatively frequent encounter, and only on travis (looked only at this month):

(git)smaug:/mnt/datasets/datalad/ci/logs/2022/10[master]git
$> git grep '^FAILED.*test_binary_data'
11/pr/7068/08876f3/travis-14700-failed/9.txt:FAILED ../datalad/distributed/tests/test_ria_basics.py::test_binary_data - da...
12/pr/7002/16a162d/travis-14714-failed/8.txt:FAILED ../datalad/distributed/tests/test_ria_basics.py::test_binary_data - da...
19/pr/7094/a88c6e1/travis-14761-failed/2.txt:FAILED ../datalad/distributed/tests/test_ria_basics.py::test_binary_data - da...
19/push/master/1de6d08/travis-14764-failed/2.txt:FAILED ../datalad/distributed/tests/test_ria_basics.py::test_binary_data - da...
21/push/maint/e94a49e/travis-14772-failed/4.txt:FAILED ../distributed/tests/test_ria_basics.py::test_binary_data - datalad.su...
24/push/maint/969bb82/travis-14794-failed/9.txt:FAILED ../datalad/distributed/tests/test_ria_basics.py::test_binary_data - da...
git grep '^FAILED.*test_binary_data'  4.04s user 3.52s system 1016% cpu 0.743 total

and seems to happen not only in the "heavy debug" run

[bpoldrack]

Yes, saw this, too. Looking into it.

[bpoldrack]

First things first: What is failing here is a call to datalad download-url in the test's setup. RIA/ORA is not yet involved at this point.

So, the test is trying to download https://github.com/datalad/example-dicom-functional/blob/master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586 and apparently (occasionally) this is leading to an AccessDeniedError in BaseDownloader.access(), which then gets into the business of querying for authentication.
Two issues from my POV:

• occasional AccessDeniedError seems odd. Need to figure what the error really says when it happens.
• downloaders/authenticators shouldn't crash with the AttributeError, but give a proper message. We know we are non-interactive. The ui backend should probably implement the yesno in order to fail properly.

[bpoldrack]

And here is the issue: Access to https://github.com/datalad/example-dicom-functional/blob/master/dicoms/MR.1.3.46.670589.11.38317.5.0.4476.2014042516042547586 has failed: status code 429

It's "too many requests" being treated as "Access denied" by the downloader.

[yarikoptic-gitmate]

Issue fixed in 0.17.9