BF: fix authentication + download from S3 for NDA #4824
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## maint #4824 +/- ##
==========================================
- Coverage 89.68% 89.46% -0.22%
==========================================
Files 288 288
Lines 40362 40378 +16
==========================================
- Hits 36197 36123 -74
- Misses 4165 4255 +90
Continue to review full report at Codecov.
|
yarikoptic
changed the title
yarikoptic
added
the
merge-if-ok
failed travis debug run is unrelated test_ria_basics fail I think we sawERROR: datalad.distributed.tests.test_ria_basics.test_binary_data('datalad-test',)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/tests/utils.py", line 271, in _wrap_skip_ssh
return func(*args, **kwargs)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/tests/utils.py", line 731, in _wrap_with_tempfile
return t(*(arg + (filename,)), **kw)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/tests/utils.py", line 731, in _wrap_with_tempfile
return t(*(arg + (filename,)), **kw)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/distributed/tests/test_ria_basics.py", line 468, in _test_binary_data
ds.download_url(url, path=file, message="Add DICOM file from github")
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/distribution/dataset.py", line 503, in apply_func
return f(**kwargs)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/interface/utils.py", line 494, in eval_func
return return_func(generator_func)(*args, **kwargs)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/interface/utils.py", line 482, in return_func
results = list(results)
File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad/interface/utils.py", line 469, in generator_func
msg="Command did not complete successfully")
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'download_url',
'message': "'ConsoleLog' object has no attribute 'yesno' "
'[__init__.py:__getattribute__:101]',
'path': '/tmp/datalad_temp__test_binary_datajhq0ejj_/dicomfile',
'status': 'error',
'type': 'file'}] |
kyleam
approved these changes
Aug 31, 2020
| @@ -158,3 +158,12 @@ def test_boto_host_specification(tempfile): | |||
| with swallow_outputs(): | |||
| providers.download(url_dandi1, path=tempfile) | |||
| assert_equal(md5sum(tempfile), '97f4290b2d369816c052607923e372d4') | |||
|
|
|||
|
|
|||
| def test_restricted_bucket_on_NDA(): | |||
There was a problem hiding this comment.
So, this only runs on your local machine, correct?
There was a problem hiding this comment.
yes unfortunately. NDA access, and to this specific dataset is required and it is a not that easy procedure to get. And with their specific settings for the bucket - the value from the test is really in talking to NDA.
May be at some point we could setup a github workflow worker which would sit on "trusted" infrastructure and has access to the credentials, but then we might need to (as I think I did with buildbot in the past) to make it run tests only if PRs submitted by trusted github users or a dedicated label is added to signal that PR diff is good (no malicious code which would expose secrets) to be tested on
For me it still fails, but may be because ATM I have no access to anything there?
…cessDeniedError That should cause UI to ask for new credentials
…orbidden Happens eg. for NDA which has a VERY restricted ACL: you cannot HEAD the bucket etc. So initial HEAD boto issues for the bucket gets "Forbidden".
Comment in the code diff outlines the situation: in buckets with minimal ACL you cannot download a key by specifying versionId, although boto does discover it while first obtaining the key instance.
…t any sensitive IMHO data
I think they are not yet used. Could be re-added whenever we start testing against development instance
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Although it is bug fix in nature, I am ok to reposition it against
masterif so deemed better.