Skip to content

Commit

Permalink
BF: s3 - do not download versioned key if no versionId was in URL
Browse files Browse the repository at this point in the history
Comment in the code diff outlines the situation:  in buckets with minimal ACL
you cannot download a key by specifying versionId, although boto does discover
it while first obtaining the key instance.
  • Loading branch information
yarikoptic committed Aug 27, 2020
1 parent 272b882 commit 5620cbc
Showing 1 changed file with 14 additions and 0 deletions.
14 changes: 14 additions & 0 deletions datalad/downloaders/s3.py
Expand Up @@ -238,6 +238,20 @@ def get_downloader_session(self, url, **kwargs):
# Consult about filename
url_filename = get_url_straight_filename(url)

if 'versionId' not in params and key.version_id:
# boto adds version_id to the request if it is known present.
# It is a good idea in general to avoid race between moment of retrieving
# they key information and actual download.
# But depending on permissions, we might be unable (like in the case with NDA)
# to download a guaranteed version of the key.
# So we will just download the latest version (if still there)
# if no versionId was specified in URL
# Alternative would be to make this a generator and generate sessions
# but also remember if the first download succeeded so we do not try
# again to get versioned one first.
key.version_id = None
# TODO: ask NDA to allow download of specific versionId?

return S3DownloaderSession(
size=target_size,
filename=url_filename,
Expand Down

0 comments on commit 5620cbc

Please sign in to comment.