Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expiration time for exported objects should be configurable #2053

Closed
lmsurpre opened this issue Mar 11, 2021 · 2 comments
Closed

Expiration time for exported objects should be configurable #2053

lmsurpre opened this issue Mar 11, 2021 · 2 comments
Assignees
Labels
bulk-data enhancement New feature or request

Comments

@lmsurpre
Copy link
Member

lmsurpre commented Mar 11, 2021

Is your feature request related to a problem? Please describe.
In #882 we added support for using presigned URLs for clients to fetch the resources.
Currently the expiration period for these is 24 hours (86400 seconds).

Besides this, we also support setting an ACL policy for objects (to make them temporarily public). I believe that is currently hard-coded to 2 hours.

Describe the solution you'd like
At least the presigned expiration period should be configurable.
I think the ACL one can stay at 2 hours because I want to mark that deprecated anyway (#2054).

Describe alternatives you've considered
Continue hard-coding it.

Acceptance Criteria
At least one acceptance criteria is included.

  1. GIVEN a config with presigned=false and isPublic=true and an expiration time of 10 minutes
    WHEN an export is performed
    AND more than 10 minutes elapses
    THEN any existing download request should succeed
    AND the user should no longer be able to start new downloads from the provided URLs

  2. GIVEN a config with presigned=true and isPublic=false and an expiration time of 10 minutes
    WHEN an export is performed
    AND more than 10 minutes elapses
    THEN any existing download request should succeed
    AND the user should no longer be able to start new downloads from the provided URLs

Additional context
As mentioned in the docs:

Please note that IBM COS does not support expiration time for each single COS object, so please configure retention policy (e.g, 1 day) for the bucket if IBM COS is used.

@prb112
Copy link
Contributor

prb112 commented Mar 11, 2021

There are four spots that need to be updated to support this... DownloadUrl ConfigurationAdapter AbstractConfigurationAdapterImpl and the Users guide.

@prb112 prb112 self-assigned this Mar 30, 2021
@prb112 prb112 added this to the Sprint 2021-05 milestone Mar 30, 2021
prb112 added a commit that referenced this issue Mar 30, 2021
Signed-off-by: Paul Bastide <pbastide@us.ibm.com>
@lmsurpre
Copy link
Member Author

I set presignedExpiry to 3600 (1 hour) in our connectathon environment and kicked off a big export.
After completion, I used the presigned urls from the $bulkdata-status operation to download the files.
At 59 minutes, I kicked off a bunch of donwloads.
While they were downloading, I kicked off some more after 60 minutes.
The ones I kicked off after 60 minutes failed as expected (ExpiredRequest) while the ones I kicked off at 59 minutes completed with success (even after the 60 minute mark).

I then invoked the $bulkdata-status endpoint again and confirmed that I received new links which worked for downloading the content.

This all worked very well and I'm quite happy :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bulk-data enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants