-
Notifications
You must be signed in to change notification settings - Fork 10.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for temporary security credential in AWS auth #5210
Conversation
Please fix the test failures (and ideally run them locally before submitting the changes). Also, if the doc says using this is discouraged, do we need this added? |
Sorry, have some problem running the tests locally. It takes too much time to install dependencies (stuck at
We need it. It's discouraged because this kind of credential is short term. But what if a user can't get a normal (long term and never expiring) credential? I'll tweak the words in the doc later to make it clear. |
4ab6ca7
to
fa66c36
Compare
Codecov Report
@@ Coverage Diff @@
## master #5210 +/- ##
==========================================
- Coverage 88.42% 88.42% -0.01%
==========================================
Files 162 162
Lines 10522 10528 +6
Branches 1521 1522 +1
==========================================
+ Hits 9304 9309 +5
Misses 944 944
- Partials 274 275 +1
|
You seem to both agree that it’s discouraged. I failed to find such information in the 2 linked references. But even if it is discouraged, as long as it is not deprecated (i.e. as long as they do not ask people to stop using it, and instead ask them to consider alternatives first), +1 to add support. |
I meant the doc in the change itself. |
Temporary credential (e.g. setting The problem of a temporary credential is that it will expire (you can set it when creating one). This introduce a risk that your spider may haven't got the scraping done when the credential expires. |
I kind of expect people to realize this based on upstream documentation, but I guess it would be OK to include it as a warning or note. In any case, I would avoid the word “discouraged”, and simply state the facts: if the credentials expire in the middle of a crawl, . I’m not sure what actually happens, if the stuff is simply not delivered of if such an error can stop the spider. |
This is what happens when you don’t review the changes before commenting. Sorry! |
Co-authored-by: Adrián Chaves <adrian@chaves.io>
a7f4730
to
8e7b96d
Compare
Support temporary security credentials by adding a new setting
AWS_SESSION_TOKEN
. S3 client or credentials creation is done bySession.create_client()
in pipelines, feed exporterCredentials.__init__()
inS3DownloadHandler
Both of the methods from
botocore
accept another token parameter.AWS_SESSION_TOKEN
is only required by temporary security credentials, which is short term.References about the
AWS_SESSION_TOKEN