-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SAM 0.17+ prompts for MFA credentials on every request when running API locally #1375
Comments
Hmm.. OK on closer inspection, dropping down to 0.19.0 did not fix the issue. Neither has going down to 0.18.0. It used to work as expected. Last week I upgraded to sam cli 0.20.1 and presumed it was the culprit. Maybe it's something else? (I can't guarantee what version I was on before, I think it was 0.18.0) Is there anything I can do to work around this? Having to type the MFA code in every time is driving me crazy. |
OK, I went back to sam 0.16 and the problem has now cleared up. So either in 0.17 or 0.18 it seems to have changed behaviour. As a side note - I've also noticed that if I Ctrl+C to quit the process whilst it's asking me for an MFA then it renders the whole terminal unusable. (Well, by that I mean I can't see what I'm typing - but I can actually type blindly in and it works). But, anyway... any idea what's changed in SAM that would affect this behaviour? Any workaround or do I just need to stick to version 0.16? |
Interesting is this just a windows 10 issue, do you see it on any other platform? |
I've only tried it on Ubuntu WSL on Win10. I'll try messing around with it on Win10 itself, and maybe on my Mac. Just need to find a few spare minutes from somewhere first. I also need to try 0.17, but frankly yesterday I was so relieved to put a stop to it I didn't want to mess with it any more! |
Tried in normal Win10 command prompt and same behaviour on 0.21.0. (IE being asked over and over again)l |
Sorry, only just had a few moments spare. I've tried this at home using OSX Mojave. Sam 0.16 - I only get prompted for MFA credentials once. So looks like it's not OS specific, and it was introduced in version 0.17. |
Any thoughts? It means we can't upgrade beyond 0.16 unless we are going to spend the entire day entering in mfa credentials. |
Saw 0.22 has been released, so tried it there as well but no difference. Rolling back to 0.16 again. I really hope this can be resolved. |
@johnc44 Thanks for the all the information you have provided and keeping this up-to-date. I am looking at the changes we did from 0.16 to 0.17, one PR stood out to me: #1059. I have not verified yet, but my hunch is this line: https://github.com/awslabs/aws-sam-cli/pull/1059/files#diff-71d3cd43730ae0d5ad972d2db90e44c2R213 Looking at the history, we originally had it the way it was today: e607d66fbaa376f8d2f42a5c830be8aa4fb873eb but was changed to match what the comment suggested: f88fb592e072314a4902e1f00489329a58c79519. Later we noticed some race conditions happening so we reverted in #1059 back to what is was originally (in commit e607d66). I need do get some things setup to verify this but this is my best guess, as it would match the need to get creds on every invoke. |
@johnc44 I am struggling getting my MFA setup and working. Can you try two things, when you get some time:
I am still trying to get this MFA setup, but would at least give me the confidence my hunch is correct. |
@jfuss I am having the same problem as @johnc44 so I thought I would give you a hand with this. I created a new project and installed v0.7.0 with pip |
@sosdmike Thank you! After talking with some team members, I think the always creating the session is the culprit here. I think what we want to do is cache this somehow. |
After some investigation here are some deeper details: Boto3 sessions are not thread safe: https://github.com/awslabs/aws-sam-cli/blob/develop/samcli/commands/local/lib/local_lambda.py#L220 this is due to boto3 sessions dependency on urllib which is (also) not thread safe. This is why we get credentials every time for invoke. For most of the use-cases, this isn't a problem but for the occasions of multiple requests on
Option 3 isn't a really valid option, as this really hinders customers using MFA with Option 2 seems like the best out of the remaining options. To be done right (I think), we would need a way to invalidate the creds and get new ones for long sessions. Otherwise, we will be mounting expired credentials into the container and the only way to refresh would to be rerun the command (not the end of the world but not the greatest experience). I would have to dig deeper to understand if there is timestamp information we could use for this. It's worth calling out that this would only affect debugging/invoking sessions that are longer than the life of the credentials. There are trade-offs on both Option 1 and Option 2. To make sure we feel we are making the right decision, I will bring this up with the team and circle back here. |
Pre 0.17, the credentials would expire after a bit, and I'd have to run sam local again. It's very much less annoying than being prompted every time, so I'd personally be quite happy if it reverted to that behaviour. |
Addresses Issue aws#1375. Due to Boto3.session.Session objects not being thread safe, we create a Session object on every invoke to get credentials to mount into the container. This works well except for customers using MFA. When using MFA, this dehavior requires customers to provide MFA code/pin on every invoke. This creates much longer debug sessions for customers and pauses invokes until the MFA code/pin is provided. This PR changes that behavior to one we had between version v0.8.0 and v0.16.0, which is caching the creds/session to be reused on other invokes until the credentials expire. Once the credentials expire, customers will need to restart the command (which was the same behavior on version 0.8.0 to 0.16.0.
) Addresses Issue #1375. Due to Boto3.session.Session objects not being thread safe, we create a Session object on every invoke to get credentials to mount into the container. This works well except for customers using MFA. When using MFA, this dehavior requires customers to provide MFA code/pin on every invoke. This creates much longer debug sessions for customers and pauses invokes until the MFA code/pin is provided. This PR changes that behavior to one we had between version v0.8.0 and v0.16.0, which is caching the creds/session to be reused on other invokes until the credentials expire. Once the credentials expire, customers will need to restart the command (which was the same behavior on version 0.8.0 to 0.16.0.
Patched was released in 0.38.0. Closing |
@jfuss this seems to be happening again in 0.40.0. Running |
@silvait What version of SAM CLI are you running? In the local suite of commands, we cache for the life time of the command or when the creds expire with MFA. I can't remember if this prompts after the MFA expires or not. |
@jfuss I'm using |
While a general solution isn't created, I would suggest using aws-vault A very quick usage guide (works for me): Place a entry in your aws config
Then on your command line:
This will launch a new bash session and request your MFA (my case, keeps it in the identity manager), and will keep using it for as long as it is active. |
Description
We use Multi-Factor Authentication to access our AWS accounts.
In previous versions (0.19.0), SAM CLI would prompt for my MFA token when I first invoked the lambda, and would then retain those credentials for future requests. Eventually they would expire and I'd have to quit/restart to get it to work again, but it was quite workable.
In 0.20.1, however, I am asked for my MFA credentials on literally every request. This makes it completely unusable for me, so I've had to downgrade again.
Steps to reproduce
Using Ubuntu WSL:
export AWS_PROFILE=
aws local start-api -n environment.json -p 3100
Then hit the API. It sits there twiddling its thumbs. Switch back to Ubuntu and it's asking for the MFA code. Enter it in, everything works.
Hit the API a second time. Again, it sits there thinking about it. Switch back to Ubuntu and it's asking for the MFA again. This is not the case in 0.19.0 which would remember my token.
Observed result
Asked for MFA code on every single request.
Expected result
MFA token should be cached between requests.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
sam --version
: 0.20.1The text was updated successfully, but these errors were encountered: