Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow AWS temporary credentials with session tokens #958

Closed
wants to merge 1 commit into from

Conversation

@rvl
Copy link

@rvl rvl commented May 23, 2018

This allows passing the session token of AWS temporary credentials to boto. This helps when the AWS credentials came from assuming a role or using a MFA token.

I just hacked this up now and it works for my ec2/route53 test. If the approach seems right to you, I can add some documentation and check that the use case of #938 can work by specifying an empty access_key_id.

@rvl rvl force-pushed the rvl:aws-temporary-creds branch from 6ff314c to ab86e52 May 23, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request May 24, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request May 30, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jun 18, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jun 26, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jun 27, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jun 28, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jul 3, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jul 20, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Jul 23, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Aug 3, 2018
rvl added a commit to input-output-hk/iohk-ops that referenced this pull request Aug 13, 2018
Fixes #938
@rvl rvl force-pushed the rvl:aws-temporary-creds branch from ab86e52 to ba5cfd7 Sep 24, 2018
Copy link

@Nekroze Nekroze left a comment

I 100% must use temporary creds with nixops and so require this to be merged so I figured i would review. Looks ok to me fwiw.

@Nekroze
Copy link

@Nekroze Nekroze commented May 22, 2019

Can confirm this works with creds on a temporarily assumed role.

@Nekroze
Copy link

@Nekroze Nekroze commented May 22, 2019

While this works for deployment I suspect there may be a codepath that has been missed as I get the same 401 error I got without this patch when trying to delete a ec2 keypair:

  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/bin/.nixops-wrapped", line 990, in <module>
    args.op()
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/bin/.nixops-wrapped", line 432, in op_destroy
    wipe=args.wipe)
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1140, in destroy_resources
    self.run_with_notify('destroy', lambda: self._destroy_resources(include, exclude, wipe))
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1045, in run_with_notify
    f()
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1140, in <lambda>
    self.run_with_notify('destroy', lambda: self._destroy_resources(include, exclude, wipe))
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1134, in _destroy_resources
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.resources.values(), worker_fun=worker)
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 1127, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/nix/store/kavpbizqsq60i65r7c63cy2zpbs1sn9x-python2.7-nixops-1.6.1pre0_abcdef/lib/python2.7/site-packages/nixops/resources/ec2_keypair.py", line 127, in destroy
    self._conn.delete_key_pair(self.keypair_name)
  File "/nix/store/vi27lqbnvq7xfskk9nn9ap7zinq28949-python2.7-boto-2.47.0/lib/python2.7/site-packages/boto/ec2/connection.py", line 2898, in delete_key_pair
    return self.get_status('DeleteKeyPair', params, verb='POST')
  File "/nix/store/vi27lqbnvq7xfskk9nn9ap7zinq28949-python2.7-boto-2.47.0/lib/python2.7/site-packages/boto/connection.py", line 1227, in get_status
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>redacted-redacted-redacted-redacted-redacted</RequestID></Response>```
@rvl
Copy link
Author

@rvl rvl commented May 23, 2019

Hi @Nekroze, I hope you find this PR useful. We use it on a fork of nixops.

Looking at the connect() and destroy() functions of ec2_keypair.py, I can't see any reason why it shouldn't work.

Using the same temporary credentials with awscli, are you able to run aws ec2 delete-key-pair?

@Nekroze
Copy link

@Nekroze Nekroze commented May 23, 2019

Hi @Nekroze, I hope you find this PR useful. We use it on a fork of nixops.

Very, thanks for making it. I really hope it lands soon.

Using the same temporary credentials with awscli, are you able to run aws ec2 delete-key-pair?

Indeed I can, I am getting this on nixops check as well although I am not sure which resource/type is causing it.

@GlennS
Copy link
Contributor

@GlennS GlennS commented Feb 3, 2020

It seems quite important to get this merged.

Until then, Nixops is broken for AWS if you follow Amazon's recommended way of setting up your users and roles.

@Nekroze
Copy link

@Nekroze Nekroze commented Feb 3, 2020

It seems quite important to get this merged.

Sadly I do not think this branch is quite ready for merge, last I used it various operations still failed and did not seem to be utilizing the keys. From memory things like doing a check caused errors from the AWS API.

I have since switched to using Hashicorp Vault to generate users dynamically so I get a similar kind of thing to the STS sessions but nixops does not complain... so I am not so up to date with what works with this branch and what does not for now.

@grahamc
Copy link
Member

@grahamc grahamc commented Mar 26, 2020

Hello!

Thank you for this PR.

In the past several months, some major changes have taken place in
NixOps:

  1. Backends have been removed, preferring a plugin-based architecture.
    Here are some of them:

  2. NixOps Core has been updated to be Python 3 only, and at the
    same time, MyPy type hints have been added and are now strictly
    required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
release
. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work
on.

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it
. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.

Graham

@grahamc grahamc closed this Mar 26, 2020
@rvl
Copy link
Author

@rvl rvl commented Apr 27, 2020

Thanks @grahamc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.