Rewriting to use boto3

[takeda]

The way how my AWS accounts are setup (assumeRole + MFA) makes it impossible to use old version of nixops. I initially tried to add an option to specify a profile, but turns out that old boto doesn't seem to support this kind of setup (since it was introduced later on) and it mandated porting to boto3.

So this version works fine with assumeRole + MFA and I also added an option to use the same cache that aws command uses. That way you don't have to enter MFA every for every single command.

Currently all of the commands that I used appear to be working, these are:

• list
• create
• modify
• delete
• info
• check
• deploy
• send-keys
• destroy
• stop
• start
• reboot
• show-arguments
• show-physical
• ssh
• ssh-for-each
• scp
• mount
• rename
• backup
• backup-status
• clean-backups
• restore (you need to provide backup id, but it looks like that was unrelated to my change?).
• show-option
• list-generations
• show-console-output
• dump-nix-paths
• export
• edit

If you can, please test it (clone my branch and run nix-env -f release.nix -iA build.<your platform>) if there are still broken commands let me know otherwise perhaps this can be merged.

[shmish111]

I'm currently looking into this, can't guarantee anything at this point but thought I'd mention it so as to avoid duplicate effort.

[takeda]

any updates?

[shmish111]

Hi @takeda, unfortunately I decided not to do this as it was too much to take on at the time however I did do quite a bit of work and I believe a colleague is taking it on. Again, no guarantees but if anyone else is considering working on this it is worth getting together, I can put you in touch.

FYI I discovered that a lot of this was pretty simple and the authentication etc just works with boto3 however there was a lot of small details that were tricky and I also went down the wrong path of using the non-OO API where as I think the conversion would be easier to use the resources (the OO API).

[takeda]

@shmish111 the Resources API is incomplete and doesn't cover all calls (it's more about taking given resource and then obtaining information about it, or performing simple actions), when working on it I was preferring to use Resources, but in many places I absolutely had to use Client calls.

BTW to give some background for people not familiar. The Client is the real AWS API, the code there is actually dynamically generated. The Resources is created by hand, and internally is implemented using the Client calls. Very often one Resource call will issue multiple Client calls, Resources also don't implement all functionality and are more about inspecting/performing operations on existing resources.

[wavewave]

👍 @takeda Thanks for this!

[tewfik-ghariani]

I've tried to test the nixops-aws plugin using your branch but immediately ran into some issues related to the newly defined "profile" option.

Not only the default value should be adjusted, but you have to take into account the case whereby the "profile" is not used at all ( which is our case ).

error: The option profile' is used but not defined.
(use '--show-trace' to show detailed location information)
`

Thanks for all the efforts though!

[tewfik-ghariani]

The default value should be either an empty string or just "default" but not null value

error: The option value deployment.ec2.profile' in /nix/store/ydszllb0yml02kzi8m4l5sp5mk6z0h3i-nixops-aws-1.7pre0_abcdef/share/nix/nixops-aws/ec2.nix' is not of type string'.
`

[bhipple]

I'd consider upstreaming all of these changes and some of the mypy annotations in a separate no-op cleanup PR, to make it easier to review the functional changes.

It'd be nice if we just hose down the whole project with an auto-formatter like yapf or black and call it a day. I don't have any real style preference personally, but those tools do a pretty decent job and free us up to look at more interesting things.

[Mic92]

We use black now in NixOS tests as well.