AWS VPC resources + Diff engine #697

AmineChikhaoui · 2017-07-06T13:58:36Z

This PR adds integration for AWS VPC resources and an implementation of the ideas discussed in #648 .
The suggested way of implementing resources is by using a diff engine that can be instantiated using nixops.resources.ec2_common.setup_diff_engine() this diff engine will handle the comparison between the state and the target configuration and then generate a plan which is composed of the various handlers that needs to be called to realize the target state.
A resource can basically define the handlers and setup the right dependencies between them e.g self.handle_entries = Handler(['entries'], after=[self.handle_create_network_acl], handle=self.realize_entries_change)
This means that this handler will handle the entries option changes and will run after create_network_acl if they're invoked in the same deploy operation and will use the method realize_entries_change to realize the required changes.
There is an experimental option --plan-only (supported only for the VPC resources which can be added to the nixops deploy cmd to show the changes that will be made and exit.

Note that this doesn't implement a global DAG for the resources in a network but rather for the attributes handlers per resource.

For the VPC resources, what was already implemented is:

The remaining work:

VPC peering connection (will be done in a separate PR due to some complexity in the resource state transitions)
Add examples/documentation.

I would appreciate some initial feedback on the suggested implementation of #648 though to be able to make appropriate changes if needed.

…e state file

- Experimenting with a different state represention based on NixOS#648 - Experimenting with a diff structure that allows showing resource attributes changes before deploying if NIXOPS_PLAN env var is set. * Resource VPC dhcp options: initial nix options

need to associate the default one before deleting the dhcp options.

the dhcp options. Use the new diff structure maybe ?

* Introducing a seperate Handler class, that allows a handler to define a set of keys/options that it can manage and have an abstract method handle to be implemented by developers adding resources. A handler can also set the different dependencies that it needs to wait for before being called. * During plan operation, the diff engine chooses the minimal set of handlers that are needed to realize the change then performs a topological sort of the chosen handlers by representing them as a DAG based on the dependencies between them.

as it's a "dangerous" operation

Mic92 · 2017-10-20T11:04:46Z

examples/vpc.nix

+        inherit region accessKeyId;
+        vpcId = resources.vpc.vpc-nixops;
+        rules = [
+          { toPort = 22; fromPort = 22; sourceIp = "41.231.120.171/32"; }


Please add a check if fromPort is set otherwise the following error message is thrown:

AttributeError: 'NoneType' object has no attribute 'get'

from

from_port = int(rule_xml.find("attr[@name='fromPort']/int").get("value"))

UPDATE this problem is probably independent from this pull request. please ignore.

Mic92 · 2017-10-20T11:13:11Z

~~Can you add an example how to add an elastic ip to a machine in a VPC subnet?~~

I tested the following: https://gist.github.com/Mic92/f262cbd1c386a15e8ffa166fda8a861f

a nixops deploy cmd is interrupted the user can run nixops check or nixops deploy --check to have a consistent state.

Mic92 · 2017-11-08T20:53:51Z

Removing ip addresses also does not work yet.

AmineChikhaoui · 2017-11-13T10:30:39Z

@Mic92 can you share more details on the issue you had ? is this during a machine destroy or elastic ip destroy ?

AmineChikhaoui · 2017-11-13T10:54:11Z

@domenkozar @rbvermaa I'm ok to merge this if there are no objections. I tried to address some of Eelco's comments and I'll create few issues for the remaining improvements to work on them later.
But I think globally it would be nice if we can merge this so that we can use VPC resources and use the diff engine for any new resource as it make it much easier to handle the state.
Let me know if you have any comments :)

Mic92 · 2017-11-13T11:47:58Z

I have a machine, that I only start for benchmarks. As I have not found a way to remove an elastic ip, when the machine was stopped, I have removed it manually in the web interface. To assign a new ip I created a new resource in nixops each time. Now I have a number of obsolete nix-ip$i resources, I cannot get rid off:

$ nixops deploy -d cntr --kill-obsolete --show-trace --debug
resource ‘cntr-ip3’ is obsolete
resource ‘cntr-ip2’ is obsolete
resource ‘cntr-ip4’ is obsolete
resource ‘cntr-ip’ is obsolete
...
error: Multiple exceptions: list index out of range, list index out of range, list index out of range, list index out of range
------------------------------
Traceback (most recent call last):
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/deployment.py", line 1066, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 109, in destroy
    eip = self.describe_eip()
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 103, in describe_eip
    return response['Addresses'][0]
IndexError: list index out of range
------------------------------
Traceback (most recent call last):
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/deployment.py", line 1066, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 109, in destroy
    eip = self.describe_eip()
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 103, in describe_eip
    return response['Addresses'][0]
IndexError: list index out of range
------------------------------
Traceback (most recent call last):
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/deployment.py", line 1066, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 109, in destroy
    eip = self.describe_eip()
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 103, in describe_eip
    return response['Addresses'][0]
IndexError: list index out of range
------------------------------
Traceback (most recent call last):
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/deployment.py", line 1066, in worker
    if m.destroy(wipe=wipe): self.delete_resource(m)
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 109, in destroy
    eip = self.describe_eip()
  File "/home/joerg/.homesick/repos/dotfiles/nixos/vms/nixops/nixops/resources/elastic_ip.py", line 103, in describe_eip
    return response['Addresses'][0]
IndexError: list index out of range

eip api call doesn't throw a client error in case the eip doesn't exist and only return an empty list.

AmineChikhaoui · 2017-11-13T12:06:09Z

@Mic92 thanks should be fixed in f6de2a9.

Mic92 · 2017-11-13T15:40:45Z

@AmineChikhaoui thanks. that solved the issue.

methods.

resources. use a smaller instance for tests/example.

AmineChikhaoui · 2017-12-30T21:49:25Z

@domenkozar @rbvermaa Would you mind looking at this again. I've added few basic tests to tests/functional/vpc.py. Let me know if there are any specific tests you think should be added.

AmineChikhaoui · 2018-01-02T15:23:36Z

yay 🎆 , thanks @edolstra

AmineChikhaoui added 30 commits May 11, 2017 11:09

initial structure of vpc options

4e4f076

Initial defn/state of vpc

d121488

adding files

f4fb92f

Working version with boto3 instead of boto2

a604876

changes to support boto3

b2f1db3

Handle vpc deletion from outside nixops and add more attributes to th…

f2b34ac

…e state file

adding a warning message if the vpc is deleted from outside nixops

3bc84fb

add vpcId as an output option to pass to other resources

a91a4c9

Adding VPC subnet resource.

e5b0b31

* Resource VPC internet gateway:

6dfa660

- Experimenting with a different state represention based on NixOS#648 - Experimenting with a diff structure that allows showing resource attributes changes before deploying if NIXOPS_PLAN env var is set. * Resource VPC dhcp options: initial nix options

Add dhcp options state handling.

216e1be

AWS doesn't have an api method to disassociate dhcp options so we

19a52d8

need to associate the default one before deleting the dhcp options.

Properly destroy and fixes to the values field in the dhcp config

04bbd77

Ugly but working comparaison between the dhcp options in state/defn

6f9b734

Handle already removed dhcp options

58196c2

Handling deleted vpcs in the destroy

91a8e6d

Add a fixme note for a better handling of the comparison between

dadcf67

the dhcp options. Use the new diff structure maybe ?

remove extra line

47bf80b

Merge branch 'master' of https://github.com/NixOS/nixops into vpc

111dfd8

move check of key not found

b351c1d

Make the vpc definition change require an --allow-reboot to deploy

d0b48d4

as it's a "dangerous" operation

Start to migrate VPC subnet to the new diff engine

cb45df9

Make the zone required in the vpc subnet definition

7f9aff5

Merge branch 'master' of https://github.com/NixOS/nixops into vpc

3c4eceb

Add network ACL nix definition

8eb1630

Add vpc network ACLs to the top level eval-machine-info

6b77a54

Fix nix definitions

ebd09c2

Merge branch 'master' of https://github.com/NixOS/nixops into vpc

da8a73a

Initial setup of a network ACL

f87aac7

oops missed one conflict

17544ad

Mic92 reviewed Oct 20, 2017

View reviewed changes

AmineChikhaoui added 2 commits October 21, 2017 11:24

Merge branch 'master' of github.com:NixOS/nixops into vpc

cdc90d7

VPC: Move the ensure state up logic to _check() so that in case

e8e5781

a nixops deploy cmd is interrupted the user can run nixops check or nixops deploy --check to have a consistent state.

AmineChikhaoui added 2 commits November 13, 2017 11:41

Merge branch 'master' of github.com:NixOS/nixops into vpc

daf1f79

remove print statement

2c63c66

Handle already deleted EIPs properly as it seems the describe

f6de2a9

eip api call doesn't throw a client error in case the eip doesn't exist and only return an empty list.

AmineChikhaoui added 10 commits December 8, 2017 17:13

Fix ec2 tests

4255db5

fix merge conflict

1dcd8f2

initial tests for vpc resources.

c6e6104

add few more tests of subnets and ipv6 setup.

7016a82

add nat gateway tests, factor out some common setup into separate

f0df427

methods.

remove debug statement

cfba178

add basic test of a VPC machine, including route table/internet gateway

e2ad96d

resources. use a smaller instance for tests/example.

oops this was changed during a local test

7e9605e

doh! remove my ip

86f5d7b

move the configurations to the end of the file for better readability.

a8e994a

AmineChikhaoui added 2 commits January 2, 2018 15:19

handle boto3 retriable errors properly

34ecd24

Merge branch 'master' of github.com:NixOS/nixops into vpc

7cd9a1d

edolstra merged commit f790182 into NixOS:master Jan 2, 2018

lukebfox mentioned this pull request Aug 9, 2020

Fix DiffEngineResourceState #1391

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS VPC resources + Diff engine #697

AWS VPC resources + Diff engine #697

AmineChikhaoui commented Jul 6, 2017 •

edited

Mic92 Oct 20, 2017 •

edited

Mic92 commented Oct 20, 2017 •

edited

Mic92 commented Nov 8, 2017

AmineChikhaoui commented Nov 13, 2017

AmineChikhaoui commented Nov 13, 2017

Mic92 commented Nov 13, 2017 •

edited

AmineChikhaoui commented Nov 13, 2017

Mic92 commented Nov 13, 2017

AmineChikhaoui commented Dec 30, 2017

AmineChikhaoui commented Jan 2, 2018

AWS VPC resources + Diff engine #697

AWS VPC resources + Diff engine #697

Conversation

AmineChikhaoui commented Jul 6, 2017 • edited

Mic92 Oct 20, 2017 • edited

Choose a reason for hiding this comment

Mic92 commented Oct 20, 2017 • edited

Mic92 commented Nov 8, 2017

AmineChikhaoui commented Nov 13, 2017

AmineChikhaoui commented Nov 13, 2017

Mic92 commented Nov 13, 2017 • edited

AmineChikhaoui commented Nov 13, 2017

Mic92 commented Nov 13, 2017

AmineChikhaoui commented Dec 30, 2017

AmineChikhaoui commented Jan 2, 2018

AmineChikhaoui commented Jul 6, 2017 •

edited

Mic92 Oct 20, 2017 •

edited

Mic92 commented Oct 20, 2017 •

edited

Mic92 commented Nov 13, 2017 •

edited