iptables entries are not reconciled

[tgross]

When a client is stopped, the tasks on that client are left running. When the client restarts, it goes through a restore process to get handles to all its tasks again. If a task fails or is removed while the client is shutdown, the client should be able to garbage collect any of its dangling resources (like alloc dirs) and restart the task. This is not happening with iptables.

Fortunately we "tag" all the iptables rules in one of two ways:

• Placing them in a chain named CNI-xxxx/CNI-DN-xxxx. I don't know what that xxxx is but it's not the alloc ID, container ID, or network namespace ID.
• Adding a comment in the form /* name: "nomad" id: "<alloc ID>" *.

So it should be possible to identify "Nomad owned" rules and clean them up if they don't belong to an allocation we know about if we can figure out the naming for the CNI chains.

Nomad version

Nomad v0.10.0-dev (e2761807a346c5e3afd577b7994cfc788700bb15+CHANGES)

(But probably any recent version.)

Reproduction steps

1. Run Nomad under systemd.
2. Run our Consul Connect demo job: nomad job run ./e2e/connect/input/demo.nomad
3. Stop the job: nomad job stop countdash
4. Observe that the tasks and iptables are cleaned up properly.
   • docker ps
   • sudo iptables -t nat -L -v -n
5. Run the job again: nomad job run ./e2e/connect/input/demo.nomad
6. Stop the Nomad client with sudo systemctl stop nomad.
7. Observe that the tasks and iptables are still in place.
   • docker ps
   • sudo iptables -t nat -L -v -n
8. Remove the tasks: docker rm -f $(docker ps -a)
9. Restart Nomad: sudo systemctl start nomad
10. Observe that the tasks are started: docker ps
11. Stop the job cleanly: nomad job stop countdash
12. Observe that iptables are left behind: sudo iptables -t nat -L -v -n

Logs

iptables after repro steps

vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$ sudo iptables -t nat -L -v --line-numbers -n
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       29  1276 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
2       20   880 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 6 packets, 360 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        5   300 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
2      279 16740 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 6 packets, 360 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      348 20568 CNI-HOSTPORT-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd requiring masquerade */
2        0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
3        0     0 CNI-6fcd2f53d5f720ec4eb5f04d  all  --  *      *       172.26.64.102        0.0.0.0/0            /* name: "nomad" id: "3e803d29-4f9d-ad8b-adb6-31456a39db69" */
4        0     0 CNI-06d73cb6cdf7130196e2018a  all  --  *      *       172.26.64.101        0.0.0.0/0            /* name: "nomad" id: "ee25f5d7-dcb9-b336-fe3e-27e365aa5cd0" */

Chain CNI-06d73cb6cdf7130196e2018a (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     all  --  *      *       0.0.0.0/0            172.26.64.0/20       /* name: "nomad" id: "ee25f5d7-dcb9-b336-fe3e-27e365aa5cd0" */
2        0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4          /* name: "nomad" id: "ee25f5d7-dcb9-b336-fe3e-27e365aa5cd0" */

Chain CNI-6fcd2f53d5f720ec4eb5f04d (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     all  --  *      *       0.0.0.0/0            172.26.64.0/20       /* name: "nomad" id: "3e803d29-4f9d-ad8b-adb6-31456a39db69" */
2        0     0 MASQUERADE  all  --  *      *       0.0.0.0/0           !224.0.0.0/4          /* name: "nomad" id: "3e803d29-4f9d-ad8b-adb6-31456a39db69" */

Chain CNI-HOSTPORT-DNAT (2 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain CNI-HOSTPORT-MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       59  3540 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Chain CNI-HOSTPORT-SETMARK (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1       59  3540 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd masquerade mark */ MARK or 0x2000

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

cc @davemay99 @angrycub as a heads up

[tgross]

Summary of the investigation at this point:

• When the client restarts, the network hook's Prerun fires and tries to recreate the network and setup the iptables via CNI.
• This fails because the netns already exists, as expected. So we tear down the task and start over.
• But in the next pass when we setup the iptables via CNI, we collide with the iptables left behind.
• To fix this we need the network namespace path (which is used by CNI as part of the handle for go-cni#Network.Remove).
• In the non-Docker case, Nomad controls the path to the netns and it's derived from the alloc ID, but in the Docker case (which includes all cases with Connect integration b/c of the Envoy container), Docker owns that path and derives it from the pause container name. So we can't use a deterministic name as a handle to clean up. But we can't get it from Docker either because at the point we need it that container has already been removed.

I've verified the following more common failure modes are handled correctly:

• Tasks recover fully when the client restarts (after a few PRs we landed in the current 0.10.0 release branch)
• There's no resource leak when the client restarts if the containers aren't removed.
• There's no resource leak when the client restarts as part of a node (machine) reboot.

Status:

• We could try to fix this by threading state about the network namespace from the allocation runner back into the state store, similar to how we deal with deployment health state. But this will always be subject to races between client failures and state syncs.
• We already have a PR open for 0.10.x to reconcile and GC Docker containers. Because all the rules we're creating are tagged with the string "nomad" and Nomad's alloc IDs, we can make a similar loop for iptable GC.
• Because I've verified that this leak doesn't happen in the common failure modes of a client or node reboot, we're not going to block the 0.10.0 release on this. We'll work up a PR for an out-of-band reconcile loop for 0.10.x.

Moving this issue out of the 0.10.0 milestone.