Getting a CNI removal error when doing ignite vm stop

[chanwit]

No description provided.

[stealthybox]

bug

sudo ./ignite run weaveworks/ignite-ubuntu --runtime docker
INFO[0000] Created VM with ID "17d4cc046e3a9bda" and name "divine-field" 
INFO[0001] Networking is handled by "cni"               
INFO[0001] Started Firecracker VM "17d4cc046e3a9bda" in a container with ID "24219c67c4031d6bec1250062be64c72fb3652affac2dad2ed6ae990b8d99fdb" 

sudo ./ignite stop divine-field --runtime docker
INFO[0000] Removing the container with ID "ignite-17d4cc046e3a9bda" from the "cni" network 
FATA[0000] invalid CIDR address: !192.168.122.0/24%!(EXTRA string=could not parse destination) 

versions:

sudo ./ignite version
Ignite version: version.Info{Major:"0", Minor:"6+", GitVersion:"v0.6.0-2+069399772e699a-dirty", GitCommit:"069399772e699ac01886c62be1355ea35e544819", GitTreeState:"dirty", BuildDate:"2019-09-03T21:05:07Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Firecracker version: v0.17.0

CNI_VERSION=v0.8.2

docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfca03
 Built:             Thu Aug 29 05:29:17 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfca03
  Built:            Thu Aug 29 05:27:52 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

git diff is small:

diff --git a/Makefile b/Makefile
index acef5f6..24d12f1 100644
--- a/Makefile
+++ b/Makefile
@@ -74,6 +74,7 @@ endif
 ifeq ($(GOARCH),$(GOHOSTARCH))
        # Only tag the development image if its architecture matches the host
        docker tag $(IMAGE):${IMAGE_DEV_TAG}-$(GOARCH) $(IMAGE):${IMAGE_DEV_TAG}
+       docker image save $(IMAGE):${IMAGE_DEV_TAG} | sudo ctr -n firecracker image import -
 endif
 ifeq ($(IS_DIRTY),0)
        docker tag $(IMAGE):${IMAGE_DEV_TAG}-$(GOARCH) $(IMAGE):${IMAGE_TAG}-$(GOARCH)

[stealthybox]

Not working -- ignite stop gives error:

• --runtime containerd --network-plugin cni (networking does not work)
• --runtime docker --network-plugin cni (networking works when running)

Working:

• --runtime docker --network-plugin docker-bridge

[chanwit]

It seems to be working fine with #421 applied, both for containerd + cni and docker + cni.

# VM 17b started with containerd + CNI
$ sudo bin/ignite stop 17b
INFO[0000] Removing the container with ID "ignite-17b100016889f1d3" from the "cni" network 
INFO[0001] Stopped VM with name "white-wave" and ID "17b100016889f1d3" 
$ sudo bin/ignite start 17b
INFO[0000] Networking is handled by "cni"               
INFO[0000] Started Firecracker VM "17b100016889f1d3" in a container with ID "ignite-17b100016889f1d3" 

# Docker Runtime + CNI
$ sudo bin/ignite run weaveworks/ignite-ubuntu --ssh --runtime=docker
INFO[0006] Created VM with ID "ab5c7862cf62ed5c" and name "twilight-breeze" 
INFO[0008] Networking is handled by "cni"               
INFO[0008] Started Firecracker VM "ab5c7862cf62ed5c" in a container with ID "d695a1ecada320fa1ba8d3f6d40ba25912136a2e8879373e821a31979897ca7c" 
$ sudo bin/ignite --runtime=docker stop twilight-breeze
INFO[0000] Removing the container with ID "ignite-ab5c7862cf62ed5c" from the "cni" network 
INFO[0001] Stopped VM with name "twilight-breeze" and ID "ab5c7862cf62ed5c" 
$ sudo bin/ignite --runtime=docker start twilight-breeze
INFO[0001] Networking is handled by "cni"               
INFO[0001] Started Firecracker VM "ab5c7862cf62ed5c" in a container with ID "d84c8045f2723e2e398dcfca2a122066093846c7c30619e8e1ebd067e29341ca" 
$ sudo bin/ignite --runtime=docker ssh twilight-breeze
root@ab5c7862cf62ed5c:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=37.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=48 time=37.3 ms

Note that my CNI implementation is the newest one from Project Atomic PPA.

[stealthybox]

This is a bug in coreos/go-iptables. See the linked issue ^

[stealthybox]

If you are affected by this bug, you can clean up all ignite nat chains like so:

sudo iptables -S -t nat | grep ignite | sed 's/^-A/sudo iptables -t nat -D/' | sh