Restarting faasd leaves containers

[martindekov]

While working on:
#40

I had to restart the faasd along with the faasd-provider service to to load the new faasd binary with the changes. With every restart the IP address of the gateway moved up with some value like:

• First run: http://10.62.1.151:8080
• Second run: http://10.62.1.161:8080
• Third run: http://10.62.1.171:8080

It seems like with every restart the containers left on the file system. When hitting the endpoint it load balanced between three figlet containers. Not sure if restarting the service should stop/delete the containers.

Expected Behaviour

Restarting the services should delete all containers.

Current Behaviour

Restarting the services leaves containers.

Possible Solution

I might be restarting something wrong I would need more time to research this but I am opening directly and Issue for the record. If this is not a problem (I am restarting the services the wrong way with systemctl restart faasd && systemctl restart faasd-provider) it might be worth documenting how to properly restart the faasd and faasd-provider services.

Steps to Reproduce (for bugs)

1. Run faasd faasd-provider
2. Deploy figlet with no labels
3. Restart faasd faasd-provider (systemctl)
4. Deploy figlet with labels
5. Start hitting /system/functions and see how figlet with labels and figlet with null labels appear like:

[{"name":"nodeinfo","image":"docker.io/functions/nodeinfo:latest","invocationCount":0,"replicas":0,"envProcess":"","availableReplicas":0,"labels":{"label":"saws"},"annotations":null,"namespace":"openfaas-fn"},{"name":"figlet","image":"docker.io/functions/figlet:0.13.0","invocationCount":0,"replicas":1,"envProcess":"","availableReplicas":0,"labels":{"label":"saws"},"annotations":null,"namespace":"openfaas-fn"}]

[{"name":"nodeinfo","image":"docker.io/functions/nodeinfo:latest","invocationCount":0,"replicas":0,"envProcess":"","availableReplicas":0,"labels":null,"annotations":null,"namespace":"openfaas-fn"},{"name":"figlet","image":"docker.io/functions/figlet:0.13.0","invocationCount":0,"replicas":1,"envProcess":"","availableReplicas":0,"labels":null,"annotations":null,"namespace":"openfaas-fn"}]

Also if you delete function from the UI and reload couple of time the deleted function might show again, due to the backend showing actually container which was spawn before restarting the faasd service.

Context

While working on #40 feature I was confused why it didn't work, but the problem is that the backend was just showing random figlet containers. Had to reload /system/functions couple of times by mistake in order to see that actually I was able to apply the label.

Your Environment

• OS and architecture:
  Ubuntu 18.04
• Versions:

go version
go version go1.12.7 linux/amd64

containerd -version
containerd github.com/containerd/containerd 1.3.0+unknown 

uname -a
Linux mdekov-Lenovo-Y520-15IKBN 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[alexellis]

@carlosedp please could you take a look?

[martindekov]

I can explore this further so I will assign myself

[martindekov]

Derek assign: me

[carlosedp]

The containers shouldn't stay but the IP allocation is kept on the filesystem.
The CNI plugin uses the "host-local" IPAM that by default stores the allocated IPs in /var/lib/cni/networks/$NETWORK_NAME.

I think it would be a good thing to move this directory to /var/run/cni for example so it would be cleaned between reboots.

What do you think @alexellis?

[carlosedp]

This does not happen here, as can be seen below, first I curl directly to the gateway IP address (.10) and list the containerd tasks. Gateway has the PID 5736.

Then I restart faasd and the gateway container has other PID (6664). Also it has a new IP address (.15).

Curl to it on a deployed function returns as well. Also there are no duplicate containers.

❯ curl -d Test 10.62.0.10:8080/function/figlet1
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

❯ sudo ctr  task ls
TASK                 PID     STATUS
prometheus           5612    RUNNING
gateway              5736    RUNNING
queue-worker         5852    RUNNING
basic-auth-plugin    5393    RUNNING
nats                 5493    RUNNING
❯ sc-restart faasd.service
❯ sudo ctr  task ls
TASK                 PID     STATUS
basic-auth-plugin    6311    RUNNING
nats                 6411    RUNNING
prometheus           6535    RUNNING
gateway              6664    RUNNING
queue-worker         6778    RUNNING
❯ sudo cat  /var/lib/cni/results/openfaas-cni-bridge-gateway-6664-eth1
{"cniVersion":"0.4.0","interfaces":[{"name":"openfaas0","mac":"72:db:c2:d4:d3:69"},{"name":"veth6470e54c","mac":"2e:24:51:57:34:c1"},{"name":"eth1","mac":"8e:92:84:46:5c:8a","sandbox":"/proc/6664/ns/net"}],"ips":[{"version":"4","interface":2,"address":"10.62.0.15/16","gateway":"10.62.0.1"}],"routes":[{"dst":"0.0.0.0/0"}],"dns":{}}%                                                           ❯ curl -d Test 10.62.0.15:8080/function/figlet1
 _____         _
|_   _|__  ___| |_
  | |/ _ \/ __| __|
  | |  __/\__ \ |_
  |_|\___||___/\__|

❯ sudo ctr container ls
CONTAINER            IMAGE                                           RUNTIME
basic-auth-plugin    docker.io/openfaas/basic-auth-plugin:0.18.10    io.containerd.runc.v2
gateway              docker.io/openfaas/gateway:0.18.8               io.containerd.runc.v2
nats                 docker.io/library/nats-streaming:0.11.2         io.containerd.runc.v2
prometheus           docker.io/prom/prometheus:v2.14.0               io.containerd.runc.v2
queue-worker         docker.io/openfaas/queue-worker:0.9.0           io.containerd.runc.v2

[carlosedp]

Also I can't deploy the same figlet function twice:

❯ faas ls
Function                      	Invocations    	Replicas
figlet1                       	0              	1
❯ faas store deploy figlet --name=figlet1 --env write_timeout=1s
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
Function figlet1 already exists, attempting rolling-update.

Deployed. 200 OK.
URL: http://127.0.0.1:8080/function/figlet1

❯ faas ls
Function                      	Invocations    	Replicas
figlet1                       	0              	1

I believe a function with or without labels should be unique by it's name.

[martindekov]

Thanks for looking at this @carlosedp I have also tried to replicate and it seems like the problem did not persist. I only deployed the same function with two different set of labels this time though and restarted the faasd and faasd-provider services which is a subset of the things I did originally.

I will close this I think what I have noticed was related with me recompiling the binary or some very corner case.