docker disappeared

[g00nix]

Docker disappeared from microk8s:

root@kube:~# microk8s.docker
microk8s.docker: command not found
root@kube:~# ls /snap/bin/  
microk8s.config  microk8s.ctr  microk8s.disable  microk8s.enable  microk8s.inspect  microk8s.istioctl  microk8s.kubectl  microk8s.reset  microk8s.start  microk8s.status  microk8s.stop
root@kube:~# cat /etc/issue
Ubuntu 18.04.2 LTS \n \l

It it still going to be used in the project? Is there an alternative to inspect what kube is doing in the background?

If this was a planned change, is there documentation/release notes?

[mattiasarro]

They have replaced docker daemon with containerd.

For me, this is a problem, since I was relying on exposing the Docker daemon on a TCP port. I did this by adding -H tcp://0.0.0.0:$PORT to /var/snap/microk8s/current/args/dockerd, which meant I could execute docker commands from outside the machine (with docker -H tcp://$HOST:$PORT).

Now I don't really know if I can connect to the containerd daemon the same way, and if so, how.

[g00nix]

For me, this is a problem

I can't say it's a "problem" for me. It was just a surprise. We were using the that docker to also upload images into the registry, but we just went for apt install docker for that scope.

Well, if that is the case, I guess it's up to the devs if they close the ticket or not.

[ktsakalozos]

Hi,

Indeed in the 1.14 release contanerd replaced dockerd. We gave the heads up on this change in the on this topic https://discuss.kubernetes.io/t/containerd-and-security-updates-on-the-next-microk8s-release/4844 and on the #microk8s channel at https://k8s.slack.com/ some time ago. For those who cannot do the transition dockerd is available from the 1.13 channel:

snap install microk8s --classic --channel=1.13/stable

One of the reasons why we moved ahead with containerd is that a few users wanted to deploy MicroK8s next to a local dockerd and that was causing unexpected issues. As @gun1x mentions this should not be a problem anymore.

@mattiasarro, I am not aware of how you are using dockerd. Could you provide more info so we can offer some suggestions?

Apologies for any inconvenience we may have cause.

[termie]

this also took me a bit by surprise, i think i didn't read the expectations that snaps would autoupdate and would have pegged a specific version channel if i had. The microk8s docker did indeed cause some conflicts for me when i was first setting it up (and has some restrictions on it that make using it as a normal docker install kind of weird), so i'm generally for this change, but it has left a bunch of systems of mine in a (so far) unrecoverable state. Am hunting through logs to figure out why kubelet-daemon refuses to start after a reboot and will file another issue.

[carlososiel]

I cannot configure containerd correctly for use insecure registries after these update. I configure containerd as say in the documentation and kubelet continue trying to pull from https instead of http.

[mattiasarro]

Thanks for the quick response @ktsakalozos!

I am developing on macOS and running microk8s on a ubuntu-based PC, to which I connect directly over ethernet. I have enabled the private registry on microk8s (microk8s.enable registry) and would like to push Docker images to it. So I have exposed the dockerd daemon on a tcp port by editing /var/snap/microk8s/current/args/dockerd and adding a line -H 0.0.0.0:$PORT to it, which makes it available on $PORT of my ubuntu instance. Which means I can do on my mac:

docker -H $UBUNTU_MACHINE_IP:$PORT build -t mytag .
docker -H $UBUNTU_MACHINE_IP:$PORT tag mytag localhost:32000/mytag
docker -H $UBUNTU_MACHINE_IP:$PORT push localhost:32000/mytag

and now the image is available to the microk8s cluster.

Do you know if it's possible to expose containerd in a similar way and push images to it over the network?

[ktsakalozos]

@mattiasarro in short you will have to use the image build tool chain of your liking and upload the image you build into the insecure registry MicroK8s provides.

Here is a way to do this. I have on my host a VM with MicroK8s:

> multipass list
Name                    State             IPv4             Release
microk8s-vm             RUNNING           10.141.241.134   Ubuntu 18.04 LTS

In that VM I have the registry enabled:

> multipass exec microk8s-vm -- /snap/bin/microk8s.kubectl get all -n container-registry
NAME                           READY   STATUS    RESTARTS   AGE
pod/registry-7d65c894c-d88jh   1/1     Running   0          39m

NAME               TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/registry   NodePort   10.152.183.66   <none>        5000:32000/TCP   39m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/registry   1/1     1            1           39m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/registry-7d65c894c   1         1         1       39m

On the host machine I have docker running (installed via apt install docker.io), you can have any image building toolchain. The dockerd running on the host should trust the registry on the microk8s-vm. To do this:

echo "{
  \"insecure-registries\" : [\"10.141.241.134:32000\"]
}" | sudo tee /etc/docker/daemon.json

systemctl restart docker

Note that we use the IP of the VM as this is where the registry is.

To put an image in the registry we:

docker pull busybox
docker tag busybox 10.141.241.134:32000/my-busybox
docker push 10.141.241.134:32000/my-busybox

Now that the registry is populated we can create a pod in MicroK8s with:

multipass@microk8s-vm:~$ cat > bbox.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: localhost:32000/my-busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
multipass@microk8s-vm:~$ microk8s.kubectl apply -f ./bbox.yaml 
pod/busybox created

If you apt-get install docker.io inside the VM and configure the docker daemon to a) trust the registry at localhost:32000 and b) have it listening on 0.0.0.0:{A_PORT}. You can have exactly the same behavior with the one you had so far.

[g00nix]

We gave the heads up on this change in the on this topic https://discuss.kubernetes.io/t/containerd-and-security-updates-on-the-next-microk8s-release/4844 and on the #microk8s channel at https://k8s.slack.com/ some time ago.

Do you think it is possible to document changes in the release history on git? People have a tendency to pick their specific socializing platforms (example: facebook, telegram, irc), so they might miss the news if it's announced on slack, and I also doubt everybody is subscribed to discuss.kubernetes.io. Documenting stuff within git would help a lot, since everybody will go back to the git repo when there is an issue.

I think this would be great as the user base of microk8s grows. It's a very cool product and I think that helping the community with extended documentation would help microk8s become even better.

[RemcoPerlee]

I can't seem to configure this correctly. I've used the steps from @ktsakalozos which seemed to work, however the Pod stays in the status Waiting: ContainerCreating, and I have no idea what's going on behind the scenes...

Any way I can debug this (what logs to look at, etc)?

The kubectl logs state the following:

Error from server (BadRequest): container "busybox" in pod "busybox" is waiting to start: ContainerCreating

When I try this /snap/bin/microk8s.ctr image pull localhost:32000/my-busybox
I get an error, but it might be my lack of understanding CTR:
ctr: failed to resolve reference "localhost:32000/my-busybox": object required

[g00nix]

Quick offtopic: now everybody knows I am bad with grammar. 🤣

[RemcoPerlee]

well, I gave up and decided to rollback to 1.13, but now my cluster won't boot again. Can I state here, that I really dislike breaking changes like this?