18.04.2 microk8s issue starting

[crsavage1]

Brand new install of 18.04.2 along with microk8s which installs fine. Trying to run rancher/vm but either everything sits in Evicted or Pending. Network host is attached to is 10.1.1.0/24 which I changed in the config to be 10.10.1.1/24 as I thought maybe that was the issue. I have tried various resolutions including re-install, changing br0 etc etc. I am sure I am doing something stupid. Thanks for any insight.

inspection-report-20190509_162949.tar.gz

Please run microk8s.inspect and attach the generated tarball to this issue.

We appreciate your feedback. Thank you for using microk8s.

[ktsakalozos]

Hi @crsavage1

I see your node is under disk pressure (running low on disk), see microk8s.kubectl describe no.

The limits we set are --eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi" defined in /var/snap/microk8s/current/args/kubelet. You can update the limits and do a microk8s.stop, microk8s.start.

[crsavage1]

Please forgive my ignorance, but is there an easy way to see this, the volumes have plenty of space so I am unsure where we set the size within microk8s..better said, how to actually adjust the space microk8s has access to.

[ktsakalozos]

@crsavage1 this is a great question that should reach our official docs eventually.

In MicroK8s container data are kept under /var/snap/microk8s/common/var/lib/containerd and /var/snap/microk8s/common/run/. If you do a df -h you should see that the filesystem holding those paths (probably the root filesystem) is running low on disk space. As I understand you have some other volumes with free disk space you want to use. To do so you have to configure MicroK8s accordingly. Edit /var/snap/microk8s/current/args/containerd and point the --root and --state to the volume you want to use. Here is an example where we use the /mnt

--config ${SNAP_DATA}/args/containerd.toml
--root /mnt/var/lib/containerd
--state /mnt/run/containerd
--address ${SNAP_COMMON}/run/containerd.sock

You must restart MicroK8s for this configuration to take effect: microk8s.stop, microk8s.start.

[crsavage1]

Thanks, I added available space to my root filesystem. It shows there is no pressure now, but they remain in Pending with the exception of the ingress controller which shows as running.
inspection-report-20190510_140210.tar.gz

Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

---

MemoryPressure False Fri, 10 May 2019 14:07:06 +0000 Thu, 09 May 2019 15:10:03 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 10 May 2019 14:07:06 +0000 Fri, 10 May 2019 13:52:31 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 10 May 2019 14:07:06 +0000 Thu, 09 May 2019 15:10:03 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 10 May 2019 14:07:06 +0000 Thu, 09 May 2019 15:56:45 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: *******
Hostname: celestial
Capacity:
cpu: 24
ephemeral-storage: 507932Mi
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65930172Ki
pods: 110
Allocatable:
cpu: 24
ephemeral-storage: 506908Mi
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65827772Ki
pods: 110
System Info:
Machine ID: 51c7ec565997439da7dccb79824b3be3
System UUID: 4AFEEF13-8372-8F4E-B4AD-4D17C62F465D
Boot ID: 0be61ebf-2506-4645-a30b-91f54a444e8e
Kernel Version: 4.15.0-48-generic
OS Image: Ubuntu 18.04.2 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.2.5
Kubelet Version: v1.14.1
Kube-Proxy Version: v1.14.1
Non-terminated Pods: (1 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE

---

default nginx-ingress-microk8s-controller-d9mrr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 9m11s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

---

cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message

---

Normal Starting 9m11s kubelet, celestial Starting kubelet.
Warning InvalidDiskCapacity 9m11s kubelet, celestial invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 9m11s kubelet, celestial Node celestial status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 9m11s kubelet, celestial Node celestial status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 9m11s kubelet, celestial Node celestial status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 9m11s kubelet, celestial Updated Node Allocatable limit across pods
Normal NodeNotSchedulable 9

[crsavage1]

Just to add some color, I am looking at my root filesystem below:

/dev/mapper/ubuntu--vg-ubuntu--lv 497G 4.3G 467G 1% /

But noticed these...

/dev/loop0 90M 90M 0 100% /snap/core/6818
/dev/loop1 205M 205M 0 100% /snap/microk8s/522
/dev/loop2 91M 91M 0 100% /snap/core/6350

Are these what you are referring to? When editing the file you referenced, where is it getting the variable for SNAP_COMMON? I assume this is the issue on my install...

--config ${SNAP_DATA}/args/containerd.toml
--root ${SNAP_COMMON}/var/lib/containerd
--state ${SNAP_COMMON}/run/containerd
--address ${SNAP_COMMON}/run/containerd.sock

Sorry again for the dumb questions...

[ktsakalozos]

Here is the set of environment variables a snap has available at runtime https://docs.snapcraft.io/environment-variables

I do not see any problem with those.

[crsavage1]

So those are normal to reflect 100%? Any idea why the rest are in Pending?

NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/default-http-backend-5769f6bc66-fhghl 0/1 Pending 0 66m
default pod/nginx-ingress-microk8s-controller-d9mrr 1/1 Running 0 66m
kube-system pod/heapster-v1.5.2-5c5498f57c-2ghcm 0/4 Pending 0 66m
kube-system pod/kube-dns-6bfbdd666c-br2hl 0/3 Pending 0 66m
kube-system pod/kubernetes-dashboard-6fd7f9c494-4xjd8 0/1 Pending 0 66m
kube-system pod/monitoring-influxdb-grafana-v4-78777c64c8-qgjdh 0/2 Pending 0 66m

[ktsakalozos]

Under /snap is where the read only part of the snap is. The container images are under /var/snap/ where you have enough disk.

I see your node is marked yesterday

"unschedulable": true,
                "taints": [
                    {
                        "key": "node.kubernetes.io/unschedulable",
                        "effect": "NoSchedule",
                        "timeAdded": "2019-05-09T15:43:37Z"
                    }

Can you try microk8s.kubectl uncordon "name_of_your_node" as described in https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#maintenance-on-a-node

[crsavage1]

Awesome. Worked flawlessly. For my own education, is it just because the node was instantiated yesterday before I made the changes?

[ktsakalozos]

I am not sure exactly what happened, perhaps you can see the microk8s.kubectl describe no what went wrong.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dsidlo]

@crsavage1 this is a great question that should reach our official docs eventually.

In MicroK8s container data are kept under /var/snap/microk8s/common/var/lib/containerd and /var/snap/microk8s/common/run/. If you do a df -h you should see that the filesystem holding those paths (probably the root filesystem) is running low on disk space. As I understand you have some other volumes with free disk space you want to use. To do so you have to configure MicroK8s accordingly. Edit /var/snap/microk8s/current/args/containerd and point the --root and --state to the volume you want to use. Here is an example where we use the /mnt

--config ${SNAP_DATA}/args/containerd.toml
--root /mnt/var/lib/containerd
--state /mnt/run/containerd
--address ${SNAP_COMMON}/run/containerd.sock

You must restart MicroK8s for this configuration to take effect: microk8s.stop, microk8s.start.

@ktsakalozos, I had problems with microk8s on my system. It is not clearly documented anywhere that btrfs might cause issues. It was only when I suspected btrfs and googled with the appropriate question that your solution came up.
In my case, I had to convert one of my btrfs volume to ext4, the pointed "--state" to an appropriate dir on that device.
Then things started working as expected.
Thanks!