API Server (inside kubelite) fails to start for unknown reason

[Richard87]

I have tried current stable and previous stable version (and reverted back to current stable now)

I'm getting these related logs:

2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: Error: context deadline exceeded
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: F0731 12:52:31.422617 3250825 daemon.go:67] API Server exited context deadline exceeded
...
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1509
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/cmd/kubelite/app/daemons.StartAPIServer(0xc00053fa00, 0x1a, 0x20, 0xc00002b800)

microk8s inspect:

root@w ➜  richard microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-kubelite is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

WARNING:  The memory cgroup is not enabled. 
The cluster may not be functioning properly. Please ensure cgroups are enabled 
See for example: https://microk8s.io/docs/install-alternatives#heading--arm 
 FAIL:  Cgroup v1 seems not to be enabled. Please enable it 
	by executing the following command and reboot: 
	grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" 
Building the report tarball
  Report tarball is at [ir.tar.gz](https://github.com/ubuntu/microk8s/files/6911088/ir.tar.gz)inspection-report-20210731_131740.tar.gz

Kubelite logs:

snap logs microk8s.daemon-kubelite -f 
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: --apiserver-args-file=$SNAP_DATA/args/kube-apiserver
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: --kubeconfig-file=$SNAP_DATA/credentials/client.config
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: --start-control-plane=true)'
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: + exec /snap/microk8s/2346/kubelite --scheduler-args-file=[ir.tar.gz](https://github.com/ubuntu/microk8s/files/6911088/ir.tar.gz)args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/2346/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/2346/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/2346/args/kubelet --apiserver-args-file=/var/snap/microk8s/2346/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/2346/credentials/client.config --start-control-plane=true
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: Starting kubelite
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: I0731 12:47:30.831790 3250825 daemon.go:73] Waiting for the API server
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: I0731 12:47:30.832263 3250825 daemon.go:65] Starting API Server
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: I0731 12:47:30.832647 3250825 server.go:654] external host was not specified, using 192.168.2.42
2021-07-31T10:47:30Z microk8s.daemon-kubelite[3250825]: W0731 12:47:30.832657 3250825 authentication.go:507] AnonymousAuth is not allowed with the AlwaysAllow authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: Error: context deadline exceeded
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: F0731 12:52:31.422617 3250825 daemon.go:67] API Server exited context deadline exceeded
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 150 [running]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00013c001, 0xc0005681c0, 0x58, 0x1b9)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x98122c0, 0xc000000003, 0x0, 0x0, 0xc00040e380, 0x7cd17cb, 0x9, 0x43, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x191
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printf(0x98122c0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x5f2a447, 0x14, 0xc00069bcb0, 0x1, ...)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:751 +0x191
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1509
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/cmd/kubelite/app/daemons.StartAPIServer(0xc00053fa00, 0x1a, 0x20, 0xc00002b800)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelite/app/daemons/daemon.go:67 +0x225
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/cmd/kubelite/app.glob..func1
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelite/app/server.go:42 +0x177
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 1 [select]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc00078fb00, 0xc000fd1c08, 0xc00002bc20, 0x0, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:539 +0xf1
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollInternal(0xc00078fb00, 0xc000f6fc08, 0x0, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:427 +0x87
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal(0xc00078fb00, 0xc000f6fc08, 0xc00078fb00, 0x44f)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:452 +0x74
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate(0x3b9aca00, 0x53d1ac1000, 0xc000f6fc08, 0x0, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:441 +0x4d
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/controller-manager/app.WaitForAPIServer(0x6a4b2e8, 0xc000d358c0, 0x53d1ac1000, 0x0, 0xc0005f9d40)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/controller-manager/app/helper.go:35 +0x98
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/cmd/kubelite/app/daemons.WaitForAPIServer(0x7ffd70dd0b0b, 0x31, 0x53d1ac1000)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelite/app/daemons/daemon.go:83 +0x2cc
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/cmd/kubelite/app.glob..func1(0x96ec2c0, 0xc00040e310, 0x0, 0x7)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelite/app/server.go:43 +0x1a5
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0x96ec2c0, 0xc000140010, 0x7, 0x7, 0x96ec2c0, 0xc000140010)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x96ec2c0, 0x10, 0xc000000180, 0x7cb6ca)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x375
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/cmd/kubelite/app.Execute()
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelite/app/server.go:63 +0x31
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: main.main()
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	_output/local/go/src/k8s.io/kubernetes/cmd/kubelite/kubelite.go:26 +0x111
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 19 [chan receive]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x98122c0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:418 +0xdf
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 74 [select]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000376fa0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:154 +0xcd
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.init.0
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 140 [sleep]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: time.Sleep(0x3b9aca00)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/runtime/time.go:193 +0xd2
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/internal/protocol.makeRetryStrategies.func1(0x12e, 0x12d)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/internal/protocol/connector.go:314 +0x65
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/Rican7/retry.shouldAttempt(0x12e, 0xc000ca87d0, 0x1, 0x1, 0x9845c01)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/Rican7/retry/retry.go:32 +0x47
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/Rican7/retry.Retry(0xc000907d48, 0xc000ca87d0, 0x1, 0x1, 0x1, 0x1)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/Rican7/retry/retry.go:19 +0x9c
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/internal/protocol.(*Connector).Connect(0xc000907df8, 0x69bd910, 0xc000447640, 0x7f2dbc0ff6d0, 0x10, 0x10)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/internal/protocol/connector.go:73 +0xc7
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/client.FindLeader(0x69bd910, 0xc000447640, 0x6944e28, 0xc0004475c0, 0xc000408420, 0x2, 0x2, 0x19ddde9d90e9a, 0xc0003760f0, 0xc00068eb40)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/client/leader.go:26 +0x1ac
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app.(*App).Leader(0xc0007188f0, 0x69bd910, 0xc000447640, 0x0, 0x2, 0xc0008e5f01)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app/app.go:403 +0x7c
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app.(*App).run(0xc0007188f0, 0x69bd910, 0xc000447640, 0x6fc23ac00, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app/app.go:458 +0x15f
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app.New
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app/app.go:238 +0x1013
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 79 [IO wait, 5 minutes]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: internal/poll.runtime_pollWait(0x7f2dbe220a70, 0x72, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/runtime/netpoll.go:222 +0x55
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: internal/poll.(*pollDesc).wait(0xc000c29d18, 0x72, 0x0, 0x0, 0x5ef65d4)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/internal/poll/fd_poll_runtime.go:87 +0x45
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: internal/poll.(*pollDesc).waitRead(...)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/internal/poll/fd_poll_runtime.go:92
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: internal/poll.(*FD).Accept(0xc000c29d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/internal/poll/fd_unix.go:401 +0x212
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: net.(*netFD).accept(0xc000c29d00, 0xc00049bee0, 0x414f38, 0x10)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/net/fd_unix.go:172 +0x45
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: net.(*TCPListener).accept(0xc00040bb30, 0x5640ce0, 0xc00082f520, 0x6941a70)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/net/tcpsock_posix.go:139 +0x32
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: net.(*TCPListener).Accept(0xc00040bb30, 0xc00013a010, 0x69bd910, 0xc000447680, 0xc00082f520)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/net/tcpsock.go:261 +0x65
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app.(*App).proxy(0xc0007188f0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app/app.go:416 +0xa3
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app.New
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/canonical/go-dqlite/app/app.go:232 +0x1227
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 146 [select]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x6158810, 0x6939b60, 0xc00043ab40, 0x1, 0xc00010e0c0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x118
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x6158810, 0x12a05f200, 0x0, 0x1, 0xc00010e0c0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0x6158810, 0x12a05f200)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 148 [syscall, 5 minutes]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: os/signal.signal_recv(0x0)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/runtime/sigqueue.go:168 +0xa5
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: os/signal.loop()
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/os/signal/signal_unix.go:23 +0x25
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by os/signal.Notify.func1.1
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/snap/go/7954/src/os/signal/signal.go:151 +0x45
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 149 [chan receive, 5 minutes]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.SetupSignalContext.func1(0xc00071ca30)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:48 +0x36
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.SetupSignalContext
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:47 +0xf3
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: goroutine 154 [select]:
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc00002bce0, 0x3b9aca00, 0x53d1ac1000, 0xc00002bc80)
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:588 +0x135
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1
2021-07-31T10:52:31Z microk8s.daemon-kubelite[3250825]: 	/build/microk8s/parts/k8s-binaries/build/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:571 +0x8c
2021-07-31T10:52:31Z systemd[1]: snap.microk8s.daemon-kubelite.service: Main process exited, code=exited, status=255/EXCEPTION
2021-07-31T10:52:31Z systemd[1]: snap.microk8s.daemon-kubelite.service: Failed with result 'exit-code'.
2021-07-31T10:52:31Z systemd[1]: snap.microk8s.daemon-kubelite.service: Consumed 1.268s CPU time.
2021-07-31T10:52:31Z systemd[1]: snap.microk8s.daemon-kubelite.service: Scheduled restart job, restart counter is at 2.
2021-07-31T10:52:31Z systemd[1]: Stopped Service for snap application microk8s.daemon-kubelite.
2021-07-31T10:52:31Z systemd[1]: snap.microk8s.daemon-kubelite.service: Consumed 1.268s CPU time.
2021-07-31T10:52:31Z systemd[1]: Started Service for snap application microk8s.daemon-kubelite.
...

[balchua]

It looks like dqlite couldn't find a leader.
Is this a multi node cluster?

Also can you show the content of the file /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml as well as the content of this folder /var/snap/microk8s/current/var/kubernetes/backend/
Thanks

[Richard87]

Hi!

No, this is not a multinode cluster.

cluster.yaml does exist, but is empty

root@w ➜  richard ll /var/snap/microk8s/current/var/kubernetes/backend/
total 170M
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:43 0000000036028504-0000000036028944
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:46 0000000036028945-0000000036029333
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:46 0000000036029334-0000000036029952
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:48 0000000036029953-0000000036030447
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:51 0000000036030448-0000000036030788
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:51 0000000036030789-0000000036031384
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:52 0000000036031385-0000000036031947
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:55 0000000036031948-0000000036032287
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:56 0000000036032288-0000000036032814
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:56 0000000036032815-0000000036033423
-rw-rw----. 1 root microk8s 8.0M Jul 25 23:59 0000000036033424-0000000036033778
-rw-rw----. 1 root microk8s 8.0M Jul 26 00:01 0000000036033779-0000000036034246
-rw-rw----. 1 root microk8s 8.0M Jul 26 00:01 0000000036034247-0000000036034851
-rw-rw----. 1 root microk8s 8.0M Jul 26 00:03 0000000036034852-0000000036035274
-rw-rw----. 1 root microk8s 8.0M Jul 26 00:06 0000000036035275-0000000036035666
-rw-rw----. 1 root microk8s 8.0M Jul 26 00:06 0000000036035667-0000000036036275
-rw-rw----. 1 root microk8s 8.0M Jul 26 00:08 0000000036036276-0000000036036763
-rw-rw----. 1 root microk8s 2.5M Jul 26 00:09 0000000036036764-0000000036036869
-rw-rw----. 1 root microk8s 2.2K May  7 22:48 cluster.crt
-rw-rw----. 1 root microk8s 3.2K May  7 22:48 cluster.key
-rw-rw----. 1 root microk8s    0 Jul 26 00:09 cluster.yaml
-rw-rw-r--. 1 root microk8s    2 Jul 31 16:33 failure-domain
-rw-rw----. 1 root microk8s   57 May  7 22:49 info.yaml
srw-rw----. 1 root microk8s    0 Jul 19 06:56 kine.sock
-rw-rw----. 1 root microk8s   63 Jul 19 06:56 localnode.yaml
-rw-rw----. 1 root microk8s   32 May  7 22:49 metadata1
-rw-rw----. 1 root microk8s  16M Jul 26 00:06 snapshot-1-36035736-580229430
-rw-rw----. 1 root microk8s   72 Jul 26 00:06 snapshot-1-36035736-580229430.meta
-rw-rw----. 1 root microk8s  16M Jul 26 00:07 snapshot-1-36036760-580326322
-rw-rw----. 1 root microk8s   72 Jul 26 00:07 snapshot-1-36036760-580326322.meta

Thanks for helping out! :)

[balchua]

That's the reason why it couldn't find the leader.
There's already a fix for that, but probably not yet in stable.
Can you show the content of /var/snap/microk8s/current/var/kubernetes/backend/info.yaml

The cluster.yaml should look just like the info.yaml.

Sample content of cluster.yaml looks like this.

- Address: 10.10.50.11:19001
  ID: 14137143738096133143
  Role: 0

It represents an array of nodes in the cluster. Since you are using a single node, then it should contain only one element.

The info.yaml should look just like the content above too.

[Richard87]

Awesome! Thanks that solved all my problems! 💯