missing test pass with containerd in kube-up, due to Incorrect set of runtime-endpoint parameters in kube-up for test clusters

[yb01]

What happened:

I checked on one of env YingH used in her tests, from the runtime’s perspective, this is not right. There could be other potential parameter issues so we might need to take a further look.

From runtime’s perspective, this is due to we take the same parameter from the Admin cluster which suppose only setup the cluster to orchestrate the performance clusters. Hence the docker shim is used.

For our kube-up, its goal is to set up a real Arktos cluster for uses, so the Kubelet commandline parameters should match with or Arktos-up ( or at least the Kubemark cluster’s settings ).

@Sonya Li, most likely an issue in our kube-up tool and we should consider fix.

ybai2016@ying-scaleout-rp-1-minion-group-fxmh:~$
ybai2016@ying-scaleout-rp-1-minion-group-fxmh:~$ ps -ef | grep kubelet
root        6733       1  2 17:10 ?        00:00:09 /home/kubernetes/bin/kubelet --v=4 --cloud-provider=gce --max-pods=110 --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --experimental-check-node-capabilities-before-mount=true --cert-dir=/var/lib/kubelet/pki/ --dynamic-config-dir=/var/lib/kubelet/dynamic-config --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --tenant-server-kubeconfig=/etc/srv/kubernetes/tp-kubeconfigs/tp-1-kubeconfig,/etc/srv/kubernetes/tp-kubeconfigs/tp-2-kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --cni-bin-dir=/opt/cni/bin --network-plugin=cni --non-masquerade-cidr=0.0.0.0/0 --volume-plugin-dir=/home/kubernetes/flexvolume --container-runtime-endpoint=unix:///var/run/dockershim.sock --config /home/kubernetes/kubelet-config.yaml --resolv-conf=/run/systemd/resolve/resolv.conf
root        6866       1  0 17:10 ?        00:00:00 bash /home/kubernetes/bin/health-monitor.sh kubelet
ybai2016   10869   10755  0 17:17 pts/0    00:00:00 grep --color=auto kubelet
ybai2016@ying-scaleout-rp-1-minion-group-fxmh:~$

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[sonyafenge]

if you are talking --container-runtime-endpoint=unix:///var/run/dockershim.sock, yes, for mizar, we are using docker as runtime.

For all default kube-up, we are using containerd as default runtime and kubelet config is like below:

# ps -ef | grep kubelet
root        2583       1  3 02:15 ?        00:00:30 /home/kubernetes/bin/kubelet --v=2 --runtime-cgroups=/system.slice/containerd.service --cloud-provider=gce --max-pods=110 --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --experimental-check-node-capabilities-before-mount=true --cert-dir=/var/lib/kubelet/pki/ --dynamic-config-dir=/var/lib/kubelet/dynamic-config --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --register-schedulable=false --cni-bin-dir=/home/kubernetes/bin --network-plugin=cni --non-masquerade-cidr=0.0.0.0/0 --volume-plugin-dir=/home/kubernetes/flexvolume --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --config /home/kubernetes/kubelet-config.yaml --resolv-conf=/run/systemd/resolve/resolv.conf

[yb01]

the container-runtime-endpoint should be

--container-runtime=remote --container-runtime-endpoint=containerRuntime,container,/run/containerd/containerd.sock;vmRuntime,vm,/run/virtlet.sock 

the kube-up script in 130, we should be able to start scaleout cluster for user to try Arktos scaleout architecture, the VM should be supported too, also containerD is what Arktos claimed container runtime which we should use.

the arktos-up.sh start Mizar successfully so funcationality wise, Mizar should be fine with contaienrD. otherwise, arktos-up.sh will fail or not be able to orchestrate test containers with Mizar.

[yb01]

Change title to reflect the concern of this setting. looks like we have two different settings in testing:

1. locally with Arktos-up.sh with mizar, it uses containers as runtime
2. with the kube-up tool to setup scaleout clusters, it is hardcoded dockershim as container runtime.

which means all the test scenarios with containerd as runtime which Arktos has claimed as default since Arktos project started.

since we cannot control user which one to use -- nowadays, most will be OOF dockershim -- missing full test pass with containerd is not ideal.