New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starting Kubernetes on an Arch Linux VM using Kubeadm #465

Closed
imriss opened this Issue Sep 25, 2017 · 24 comments

Comments

Projects
None yet
8 participants
@imriss

imriss commented Sep 25, 2017

Is this a BUG REPORT or FEATURE REQUEST?

It could be both BUG REPORT and FEATURE REQUEST.

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5+17d7182a7ccbb", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"not a git tree", BuildDate:"2017-09-12T15:19:56Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5+17d7182a7ccbb", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"not a git tree", BuildDate:"2017-09-12T15:19:56Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
ID_LIKE=archlinux
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
  • Kernel (e.g. uname -a):
    Linux Arch1 4.12.13-1-ARCH #1 SMP PREEMPT Fri Sep 15 06:36:43 UTC 2017 x86_64 GNU/Linux

What happened?

  1. Hanging for a long time (indefinite) at: [apiclient] Created API client, waiting for the control plane to become ready
  2. Hanging for a long time (indefinite) at: [apiclient] All control plane components are healthy after 16.56457 seconds

How to reproduce it (as minimally and precisely as possible)?

The [excerpts from the] working configuration:
Note: The VM has two interfaces. 10.0.3.15 is the IP on the NAT interface, while 10.13.13.103 is the IP of the Internal interface.

~/.bashrc:

export GOPATH=/home/user/gocode/
export KUBECONFIG=$HOME/admin.conf

/etc/hosts:

127.0.0.1	localhost.localdomain	localhost 
::1		localhost.localdomain	localhost 
10.0.3.15 	localhost.localdomain	arch1

/etc/kubernetes/config:

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://10.0.3.15:8080"

/etc/kubernetes/kubelet:

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override="

# location of the api-server
KUBELET_API_SERVER="" # --api-servers=http://10.0.3.15:8080"

# Add your own!
KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --require-kubeconfig --kubeconfig=/home/user/admin.conf"

/usr/lib/systemd/system/docker.service:
ExecStart=/usr/bin/dockerd -H fd:// --exec-opt native.cgroupdriver=systemd

Steps:

Step 0:
systemctl stop kubelet.service
Remove k8s containers, if any.
mount | grep "/var/lib/kubelet/*" | awk '{print $3}' | sudo xargs umount 1>/dev/null 2>/dev/null

Step 1: Change the files and reboot, if needed.

Step 2:

systemctl daemon-reload
systemctl restart docker.service
systemctl status docker.service
systemctl restart kubelet.service
systemctl status kubelet.service

Step 3:
In a terminal, switch to root, then:

kubeadm reset
kubeadm init

Then, wait until you see the message "All control plane components are healthy after 16.56457 seconds".

Step 4: In another terminal:

sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf

Then:
systemctl restart kubelet.service

Step 5:
You will see the progress in the first terminal (where you issued kubeadm init) to the end where you receive the token:
kubeadm join --token ecb54a.3456765423dd34c2 10.0.3.15:6443

What is expected?

It is expected that kubeadm init performs a systemctl restart kubelet.service at some point because it knows that the credentials have been changed.

Anything else we need to know?

  1. Hanging for a long time (indefinite) at: [apiclient] Created API client, waiting for the control plane to become ready was mentioned:
    #360
    kubernetes-sigs/kubeadm-dind-cluster#34
    #360 (comment)
    #228
    #353
    #308
    kubernetes/kubernetes#33544

  2. Hanging for a long time (indefinite) at: [apiclient] All control plane components are healthy after 16.56457 seconds was mentioned:
    kubernetes/kubernetes#39602
    kubernetes/kubernetes#23726
    kubernetes/kubernetes#44665

  3. Avoiding --api-servers was mentioned:
    kubernetes/kubernetes#23322
    kubernetes/kubernetes#36745

@jamiehannaford

This comment has been minimized.

Member

jamiehannaford commented Oct 6, 2017

@imriss I'm not exactly sure what the issue is here. Is it that you expect kubeadm to restart kubelet?

@imriss

This comment has been minimized.

imriss commented Oct 6, 2017

@jamiehannaford Yes, or alternately send a notification and ask the user to manually restart kubelet.

@jamiehannaford

This comment has been minimized.

Member

jamiehannaford commented Oct 6, 2017

@imriss Why do you need to restart kubelet on a fresh VM that kubeadm has just been installed to?

Or is this for VMs that had previous clusters on? Doesn't kubeadm reset followed by systemctl restart kubelet work?

@kad

This comment has been minimized.

Member

kad commented Oct 6, 2017

How are other components are installed ? kubeadm expects to have specific systemd service file to be used for kubelet, and if you're trying to mix components packaged in ArchLinux that have different settings, it might not work as expected.

@imriss

This comment has been minimized.

imriss commented Oct 6, 2017

@jamiehannaford @kad Thanks for the insights.
The main step in my case is that kubelet needs to use the new credentials that kubeadm generates during the init process. I need to restart kubelet.service for this. Otherwise, kuebadm init has no progress after [apiclient] All control plane components are healthy after ... message.
Thanks.

@imriss

This comment has been minimized.

imriss commented Oct 6, 2017

@kad For installation, I am using aur kubernetes package without any modifications.

@jamiehannaford

This comment has been minimized.

Member

jamiehannaford commented Oct 6, 2017

@imriss Yeah, so as @kad already mentioned, I think there's some inconsistency with the installation expectations across distros. When kubeadm is first installed with apt, for example, it creates a systemd drop-in that set's kubelet's --kubeconfig. When the file is eventually written, kubelet should dynamically pick this up without the need for a restart. However, for other Linux types that don't have this, I imagine you'll need to manually reconfigure things. We should probably document that.

/cc @luxas

@imriss

This comment has been minimized.

imriss commented Oct 7, 2017

@jamiehannaford Thanks. I set the --kubeconfig to --kubeconfig=/home/user/admin.conf in the /etc/kubernetes/kubelet file. I manually copy that file from /etc/kubernetes/admin.conf to $HOME. Is there any specific time interval that kubelet looks for any change in the --kubeconfig file? I will also try to set the --kubeconfig to /etc/kubernetes/admin.conf, and see if it picks it up.

@vkumar2808

This comment has been minimized.

vkumar2808 commented Oct 14, 2017

Hello,

Did it work? im having same issue

@imriss

This comment has been minimized.

imriss commented Oct 14, 2017

@vkumar2808 No, it still needs kubelet restart the way I listed above with some changes.

1. The changes to nano /etc/kubernetes/kubelet (relative to what is listed above):

KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --fail-swap-on=false --require-kubeconfig --kubeconfig=/etc/kubernetes/admin.conf"

2. The changes to Step 2:
kubelet service would not start (there is no /etc/kubernetes/admin.conf yet). However, it is fine.

3. The changes to Step 3:
Then, wait until you see the message "[kubeconfig] Wrote KubeConfig file to disk: "admin.conf""

4. The NEW Step 4:
In another terminal:
systemctl restart kubelet.service

@vkumar2808

This comment has been minimized.

vkumar2808 commented Oct 15, 2017

Thanks @imriss . Could you please send me sample of your kublet conf file?

@imriss

This comment has been minimized.

imriss commented Oct 15, 2017

@vkumar2808 This is the content of /etc/kubernetes/config file:

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://10.0.3.15:8080"
@imriss

This comment has been minimized.

imriss commented Oct 15, 2017

For the recommended $HOME/.kube/config place to put admin.conf, the procedure is the same. Except, setting the $KUBECONFIG would be optional.

@vkumar2808

This comment has been minimized.

vkumar2808 commented Oct 16, 2017

Thanks @imriss . I just deployed k8s v 1.8.1 on single server and it is seamlessly working fine. I had several sleepless night and weekend to work on it. I can seriously say that kubernetes document is lacking some important stuff. It may be because they have widen deployment for several platform.
Now Im very careful about each and every word written in kubernetes document. it has implied meaning and relation somewhere.
I tried with several plugin for centos 7 and calico worked fine for me. My main trouble was kubelet was not starting after server reboot but it is now. I will suggest people to use this link and in 15 min you can setup your local cluster.

https://docs.projectcalico.org/v2.6/getting-started/kubernetes/

My QA needs dashboard to access logs and it is great now..everything has started fine.

My next adventure will be install k8s on multi nodes and make sure after server reboot it survive. i have written several command and notes but nothing can beat calico instruction.

@luxas

This comment has been minimized.

Member

luxas commented Oct 20, 2017

Are there action items for kubeadm here? There are no official packages for Arch Linux for kubeadm AFAIK. cc @colemickens
You basically need to install the contents of the debs (converted to tar files for instance), and add this content to your rootfs and it should work.

@colemickens

This comment has been minimized.

colemickens commented Oct 20, 2017

There's no official package, but I do maintain kubeadm-bin (pacaur -S kubeadm-bin or yaourt -S kubeadm-bin, etc). It, with a couple other packages that should be pulled in transitively, along with an actual kubeadm init should be about all that's needed. I would expect multi-node to be a fairly straightforward extension.

I keep some scripts around to help with destroying/recreating my singlen node cluster here (https://github.com/colemickens/chimera-k8s-config), but you'll see the cluster boot is largely just:

  1. Install kubeadm/kubelet.
  2. kubeadm reset
  3. kubeadm init
  4. install calico

(One other small note, Arch packages don't start/enable services by default, hence why I don't have the kubelet or kubeadm packages enable/start kubelet.

@imriss

This comment has been minimized.

imriss commented Oct 21, 2017

Thanks all. As @colemickens mentioned there are many packages that provide kubernetes on Arch Linux. I have all the binaries installed. The purpose of this issue filing was to have a documented list of steps to bring up k8s, such as cgroup/systemd setting, among others. From all the comments I am learning more. Thanks again.

@vkumar2808

This comment has been minimized.

vkumar2808 commented Oct 23, 2017

@colemickens

This comment has been minimized.

colemickens commented Oct 26, 2017

@imriss What other steps did you need to do? At one point I thought I'd had to tweak the cgroup driver, but I'm not setting anything in the kubeadm config, and the only docker config I'm providing is changing the storage driver. Right now, as far as I can tell, I'm blowing away all Kubernetes config on my system and re-deploying it with every release of Kubernetes and all of those steps are captured in the script(s) linked above.

@vkumar2808 can you provide any more details? Why did Calico not work in a multi-node setup? What didn't work?

@imriss

This comment has been minimized.

imriss commented Oct 29, 2017

@colemickens sorry for the delay. One other thing is the need to offline-download the images (mentioned here and here) before kubeadm init.

@imriss

This comment has been minimized.

imriss commented Oct 29, 2017

The command to pull the images (to have it offline before calling kubeadm init):
echo "gcr.io/google_containers/etcd-amd64:3.0.17 quay.io/coreos/flannel:v0.8.0-amd64 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 gcr.io/google_containers/kube-apiserver-amd64:v1.8.0 gcr.io/google_containers/kube-controller-manager-amd64:v1.8.0 gcr.io/google_containers/kube-proxy-amd64:v1.8.0 gcr.io/google_containers/kube-scheduler-amd64:v1.8.0 gcr.io/google_containers/pause-amd64:3.0 " | tr " " "\n" | xargs -L 1 sudo docker pull

@colemickens

This comment has been minimized.

colemickens commented Nov 10, 2017

I don't understand why those are necessary. kubeadm just instructs docker to run things which will pull on demand. Or was this specific to your use-case where you need to actually run the cluster offline after initially pulling down the container assets?

@errordeveloper

This comment has been minimized.

Member

errordeveloper commented Jan 22, 2018

@imriss I cannot see a specific issue here that we need to fix. The information you have provided appears to document an installation process on Arch Linux. If you are calling for improvements of user experience on Arch Linux, please let us know and we will see what can be done. If simply would like to document the process, please write a blog post or consider opening a PR to add it to the docs (if it's something you are happy to maintain).

@timothysc

This comment has been minimized.

Member

timothysc commented Apr 7, 2018

Closing, stale and last comment.

@timothysc timothysc closed this Apr 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment