-
Notifications
You must be signed in to change notification settings - Fork 716
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document how to and provide scripts for running kubeadm in a container #17
Comments
From @luxas on October 27, 2016 18:14 cc @errordeveloper and @marun since they have been running systemd inside a container @andersla Be beware that running systemd this way inside a container is not supported ootb, but feel free to try it our/hack on it as it would be great for testing kubeadm that way |
From @zreigz on October 28, 2016 7:36 If you don't mind I would like take a look closer and try to fix it. |
From @andersla on October 28, 2016 8:48 @zreigz Please do!
And then:
And this is the error I get when kubeadm is being installed:
|
From @zreigz on October 28, 2016 9:10 I reproduced it and I have been working on this |
From @zreigz on October 31, 2016 7:24 There are two problems. The first one: The second one is not so trivial. Running systemctl in container fails with |
From @andersla on October 31, 2016 7:42 Great! |
From @zreigz on October 31, 2016 7:47 Because of those two lines: https://github.com/kubernetes/release/blob/master/debian/xenial/kubeadm/debian/postinst#L25
It fails on first line |
From @zreigz on October 31, 2016 7:48 this is the explanation:
|
From @zreigz on October 31, 2016 7:52 There are some configuration steps to make it work but I have to try it first. If I find something I'll let you know. |
From @zreigz on November 2, 2016 7:19 Good news. I've managed solve all issues. It needs last tests and I will post solution how to run kubeadm in Docker container |
From @andersla on November 2, 2016 7:23 Super! I will help testing as soon as it is ready! - although I am on holidays the rest of this week:) |
From @zreigz on November 2, 2016 10:13 There are two main issues regarding to installation kubeadm in Docker container. First is systemd running in container. Second is installation docker inside container. Successfully the problems were fixed. Here is the Dockerfile which must be used to prepare Ubuntu image
I use this command to build the image in the directory containing the Dockerfile
Now you can run prepared image and finish kubeadm installation.
Find running container ID
Now you can open container console:
This is your script (with small modifications) to install kubeadm
And finally you can execute
Everything works the same like on local machine. |
From @SuperStevenZ on November 17, 2016 7:21 @zreigz That solved the same problem of mine, thanks! |
From @zreigz on November 17, 2016 7:30 No problem :) |
We should set up a CI with docker-in-docker stuff. @errordeveloper @zreigz Can you take this on? |
Sounds good for me. For sure we need to put all this stuff in docker image plus some config/start scripts to distinguish between master and node. Good start would be to create project for it like kubernetes/kubeadm-docker. It would be also right place for Dockerfile ,scripts and documentation |
Create that as a private project first under zreigz/ and eventually we'll probably merge that code into this repo. But first, prototype in your own space and we'll see how it goes. |
Real assignee is @zreigz |
Yes good point. I will do it. Next week (Monday, Tuesday) I am on conference so I will start on Wednesday. |
I think this will solve weave issue: weaveworks/weave#2659 |
@andersla Yes, that seems to fix the problem. Can you try a build from HEAD? |
I used the latest image from weaveworks/weave-kube I also used the latest yaml-template https://github.com/weaveworks/weave/blob/master/prog/weave-kube/weave-daemonset.yaml Unfortunately kube-dns didn't work (it is stuch in ContainerCreating. The error message from kubelet after starting weave is:
If I only started the master node and not joining another node then kubedns came up OK when I applied weave.yaml I also tested the weave.yaml with latest weave-kube on a Vagrant installation and not in my docker-experiment and then it all worked. This is the weave.yaml I used for
|
hey guys, i ran into this thread and it freaking rocks! great stuff. i really want to use this approach for CI against our repo (which is fairly complex, honestly). we have a Helm/Tiller requirement for launching quite a few charts for CI. have any of you run into this, or have suggestions for getting this going? Tiller seems to barf all over itself in this situation:
i may try with other SDN's. we've been using Calico so far because L3 is a little more straightforward to troubleshoot in hacky situations, but if Weave is better (since it's L2)...I'll try whatever gets us past the Tiller issue. i think Tiller is unhappy because at the end of the day it appears that it associates with 127.0.0.1...and i've seen that cause problems in the past testing other things. any input would be amazing. again, really awesome props to the folks who are hacking things up! thank you!! |
Hi! Great that we are more people wanting this to work. I don't have experience with calico. On the cloud we are running Weave so thats what i wanted to get working on this project. But I am stuck and haven't had time to dig further why kube-dns doesn't come up when I apply Weave as described above. |
Now the latest stable weave is working better than before....
..but unfortunately still same issue with kube-dns not comming up, stuck in ContainerCreating:
and after applying weave, this error message stops: and instead once I see:
|
If I use Flannel as network plugin instead it works.
|
So if you use Flannel, then everything is working, here is the complete setup: Dockerfile:
Build it with: And then run: Wait a few (10-15) seconds until systemd and docker is up and running Then I start kubeadm inside the running container: When it is initiated I start a second "worker" node: And after a few seconds (until systemd and docker is up) join the master: When they have joined, - enter master and apply workaround for crashing kube-proxy:
Finally apply flannel overlay network: I had no problem installing Helm, Traefic or GlusterFS in Kubernetes in this setting:) |
kubeadm-dind-cluster basically does what the last comment outlined, providing automation so you don't have to type the commands manually (although as of now it uses CNI bridge plugin with some hacks instead of flannel, but this I'll fix quite soon). Some of the changes coming quite soon in kubeadm-dind-cluster:
|
kubeadm-dind-cluster also provides automation for e2e tests. Another interesting trait of it is that you can use same remote docker engine for both building k8s and running kubeadm-dind-cluster without copying back the binaries (it pulls them directly from build data container), which may be important if you're working with remote docker over slow connection. ... forgot to mention it configures local kubectl for you so you don't need to do |
As I already mentioned, while DIND may seem easy on surface, you can have some unexpected problems with it. Some of the problems are already fixed in kubeadm-dind-cluster and base image it uses. E.g. you need to do some mounts, also you need to use |
@ivan4th Thanks for all the work you've been doing with kubeadm and dind :) After looking quickly, I found some points that we might want to do before a possible MVP:
What do you think? Thanks for the awesome start, I can't wait to actually integrate this into something kubeadm official 👍 cc @jbeda @lukemarsden @errordeveloper @mikedanese @timothysc @sttts |
if we can devel-build, kubeadm-local-up-cluster that would be fantastic. |
I don't know really... @ivan4th |
Overall k-d-c is quite usable in its current form IMO. It also has its own public CI based on Travis (BTW I also succeeded in running DIND on CircleCI if it's of some interest) |
@luxas Maybe we can use @andersla's solution instead of a full DIND cluster? If so, would we need to host the Docker image anywhere, or just document what the Dockerfile looks like? It'd be great if we can get a fix out for this issue for 1.9 |
I don't have cycles to work on this. If anyone else, can please do! |
@jamiehannaford problem is, much of "full" DIND cluster is dedicated to handling numerous problems that arise from "simple" DIND usage. These may be quite obscure at times, see e.g. kubernetes-retired/kubeadm-dind-cluster@405c8be (I think I'll need to submit a fix for k8s for this). As of kubeadm-dind-cluster, it's still quite usable and I try to keep it up-to-date( @danehans and @pmichali are using it for k8s IPv6 e2e testing and Virtlet uses it to run it's e2e tests on CircleCI), although I spend a lot of time on other projects so I didn't manage to rewrite it in Go yet. |
We talked about this in the SIG meeting yesterday, and we're gonna close the issue. |
From @andersla on October 27, 2016 18:8
When trying to install Kubeadm inside Ubuntu 16.04 docker container it fails.
BUG REPORT
Kubernetes version (use
kubectl version
):latest
Environment:
Ubuntu 16.04 Docker container
What happened:
When trying to install Kubeadm inside Ubuntu 16.04 docker container it fails.
My Idea was to use one docker container as master "node" and a second container as a worker "node" (kubernetes in docker)
Is this a systemd issue? (something I came across when "googling" for answers)
Inside Ubuntu 16.04 docker image I install with : apt-get install -y kubeadm
setup log:
Copied from original issue: kubernetes/kubernetes#35712
The text was updated successfully, but these errors were encountered: