Provide an easy way to bootstrap a cluster on Mac OS #3244

pires · 2015-01-06T13:43:44Z

I've tried bootstrapping a cluster with boot2docker and Vagrant but without luck. As a developer using Mac OS X, this is quite frustrating. Also, going with a cloud provider like GCE comes at a cost.

Is it possible to provide a simple way of bootstrapping a cluster (even that one with only one node)?

proppy · 2015-01-06T13:49:40Z

This could be fixed by building a kubernetes-ready VM images using something like https://packer.io/.

That's something @kelseyhightower mentioned before.

This could also build up on the work @jbeda is doing in #2303

brendandburns · 2015-01-06T19:09:23Z

Vagrant should work. I know that we have run it successfully on OS X in
the past. What errors are you seeing?

--brendan

On Tue, Jan 6, 2015 at 5:50 AM, Johan Euphrosine notifications@github.com
wrote:

This could be fixed by building a kubernetes-ready VM images using
something like https://packer.io/.

That's something @kelseyhightower https://github.com/kelseyhightower
mentioned before.

This could also build up on the work @jbeda https://github.com/jbeda is
doing in #2303
#2303

—
Reply to this email directly or view it on GitHub
#3244 (comment)
.

arun-gupta · 2015-01-06T23:36:19Z

@brendanburns Are there instructions on how you run Kubernetes with Vagrant on OS X?

brendandburns · 2015-01-06T23:44:03Z

the standard vagrant instructions:

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md

Should work.

--brendan

On Tue, Jan 6, 2015 at 3:36 PM, Arun Gupta notifications@github.com wrote:

@brendanburns https://github.com/brendanburns Are there instructions on
how you run Kubernetes with Vagrant on OS X?

—
Reply to this email directly or view it on GitHub
#3244 (comment)
.

pires · 2015-01-08T14:35:21Z

I'm working on this with Vagrant + CoreOS and am almost getting there! I'm just trying to understand why my containers can't read from http://$KUBERNETES_RO_SERVICE_HOST:KUBERNETES_RO_SERVICE_PORT

java.net.SocketException: SocketException invoking http://10.244.61.139:80/api/v1beta1/pods: Unexpected end of file from server

Can anyone help me figuring this out? I'm running Kubernetes 0.8.0.

pires · 2015-01-08T14:39:14Z

In case it helps, here's minion output

core@node-02 ~ $ docker inspect e1e5179cbdcf
[{
    "AppArmorProfile": "",
    "Args": [
        "-c",
        "java -jar bootstrapper.jar"
    ],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "/bin/sh",
            "-c",
            "java -jar bootstrapper.jar"
        ],
        "CpuShares": 0,
        "Cpuset": "",
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "KUBERNETES_RO_SERVICE_HOST=10.244.61.139",
            "KUBERNETES_RO_SERVICE_PORT=80",
            "KUBERNETES_RO_PORT=tcp://10.244.61.139:80",
            "KUBERNETES_RO_PORT_80_TCP=tcp://10.244.61.139:80",
            "KUBERNETES_RO_PORT_80_TCP_PROTO=tcp",
            "KUBERNETES_RO_PORT_80_TCP_PORT=80",
            "KUBERNETES_RO_PORT_80_TCP_ADDR=10.244.61.139",
            "KUBERNETES_SERVICE_HOST=10.244.113.105",
            "KUBERNETES_SERVICE_PORT=443",
            "KUBERNETES_PORT=tcp://10.244.113.105:443",
            "KUBERNETES_PORT_443_TCP=tcp://10.244.113.105:443",
            "KUBERNETES_PORT_443_TCP_PROTO=tcp",
            "KUBERNETES_PORT_443_TCP_PORT=443",
            "KUBERNETES_PORT_443_TCP_ADDR=10.244.113.105",
            "HAZELCAST_SERVICE_HOST=10.244.157.151",
            "HAZELCAST_SERVICE_PORT=5701",
            "HAZELCAST_PORT=tcp://10.244.157.151:5701",
            "HAZELCAST_PORT_5701_TCP=tcp://10.244.157.151:5701",
            "HAZELCAST_PORT_5701_TCP_PROTO=tcp",
            "HAZELCAST_PORT_5701_TCP_PORT=5701",
            "HAZELCAST_PORT_5701_TCP_ADDR=10.244.157.151",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8"
        ],
        "ExposedPorts": {
            "5701/tcp": {}
        },
        "Hostname": "055d2eec-9742-11e4-b781-0800273f776f",
        "Image": "pires/hazelcast-k8s",
        "MacAddress": "",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OnBuild": null,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": null,
        "WorkingDir": "/opt/hazelcast-k8s"
    },
    "Created": "2015-01-08T14:38:14.550239572Z",
    "Driver": "btrfs",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": null,
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": null,
        "Dns": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "IpcMode": "",
        "Links": null,
        "LxcConf": null,
        "NetworkMode": "container:acea3ab5a34bc9a26e06b13e7f542adce0fd37482c7a266ad3ce5b33e5926fc1",
        "PortBindings": null,
        "Privileged": false,
        "PublishAllPorts": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": ""
        },
        "SecurityOpt": null,
        "VolumesFrom": null
    },
    "HostnamePath": "",
    "HostsPath": "/var/lib/docker/containers/acea3ab5a34bc9a26e06b13e7f542adce0fd37482c7a266ad3ce5b33e5926fc1/hosts",
    "Id": "e1e5179cbdcf80d7f18c00b8e32b68b71bf1815d1898cec40ac069fb2b3ec418",
    "Image": "cb1a377c9bfdbcee35e3252aff6bd66bcf09f09a2e570c1824f3df2cc3b3b8e7",
    "MountLabel": "",
    "Name": "/k8s_hazelcast.3bfd6b04_055d2eec-9742-11e4-b781-0800273f776f.default.etcd_055d2eec-9742-11e4-b781-0800273f776f_3c0267d3",
    "NetworkSettings": {
        "Bridge": "",
        "Gateway": "",
        "IPAddress": "",
        "IPPrefixLen": 0,
        "MacAddress": "",
        "PortMapping": null,
        "Ports": null
    },
    "Path": "/bin/sh",
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/acea3ab5a34bc9a26e06b13e7f542adce0fd37482c7a266ad3ce5b33e5926fc1/resolv.conf",
    "State": {
        "Error": "",
        "ExitCode": 1,
        "FinishedAt": "2015-01-08T14:38:24.109933118Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 0,
        "Restarting": false,
        "Running": false,
        "StartedAt": "2015-01-08T14:38:14.913602291Z"
    },
    "Volumes": {},
    "VolumesRW": {}
}
]

pires · 2015-01-08T14:40:49Z

Does this happens because I try http instead of tcp, which seems to be the protocol set in the container? I'm confused about this.

Also, if inside a container I try to access both insecure and secure endpoints

root@81221ff331dc:/# curl http://10.244.61.139
curl: (52) Empty reply from server
root@81221ff331dc:/# curl http://10.244.61.139:81
^C
root@81221ff331dc:/# curl https://10.244.61.139
^C
root@81221ff331dc:/# curl https://10.244.113.105
curl: (35) Unknown SSL protocol error in connection to 10.244.113.105:443
root@81221ff331dc:/# curl http://10.244.113.105:443
curl: (52) Empty reply from server

See that accessing other ports hangs curl and eventually return a timeout.
Ping'ing both IPs doesn't work, as well.
This proves that the minion can somehow access 10.244.61.139:80 and 10.244.113.105:443, but the master may be unable to route the response back.

pires · 2015-01-08T16:27:24Z

If anyone wants to help debug this, please use https://github.com/pires/kubernetes-vagrant-coreos-cluster. All instructions included.

brendandburns · 2015-01-08T16:52:10Z

No http should work. It looks like the service proxy on the minion can't
talk to the master. Can you ssh onto the minion and try to connect
directly from there without a container?

Brendan
On Jan 8, 2015 8:27 AM, "Paulo Pires" notifications@github.com wrote:

If anyone wants to help debug this, please use
https://github.com/pires/kubernetes-vagrant-coreos-cluster. All
instructions included.

—
Reply to this email directly or view it on GitHub
#3244 (comment)
.

pires · 2015-01-08T20:16:38Z

Do you mean I should try and access master at VM level, right?

$ curl http://172.17.8.101:8080
<html><body>Welcome to Kubernetes</body></html>

It works.

arun-gupta · 2015-01-08T20:21:59Z

@brendandburns I'm seeing #3326 when running on OS X 10.10.1. Any suggestions?

pires · 2015-01-08T20:37:15Z

Testing with flannel IPs doesn't work, same issue:

$ curl https://10.244.113.105
curl: (35) Unknown SSL protocol error in connection to 10.244.113.105:443
$ curl http://10.244.61.139
curl: (52) Empty reply from server

Any routing issues? IPTables on master and minions accepts everything by default.

pires · 2015-01-08T21:09:22Z

In case it helps

$ sudo iptables -t nat -L

(...)

Chain KUBE-PORTALS-HOST (1 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             10.244.113.105       /* kubernetes */ tcp dpt:https to:10.0.2.15:42774
DNAT       tcp  --  anywhere             10.244.61.139        /* kubernetes-ro */ tcp dpt:http to:10.0.2.15:59682

$ sudo netstat -atnp |grep 42774
tcp6       0      0 :::42774                :::*                    LISTEN      1033/kube-proxy

$ sudo netstat -atnp |grep 59682
tcp6       0      0 :::59682                :::*                    LISTEN      1033/kube-proxy

So, kube-proxy is working but is it proxying k8s api?

$ sudo netstat -atnp |grep proxy
tcp        0      0 172.17.8.102:49065      172.17.8.101:4001       ESTABLISHED 1033/kube-proxy
tcp        0      0 172.17.8.102:49064      172.17.8.101:4001       ESTABLISHED 1033/kube-proxy
tcp6       0      0 :::43425                :::*                    LISTEN      1033/kube-proxy
tcp6       0      0 :::59682                :::*                    LISTEN      1033/kube-proxy
tcp6       0      0 :::10249                :::*                    LISTEN      1033/kube-proxy
tcp6       0      0 :::42774                :::*                    LISTEN      1033/kube-proxy

etcd is being properly proxied. Shouldn't we expect a different result for ports that supposedly are proxying the API (42774 and 59682)?

pires · 2015-01-08T21:42:53Z

Routing table

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    1024   0        0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.0.2.2        0.0.0.0         255.255.255.255 UH    1024   0        0 eth0
10.244.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
10.244.2.0      0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.17.8.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1

derekwaynecarr · 2015-01-09T03:28:25Z

@pires if you are running a version of Vagrant greater >= 1.7.1 there was a change to how provisioners are setup that is causing the bug you are probably seeing. I think most of us on the project are still on 1.6.x plug-in version, so we did not encounter it recently.

As I said on #3326, I can fix this tomorrow AM to work on the latest version of Vagrant, and then you should be set to go.

For now, if you run Vagrant 1.6.x our current solution will work fine on Mac OS X.

pires · 2015-01-09T08:49:06Z

@derekwaynecarr you're a life-saver then, thanks. Will be waiting on your commit then.

pires · 2015-01-09T11:15:00Z

@derekwaynecarr tried with Vagrant 1.6.5. Same issue :(

derekwaynecarr · 2015-01-09T15:57:58Z

I am looking into this this morning with a colleague on a Mac, we will get this addressed. There should not be a need to define an alternate setup. This has been relatively stable for a long time, and is in wide use, but its possible there was a regression.

pires · 2015-01-09T16:08:10Z

So, it seems I only had to pass --public_address_override=172.17.8.101 to kube-apiserver.

derekwaynecarr · 2015-01-09T16:32:31Z

@csrwng is trying to reproduce this issue on mac from HEAD, I did upgrade to vagrant 1.7.2 on linux to try to see if that was the cause, but we actually appeared to work fine there too.

csrwng · 2015-01-09T17:23:29Z

So my builds were failing with boot2docker v1.2. I upgraded to boot2docker 1.4.1 (latest) and installed gnu-tar (brew install gnu-tar) and now 'make release' succeeds.

pires · 2015-01-09T17:43:58Z

@csrwng care to comment on #2378?

csrwng · 2015-01-09T17:53:22Z

@pires - I don't think it's the same issue. I was running 'make release' and everything would run, including e2e tests, up to the point of syncing the binaries back to the host. And then it would time out. The boot2docker update got me past that.

pires · 2015-01-09T18:22:53Z

Ah, my version works now. It's realy easy for anyone working with Vagrant and Virtualbox and you can set how many minions you want. Networking issues are fixed :-)

I'm closing this now. See kuebernetes-vagrant-coreos-cluster and if you're interested, I'd be more than glad to make a PR.

pires mentioned this issue Jan 6, 2015

docs/local_docker: fix documentation and bootstrap #2455

Closed

goltermann added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Jan 7, 2015

pires mentioned this issue Jan 8, 2015

Containers can't access k8s API pires/kubernetes-vagrant-coreos-cluster#1

Closed

pires mentioned this issue Jan 8, 2015

Kube-apiserver gives "Loaded: not found" #3326

Closed

proppy mentioned this issue Jan 9, 2015

contrib: add preconfigured cluster VM images #3361

Closed

pires closed this as completed Jan 9, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide an easy way to bootstrap a cluster on Mac OS #3244

Provide an easy way to bootstrap a cluster on Mac OS #3244

pires commented Jan 6, 2015

proppy commented Jan 6, 2015

brendandburns commented Jan 6, 2015

arun-gupta commented Jan 6, 2015

brendandburns commented Jan 6, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

brendandburns commented Jan 8, 2015

pires commented Jan 8, 2015

arun-gupta commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

derekwaynecarr commented Jan 9, 2015

pires commented Jan 9, 2015

pires commented Jan 9, 2015

derekwaynecarr commented Jan 9, 2015

pires commented Jan 9, 2015

derekwaynecarr commented Jan 9, 2015

csrwng commented Jan 9, 2015

pires commented Jan 9, 2015

csrwng commented Jan 9, 2015

pires commented Jan 9, 2015

Provide an easy way to bootstrap a cluster on Mac OS #3244

Provide an easy way to bootstrap a cluster on Mac OS #3244

Comments

pires commented Jan 6, 2015

proppy commented Jan 6, 2015

brendandburns commented Jan 6, 2015

arun-gupta commented Jan 6, 2015

brendandburns commented Jan 6, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

brendandburns commented Jan 8, 2015

pires commented Jan 8, 2015

arun-gupta commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

pires commented Jan 8, 2015

derekwaynecarr commented Jan 9, 2015

pires commented Jan 9, 2015

pires commented Jan 9, 2015

derekwaynecarr commented Jan 9, 2015

pires commented Jan 9, 2015

derekwaynecarr commented Jan 9, 2015

csrwng commented Jan 9, 2015

pires commented Jan 9, 2015

csrwng commented Jan 9, 2015

pires commented Jan 9, 2015