No service dns resolution...

[derailed]

Hi,

Just taking km 0.5.0 out for a spin. I have a successful pure k8s aws deploy and trying out km locally reusing my k8s recipes. Having issue linking up to services. Think it's a known issue but wanted to check status/reason why this does not seem to work.

Case and point. I have a cassandra pod and service. Launch a virtualbox vm and try to access my cqlsh in my pod using cassandra service. Works with pure k8s 1.2.0a8 but fail with Name or service not known on km 0.5.0.

ku describe svc cassandra - shows the endpoint is correctly assigned and cqlsh command works with the ips but not the service name as expected.

Thank you!

[sebgoa]

@derailed that's expected behavior right now. You need to deploy the DNS add-on in your virtualbox kmachine.

You can try to use this:
https://github.com/skippbox/configk8s

[derailed]

@Runseb Thanks! I've looked into this project and after some mods still no dice. Getting invalid range on dns clusterIP 10.0.0.10. Any ideas?

[tdeheurles]

@derailed : just in case, are you connected to the good cluster with kubectl ? (Just add the same kind of issue) ....

[derailed]

yes I think so. Running kubeclt get rc --namespace kube-system shows the
dns rc. The service creation fails for 2 reasons:

1- Invalid yaml - successThreshold/failureThreshold
2- Invalid dns ip range.

Also running ku 1.2.0a8 locally not sure if that could be an issue here??

On Tue, Mar 8, 2016 at 11:12 AM, thibault deheurles <
notifications@github.com> wrote:

@derailed https://github.com/derailed : just in case, are you connected
to the good cluster with kubectl ? (Just add the same kind of issue) ....

—
Reply to this email directly or view it on GitHub
#33 (comment).

[derailed]

@Runseb - digging further looks like kube2sky container was failing looking up the k8s server - was able to pass in the master url env and the service is now up and running but still dns does not seems to work ;-(

[sebgoa]

@derailed how did you install kmachine ?

[sebgoa]

@derailed can you give me your cassandra rc and service definitions so I can try it and debug ? a gist would be fine. thanks

[derailed]

Tx for looking into this @Runseb. I've spent most of the day on this
yesterday and could not find the magic incantation to make this work. Think
the problem might be with k8s 1.1.2 + skydns not mixing. Also the base
image run it's own etcd as does kube2sky which might also result in
conflicts. The main problem is having to specify k8s master arg on kube2sky
as otherwise it would not find it and the container will fail to start
trying to reach the master. This could be a sign of the symptoms ie why
couldn't it find the k8s master?? I was able to see services registration
with etcd on kube2sky but still cassandra dns in my examples did not
resolve. Also try the fully qualified dns name and could not resolve either
outside of kube2sky.

As far as repro you should be able to use the cassandra sample in the k8s
repo.

Sadly I would have loved to make this work but running out of time and
reverting to my vagrant solution ;-(

I still feel kmachine is such a more elegant sol than spinning vagrants and
hope one day it could be rolled into k8s. But this functionality has to
work out of the box ie skydns need to be part of the distro at the bare
minimum. Imho this is what makes k8s great for my use cases ie one should
not have to run addons separately may be for the ui+dm but think dns must
be a basic tenant of kmachine.

FYI might also be a good idea to line up with k8s revs. ie k8s 1.1.2 =
kmachine 1.1.2 - will make future visitors happier.

Good luck and thanks for your great support!

On Wed, Mar 9, 2016 at 5:31 AM, runseb notifications@github.com wrote:

@derailed https://github.com/derailed can you give me your cassandra rc
and service definitions so I can try it and debug ? a gist would be fine.
thanks

—
Reply to this email directly or view it on GitHub
#33 (comment).

[sebgoa]

thanks for the detailed info @derailed , I will try the cassandra example.
I did spent quite a bit of time testing DNS because it was a pain :) with the service accounts.

The basics should work but I will check with cassandra.

I do agree the DNS add-on should be built-in to provide a totally out of the box solution.

And I am about to change the version scheme to line up with k8s indeed.

So thoughts are similar.

thanks for the feedback

[tdeheurles]

To my mind, this kind of project will mostly be used by developper. I’m not
sure that all the security layer (that really complicate setup) is really
usefull in that kind of scenario ... As it’s already done ... Too late for
a removing ...

I think the addons need to be in the pack.

Must have:

• DNS
• proxy to access the service

The script are present so we need a way to maintain with some tests ...
Easy to say ...

Le mer. 9 mars 2016 17:05, runseb notifications@github.com a écrit :

thanks for the detailed info @derailed https://github.com/derailed , I
will try the cassandra example.
I did spent quite a bit of time testing DNS because it was a pain :) with
the service accounts.

The basics should work but I will check with cassandra.

I do agree the DNS add-on should be built-in to provide a totally out of
the box solution.

And I am about to change the version scheme to line up with k8s indeed.

So thoughts are similar.

thanks for the feedback

—
Reply to this email directly or view it on GitHub
#33 (comment).

[sebgoa]

@derailed I think I found the issue, which has to do with the fact that the cassandra pods need 512mb of HEAP. kmachine by default only gets 512mb of RAM, so the containers fail to start. If you start a kmachine with enough RAM it will work.

Below I am putting in a trace, where I use github.com/skippbox/configk8s to start the DNS Addon.

$ kmachine create -d digitalocean --digitalocean-size "4gb" koko
$ ./deploy-addons.sh 
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                 READY     STATUS    RESTARTS   AGE
default       koko-koko            5/5       Running   1          33m
kube-system   kube-dns-v11-zcsvg   3/4       Running   0          42s
$ KUBE_ROOT="https://raw.githubusercontent.com/kubernetes/kubernetes/master"
$ kubectl create -f "${KUBE_ROOT}/examples/cassandra/cassandra-service.yaml"
service "cassandra" created
$ kubectl create -f "${KUBE_ROOT}/examples/cassandra/cassandra.yaml"
pod "cassandra" created
$ kubectl create -f "${KUBE_ROOT}/examples/cassandra/cassandra-controller.yaml"
replicationcontroller "cassandra" created
$ kubectl exec busybox -- nslookup cassandra.default
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      cassandra.default
Address 1: 10.0.0.96
$ kubectl exec -ti cassandra -- nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID                               Rack
UN  172.17.0.5  68.33 KB   256     100.0%            4dde2b3c-358f-488a-aaed-39a7ec872593  rack1
UN  172.17.0.4  52.28 KB   256     100.0%            a70daa5e-e751-4534-9ef7-5a28ac9ea0ab  rack1
$ kubectl scale rc cassandra --replicas=3
replicationcontroller "cassandra" scaled
$ kubectl get rc
CONTROLLER   CONTAINER(S)   IMAGE(S)                             SELECTOR        REPLICAS   AGE
cassandra    cassandra      gcr.io/google-samples/cassandra:v8   app=cassandra   3          10m
$ kubectl exec -ti cassandra -- nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID                               Rack
UN  172.17.0.6  52.29 KB   256     64.4%             b783d244-9740-44b9-bb90-06b18ef2105d  rack1
UN  172.17.0.5  84.88 KB   256     69.1%             4dde2b3c-358f-488a-aaed-39a7ec872593  rack1
UN  172.17.0.4  68.83 KB   256     66.5%             a70daa5e-e751-4534-9ef7-5a28ac9ea0ab  rack1

I did find that the cassandra nodes take a bit of time to register and nodetool status is a bit slow to return.

[sebgoa]

I will put up a new version of kmachine which also works on all cloud providers and I will work to embed the DNS Add-on.

[derailed]

@Runseb - Thanks! Correct about cassandra. I see you are using digitalocean

• I was using virtualbox for a local install and could not get dns to
  work. Are u able to run thru the same scenario with the virtualbox driver
  successfully?

On Thu, Mar 10, 2016 at 9:35 AM, runseb notifications@github.com wrote:

I will put up a new version of kmachine which also works on all cloud
providers and I will work to embed the DNS Add-on.

—
Reply to this email directly or view it on GitHub
#33 (comment).

[sebgoa]

@derailed just tested it on virtual box as well. it seems to work on my end.

$ kmachine create -d virtualbox --virtualbox-memory "2048" foobar
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                 READY     STATUS    RESTARTS   AGE
default       busybox              1/1       Running   0          1m
default       foobar-foobar        5/5       Running   0          5m
kube-system   kube-dns-v11-a65n3   4/4       Running   0          1m
$ KUBE_ROOT="https://raw.githubusercontent.com/kubernetes/kubernetes/master"
$ kubectl create -f "${KUBE_ROOT}/examples/cassandra/cassandra-service.yaml"
service "cassandra" created
$ kubectl create -f "${KUBE_ROOT}/examples/cassandra/cassandra.yaml"
pod "cassandra" created
$ kubectl create -f "${KUBE_ROOT}/examples/cassandra/cassandra-controller.yaml"
replicationcontroller "cassandra" created
$ kubectl get pods
NAME              READY     STATUS    RESTARTS   AGE
busybox           1/1       Running   0          4m
cassandra         1/1       Running   0          1m
cassandra-mpmcu   1/1       Running   0          1m
foobar-foobar     5/5       Running   0          8m
$ kubectl exec busybox -- nslookup kubernetes.default
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      kubernetes.default
Address 1: 10.0.0.1
$ kubectl exec busybox -- nslookup cassandra.default
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      cassandra.default
Address 1: 10.0.0.210
$ kubectl exec -ti cassandra -- nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID                               Rack
UN  172.17.0.5  9.71 KB    256     100.0%            5f0ed08d-624a-4527-8b8f-62d17c9c7a91  rack1
UN  172.17.0.4  68.22 KB   256     100.0%            4b9c6459-d09f-4f63-b604-45b699c45357  rack1

[sebgoa]

and my scripts to deploy the DNS Add-on are at https://github.com/skippbox/configk8s

[sebgoa]

I am going to close this one, because I tested it quite a bit. Original error seems to be due to the lack of RAM on the machine.

[derailed]

Thanks @Runseb! - I'll try to circle back and give it another shot. Switch
out to vagrant for the being but would love to get my local on kmachine.

Do you have plans on supporting k8s 1.2.0 as it was just released?

On Thu, Mar 17, 2016 at 3:02 PM, runseb notifications@github.com wrote:

Closed #33 #33.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#33 (comment)

[sebgoa]

yes, I pushed a change to master for 1.2.0. We are working on bundling the add ons daemon in kmachine so DNS starts automatically.

[derailed]

That's excellent news. Thank you for making this happen!

Also a bit on a tangent here but what are your plans (if any) to support
for uploading local docker images or may be support a docker registry on
kmachine?
I know there have been talks about this for the last couple years on the
k8s forum but haven't seen viable options??

Think it would be killer if suck a feature existed in kmachine.

Thanks!!

On Fri, Mar 18, 2016 at 1:41 AM, runseb notifications@github.com wrote:

yes, I pushed a change to master for 1.2.0. We are working on bundling the
add ons daemon in kmachine so DNS starts automatically.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#33 (comment)

[sebgoa]

I believe there is a registry Add-on now, but I have not tried it yet.

[derailed]

Ah! I did not see that. Thank you! I'll take a look...

On Fri, Mar 18, 2016 at 11:16 AM, runseb notifications@github.com wrote:

I believe there is a registry Add-on now, but I have not tried it yet.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#33 (comment)