Basic Kubernetes Integration #441

paulbellamy · 2015-09-04T12:05:53Z

@tomwilkie, @peterbourgon any feedback would be great.

I'm not super-happy with the result, as there are still some shortcomings:

We can't yet correlate connections going through the kube-proxy into pseudo service nodes, which means that all connections go through the "Uncontained" pseudo-node. This is because I haven't been able to figure out how to do it.
We can't ever get full resolution on connections going through a service, as it might include a userspace hop (depending on configuration).
There is a bug in k8s where some pods are not listed in the API. I've raised an issue, and added a workaround.
Need to pass a --probe.kuberetes true flag on the master node to enable it (for now).
There may be some issues rendering incoming internet connections from external load-balancers, depending on how they do that routing, but none beyond what scope would already have.
Kubernetes uses a lot of ephemeral (high) ports, which breaks the assumption about a higher port number being the client, and causes the graph to be laid out strangely.

In the example, there are:

3 php (frontend) pods, accepting incoming http requests, and making requests to redis
1 redis-master pod, handling writes
2 redis-slave pods (one per minion), handling reads. The redis-slave pods maintain an open socket to redis-master for replication.

Screenshots of it:
Pods view. I'm not sure if we should show the "2 containers" under each node, or if we should skip that.

Pods-by-Service view, where pods are grouped when they belong to a common service

As you can see, all the connections routing through "Uncontained" makes the output mostly useless. However, we can filter out the loads of kube-system pods, which clutter up the view considerably.

probe/docker/container.go

@@ -16,7 +16,6 @@ import (
 	"time"

 	docker "github.com/fsouza/go-dockerclient"
-


probe/kubernetes/client.go

+func (k *client) RemoveEventListener(c chan watch.Event) error {
+	k.lock.Lock()
+	defer k.lock.Unlock()
+	return k.removeEventListener(c)


probe/kubernetes/reporter.go

+	)
+	err := r.client.WalkServices(func(s Service) error {
+		nodeID := report.MakeServiceNodeID(s.Namespace(), s.Name())
+		result.Nodes[nodeID] = s.GetNode()


render/mapping.go

+	if s := strings.SplitN(id, "/", 2); len(s) == 2 {
+		result.LabelMajor = s[1]
+		result.Node.Metadata[kubernetes.Namespace] = s[0]
+		result.Node.Metadata[kubernetes.PodName] = s[1]


tomwilkie · 2015-10-01T15:56:39Z

Quick first pass; it looks pretty good. Shame we can't make the connections work properly; something to come back to perhaps.

paulbellamy · 2015-10-01T16:03:15Z

@tomwilkie I suspect the connections is something obvious I'm missing, but yeah. Definitely to revisit.

tomwilkie · 2015-10-01T16:06:52Z

Make sure we have a bug to track the connection improvements.

app/router.go

@@ -96,6 +97,9 @@ func makeReportPostHandler(a xfer.Adder) http.HandlerFunc {
 			return
 		}
 		a.Add(rpt)
+		if len(rpt.Pod.Nodes) > 0 {
+			addKubernetesTopologies()
+		}


paulbellamy · 2015-10-04T20:54:42Z

I believe I've addressed all concerns with this, so if you could have another quick look, and let's merge this.

tomwilkie · 2015-10-05T12:12:40Z

I haven't tested the code myself (involved setting up k8s). And I'm hesitant as I don't think anyone will be regularly running this, so I very much expect it to bit rot.

But given its not enabled by default, the risks of merging this are minimal. LGTM.

tomwilkie · 2015-10-05T12:12:49Z

Pls squash!

peterbourgon · 2015-10-05T12:29:25Z

I'll give it a review at EOD. Please wait until then :)

app/router.go

+	r.RLock()
+	defer r.RUnlock()
+	return r.items
+}


peterbourgon · 2015-10-05T16:31:56Z

re: the racy thing, in the best case, you could return a deep copy, but I recognize that's super annoying... maybe just a warning comment on get and list methods to not modify the return values?

Other than those little nits, LGTM.

* Added helper for installing scope on gcloud * Added topologies Pods and Pods-by-Service * Uses k8s.io/kubernetes/pkg/client/cache for the client * Filter kube-system nodes by default * Only show the k8s topologies if we've received a non-empty k8s report

Basic Kubernetes Integration

tomwilkie reviewed Sep 4, 2015
View reviewed changes

probe/docker/container.go

@@ -16,7 +16,6 @@ import (

"time"

docker "github.com/fsouza/go-dockerclient"

This comment was marked as abuse.

Sign in to view

paulbellamy force-pushed the kubernetes branch from e209cca to 257c3db Compare September 4, 2015 12:14

peterbourgon reviewed Sep 7, 2015
View reviewed changes

probe/kubernetes/client.go

func (k *client) RemoveEventListener(c chan watch.Event) error {

k.lock.Lock()

defer k.lock.Unlock()

return k.removeEventListener(c)

This comment was marked as abuse.

Sign in to view

paulbellamy force-pushed the kubernetes branch 2 times, most recently from 7134805 to fe70172 Compare September 8, 2015 16:27

paulbellamy force-pushed the kubernetes branch 5 times, most recently from caad1b2 to fd06706 Compare September 24, 2015 08:16

paulbellamy force-pushed the kubernetes branch 10 times, most recently from 2917be9 to 914ed83 Compare October 1, 2015 08:28

paulbellamy changed the title ~~[WIP] Kubernetes Integration~~ Basic Kubernetes Integration Oct 1, 2015

tomwilkie reviewed Oct 1, 2015
View reviewed changes

probe/kubernetes/reporter.go

)

err := r.client.WalkServices(func(s Service) error {

nodeID := report.MakeServiceNodeID(s.Namespace(), s.Name())

result.Nodes[nodeID] = s.GetNode()

This comment was marked as abuse.

Sign in to view

tomwilkie reviewed Oct 1, 2015
View reviewed changes

render/mapping.go

if s := strings.SplitN(id, "/", 2); len(s) == 2 {

result.LabelMajor = s[1]

result.Node.Metadata[kubernetes.Namespace] = s[0]

result.Node.Metadata[kubernetes.PodName] = s[1]

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

paulbellamy mentioned this pull request Oct 1, 2015

Expose Kubernetes service connections #538

Closed

peterbourgon reviewed Oct 2, 2015
View reviewed changes

app/router.go

@@ -96,6 +97,9 @@ func makeReportPostHandler(a xfer.Adder) http.HandlerFunc {

return

}

a.Add(rpt)

if len(rpt.Pod.Nodes) > 0 {

addKubernetesTopologies()

}

This comment was marked as abuse.

Sign in to view

paulbellamy force-pushed the kubernetes branch from e9b28e9 to 7c6878d Compare October 2, 2015 12:32

peterbourgon reviewed Oct 5, 2015
View reviewed changes

app/router.go

r.RLock()

defer r.RUnlock()

return r.items

}

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

Basic Kubernetes integration

be1d1d5

* Added helper for installing scope on gcloud * Added topologies Pods and Pods-by-Service * Uses k8s.io/kubernetes/pkg/client/cache for the client * Filter kube-system nodes by default * Only show the k8s topologies if we've received a non-empty k8s report

paulbellamy force-pushed the kubernetes branch from 671277e to be1d1d5 Compare October 6, 2015 11:56

paulbellamy added a commit that referenced this pull request Oct 6, 2015

Merge pull request #441 from weaveworks/kubernetes

6ad182a

Basic Kubernetes Integration

paulbellamy merged commit 6ad182a into master Oct 6, 2015

paulbellamy deleted the kubernetes branch October 6, 2015 13:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Basic Kubernetes Integration #441

Basic Kubernetes Integration #441

paulbellamy commented Sep 4, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

tomwilkie commented Oct 1, 2015

paulbellamy commented Oct 1, 2015

tomwilkie commented Oct 1, 2015

This comment was marked as abuse.

paulbellamy commented Oct 4, 2015

tomwilkie commented Oct 5, 2015

tomwilkie commented Oct 5, 2015

peterbourgon commented Oct 5, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

peterbourgon commented Oct 5, 2015

		@@ -16,7 +16,6 @@ import (
		"time"

		docker "github.com/fsouza/go-dockerclient"

Basic Kubernetes Integration #441

Basic Kubernetes Integration #441

Conversation

paulbellamy commented Sep 4, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

tomwilkie commented Oct 1, 2015

paulbellamy commented Oct 1, 2015

tomwilkie commented Oct 1, 2015

This comment was marked as abuse.

paulbellamy commented Oct 4, 2015

tomwilkie commented Oct 5, 2015

tomwilkie commented Oct 5, 2015

peterbourgon commented Oct 5, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

peterbourgon commented Oct 5, 2015