TestWatch {etcd} #31262

k8s-github-robot · 2016-08-23T15:58:05Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16135/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

The text was updated successfully, but these errors were encountered:

sttts · 2016-08-23T17:01:52Z

This blocks the queue it seems.

spxtr · 2016-08-23T17:12:34Z

Assigning according to this.

ncdc · 2016-08-23T17:31:14Z

I haven't been able to reproduce locally, but I do see some odd output. It appears that TestWatch tries to connect to the sockets for the etcd nodes created in the other tests in this file.

=== RUN   TestWatch
2016-08-23 08:51:09.610505 I | integration: launching node4516792630069338985 (unix://localhost:node4516792630069338985.sock.bridge)
2016-08-23 08:51:09.615184 I | etcdserver: name = node4516792630069338985
2016-08-23 08:51:09.615307 I | etcdserver: data dir = /tmp/etcd487135095
2016-08-23 08:51:09.615389 I | etcdserver: member dir = /tmp/etcd487135095/member
2016-08-23 08:51:09.615463 I | etcdserver: heartbeat = 10ms
2016-08-23 08:51:09.615537 I | etcdserver: election = 100ms
2016-08-23 08:51:09.615698 I | etcdserver: snapshot count = 0
2016-08-23 08:51:09.615790 I | etcdserver: advertise client URLs = unix://127.0.0.1:21012.15635.sock
2016-08-23 08:51:09.615887 I | etcdserver: initial advertise peer URLs = unix://127.0.0.1:21011.15635.sock
2016-08-23 08:51:09.616072 I | etcdserver: initial cluster = node4516792630069338985=unix://127.0.0.1:21011.15635.sock
2016-08-23 08:51:09.625353 I | etcdserver: starting member 9f295cd627bdb62e in cluster df8194b67a5d422b
2016-08-23 08:51:09.625562 I | raft: 9f295cd627bdb62e became follower at term 0
2016-08-23 08:51:09.625670 I | raft: newRaft 9f295cd627bdb62e [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-08-23 08:51:09.625739 I | raft: 9f295cd627bdb62e became follower at term 1
2016-08-23 08:51:09.641947 I | etcdserver: set snapshot count to default 10000
2016-08-23 08:51:09.642128 I | etcdserver: starting server... [version: 3.0.6, cluster version: to_be_decided]
2016-08-23 08:51:09.644212 I | membership: added member 9f295cd627bdb62e [unix://127.0.0.1:21011.15635.sock] to cluster df8194b67a5d422b
2016-08-23 08:51:09.649572 I | integration: launched node4516792630069338985 (unix://localhost:node4516792630069338985.sock.bridge)
2016-08-23 08:51:09.706646 I | raft: 9f295cd627bdb62e is starting a new election at term 1
2016-08-23 08:51:09.706803 I | raft: 9f295cd627bdb62e became candidate at term 2
2016-08-23 08:51:09.706884 I | raft: 9f295cd627bdb62e received vote from 9f295cd627bdb62e at term 2
2016-08-23 08:51:09.707004 I | raft: 9f295cd627bdb62e became leader at term 2
2016-08-23 08:51:09.707092 I | raft: raft.node: 9f295cd627bdb62e elected leader 9f295cd627bdb62e at term 2
2016-08-23 08:51:09.708994 I | etcdserver: setting up the initial cluster version to 3.0
2016-08-23 08:51:09.709347 I | etcdserver: published {Name:node4516792630069338985 ClientURLs:[unix://127.0.0.1:21012.15635.sock]} to cluster df8194b67a5d422b
2016-08-23 08:51:09.709945 N | membership: set the initial cluster version to 3.0
2016-08-23 08:51:10.088592 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node4390910715892619484.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node4390910715892619484.sock.bridge" <nil>}
2016-08-23 08:51:10.174713 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node3201104790320849340.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node3201104790320849340.sock.bridge" <nil>}
2016-08-23 08:51:10.321057 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1203876774733832651.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1203876774733832651.sock.bridge" <nil>}
2016-08-23 08:51:10.457051 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node143222603956848319.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node143222603956848319.sock.bridge" <nil>}
2016-08-23 08:51:10.600320 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node235110591907564707.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node235110591907564707.sock.bridge" <nil>}
2016-08-23 08:51:11.509978 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node4390910715892619484.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node4390910715892619484.sock.bridge" <nil>}
2016-08-23 08:51:11.919793 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node3201104790320849340.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node3201104790320849340.sock.bridge" <nil>}
2016-08-23 08:51:12.127168 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node143222603956848319.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node143222603956848319.sock.bridge" <nil>}
2016-08-23 08:51:12.149581 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1203876774733832651.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1203876774733832651.sock.bridge" <nil>}
2016-08-23 08:51:12.331657 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node235110591907564707.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node235110591907564707.sock.bridge" <nil>}
2016-08-23 08:51:14.009577 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node4390910715892619484.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node4390910715892619484.sock.bridge" <nil>}
2016-08-23 08:51:14.188795 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node3201104790320849340.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node3201104790320849340.sock.bridge" <nil>}
2016-08-23 08:51:14.646463 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node235110591907564707.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node235110591907564707.sock.bridge" <nil>}
2016-08-23 08:51:14.680831 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node143222603956848319.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node143222603956848319.sock.bridge" <nil>}
2016-08-23 08:51:15.030171 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1203876774733832651.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1203876774733832651.sock.bridge" <nil>}
2016-08-23 08:51:17.694162 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node4390910715892619484.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node4390910715892619484.sock.bridge" <nil>}
2016-08-23 08:51:18.680946 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node3201104790320849340.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node3201104790320849340.sock.bridge" <nil>}
2016-08-23 08:51:18.787490 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1203876774733832651.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1203876774733832651.sock.bridge" <nil>}
2016-08-23 08:51:19.364232 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node143222603956848319.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node143222603956848319.sock.bridge" <nil>}
2016-08-23 08:51:19.447686 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node235110591907564707.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node235110591907564707.sock.bridge" <nil>}
2016-08-23 08:51:23.218406 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node4390910715892619484.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node4390910715892619484.sock.bridge" <nil>}
2016-08-23 08:51:25.070712 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node3201104790320849340.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node3201104790320849340.sock.bridge" <nil>}
2016-08-23 08:51:25.792412 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1203876774733832651.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1203876774733832651.sock.bridge" <nil>}
2016-08-23 08:51:26.697937 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node235110591907564707.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node235110591907564707.sock.bridge" <nil>}
2016-08-23 08:51:27.052865 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node143222603956848319.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node143222603956848319.sock.bridge" <nil>}
2016-08-23 08:51:34.104028 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node4390910715892619484.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node4390910715892619484.sock.bridge" <nil>}
2016-08-23 08:51:36.834508 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node3201104790320849340.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node3201104790320849340.sock.bridge" <nil>}
2016-08-23 08:51:37.381142 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1203876774733832651.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1203876774733832651.sock.bridge" <nil>}
2016-08-23 08:51:37.658273 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node235110591907564707.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node235110591907564707.sock.bridge" <nil>}
2016-08-23 08:51:38.809849 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node143222603956848319.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node143222603956848319.sock.bridge" <nil>}
2016-08-23 08:51:43.063297 I | integration: terminating node4516792630069338985 (unix://localhost:node4516792630069338985.sock.bridge)
2016-08-23 08:51:43.067599 I | v3rpc/grpc: transport: http2Client.notifyError got notified that the client transport was broken EOF.
2016-08-23 08:51:43.075521 I | integration: terminated node4516792630069338985 (unix://localhost:node4516792630069338985.sock.bridge)
--- FAIL: TestWatch (33.47s)
    resttest.go:1251: unexpected timeout from result channel

k8s-github-robot · 2016-08-23T21:04:42Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16144/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

k8s-github-robot · 2016-08-23T22:56:05Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16148/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

k8s-github-robot · 2016-08-24T03:36:01Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16156/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

k8s-github-robot · 2016-08-24T06:55:55Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16162/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

k8s-github-robot · 2016-08-24T07:57:01Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16164/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

wojtek-t · 2016-08-24T08:06:24Z

This is definitely related to this PR:
#30890

that switched unit tests to use v3 API of etcd.

@hongchaodeng @xiang90 @timothysc - seems like a blocker.

@lavalamp - FYI

k8s-github-robot · 2016-08-24T10:38:19Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16169/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

timothysc · 2016-08-24T13:25:37Z

FWIW I ran unit-tests locally tens of times without failure.

I wonder if it's similar to the resource issue we found a while ago on our jenkins env /cc @ixdy

k8s-github-robot · 2016-08-24T19:10:00Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16183/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

lavalamp · 2016-08-24T20:29:55Z

I made a rollback PR. #31371

hongchaodeng · 2016-08-24T21:09:05Z

Some findings on logs from last failure -- the test infra is leaking clients

Here are the logs for failed test:

=== RUN   TestWatch
...
2016-08-24 12:04:09.195133 I | integration: launched node417290545105324029 (unix://localhost:node417290545105324029.sock.bridge)
...
2016-08-24 12:04:09.210408 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node5478120498384533947.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node5478120498384533947.sock.bridge" <nil>}
2016-08-24 12:04:09.360063 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node1204988732873562.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node1204988732873562.sock.bridge" <nil>}
...
... a bunch of "localhost:nodeXXX.sock.bridge: connect: no such file or directory" failures
...
--- FAIL: TestWatch (32.21s)
    resttest.go:1251: unexpected timeout from result channel

Basically, TestWatch client should talk to "node417290545105324029".
For "node1204988732873562", it's created in TestResourceLocation earlier. The connection failed because TestResourceLocation already finished and unix socket files were cleaned up. But there is a client that keeps talking to it.
It's also true for a couple of other similar errors.
Apparently, this is leaking clients.

hongchaodeng · 2016-08-24T21:16:00Z

After digging into the code, we are obviously not closing any clients in a lot of tests.
Even though we rollback to v2, it's still leaking clients.

hongchaodeng · 2016-08-24T23:39:56Z

Submit a fix: #31390

k8s-github-robot · 2016-08-25T01:43:52Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16194/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

k8s-github-robot · 2016-08-25T01:43:53Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16195/

Failed: TestWatch {etcd}

resttest.go:1299: unexpected timeout from result channel

k8s-github-robot · 2016-08-25T03:20:05Z

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16198/

Failed: TestWatch {etcd}

resttest.go:1251: unexpected timeout from result channel

Automatic merge from submit-queue Revert "Enable v3 Client as the default on UTs" Reverts #30890 Fix flake problems (#31262)

lavalamp · 2016-08-25T22:01:59Z

Looks like we stopped seeing these after #31371 merged.

Automatic merge from submit-queue return destroy func to clean up internal resources of storage What? Provide a destroy func to clean up internal resources of storage. It changes **unit tests** to clean up resources. (Maybe fix integration test in another PR.) Why? Although apiserver is designed to be long running, there are some cases that it's not. See #31262 (comment) We need to gracefully shutdown and clean up resources.

wojtek-t · 2016-08-26T06:30:29Z

Which was expected BTW :)

timothysc · 2016-08-26T21:43:36Z

I've nearly got the deferFunc() plumbed through to the rest of the registry testing and will put up a PR, but I'm happy to wait till 1.5.

timothysc · 2016-08-30T15:37:01Z

fix posted.

Automatic merge from submit-queue Etcd3 v3client + grpc client leak fix This re-enables the client and plumbs through the destroyFunc to cleanup clients. Fix for #31262

Automatic merge from submit-queue return destroy func to clean up internal resources of storage What? Provide a destroy func to clean up internal resources of storage. It changes **unit tests** to clean up resources. (Maybe fix integration test in another PR.) Why? Although apiserver is designed to be long running, there are some cases that it's not. See kubernetes/kubernetes#31262 (comment) We need to gracefully shutdown and clean up resources.

k8s-github-robot assigned spxtr Aug 23, 2016

k8s-github-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. kind/flake Categorizes issue or PR as related to a flaky test. labels Aug 23, 2016

sttts mentioned this issue Aug 23, 2016

Sort hack/.linted packages #31104

Merged

sttts added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Aug 23, 2016

sttts mentioned this issue Aug 23, 2016

Add sysctl support #27180

Merged

14 tasks

spxtr assigned dchen1107 and unassigned spxtr Aug 23, 2016

Random-Liu mentioned this issue Aug 23, 2016

Increase request timeout based on termination grace period #31275

Merged

yifan-gu mentioned this issue Aug 23, 2016

rkt: Improve support for privileged pod (pod whose all containers are privileged) #31286

Merged

erictune mentioned this issue Aug 23, 2016

Fix various typos in pkg/apis #31225

Merged

caesarxuchao mentioned this issue Aug 24, 2016

[GarbageCollector] add absent owner cache #31167

Merged

ping035627 mentioned this issue Aug 24, 2016

Implement groups come from registered versions #30980

Closed

wojtek-t assigned timothysc, hongchaodeng and xiang90 and unassigned dchen1107 Aug 24, 2016

wojtek-t mentioned this issue Aug 24, 2016

Avoid unnecessary copies & allocations in field selectors #31248

Merged

lavalamp mentioned this issue Aug 24, 2016

Revert "Enable v3 Client as the default on UTs" #31371

Merged

hongchaodeng mentioned this issue Aug 24, 2016

TestWatch {etcd3} #31083

Closed

mtaufen mentioned this issue Aug 24, 2016

Refactor to simplify the hard-traveled path of the KubeletConfiguration object #29216

Merged

hongchaodeng mentioned this issue Aug 24, 2016

return destroy func to clean up internal resources of storage #31390

Merged

pwittrock added the flake-needs-czar-attention-release-1.4 label Aug 25, 2016

yujuhong mentioned this issue Aug 25, 2016

[flake] k8s.io/kubernetes/pkg/registry/petset/etcd TestWatch #31389

Closed

k8s-github-robot pushed a commit that referenced this issue Aug 25, 2016

Merge pull request #31371 from kubernetes/revert-30890-etcd3_v3client

b70a758

Automatic merge from submit-queue Revert "Enable v3 Client as the default on UTs" Reverts #30890 Fix flake problems (#31262)

pwittrock mentioned this issue Aug 25, 2016

Document selinux considerations in wordpress-mysql example #31357

Merged

lavalamp closed this as completed Aug 25, 2016

timothysc mentioned this issue Aug 30, 2016

Etcd3 v3client + grpc client leak fix #31704

Merged

This was referenced Sep 15, 2016

TestWatch {etcd} #32770

Closed

TestWatch {etcd} #33022

Closed

liggitt mentioned this issue Oct 5, 2016

TestWatch panic: test timed out after 5m0s #34072

Closed

k8s-github-robot mentioned this issue Oct 10, 2016

TestWatch {etcd} #34485

Closed

k8s-github-robot mentioned this issue Dec 24, 2016

TestWatch {etcd} #39221

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TestWatch {etcd} #31262

TestWatch {etcd} #31262

k8s-github-robot commented Aug 23, 2016

sttts commented Aug 23, 2016

spxtr commented Aug 23, 2016

ncdc commented Aug 23, 2016

k8s-github-robot commented Aug 23, 2016

k8s-github-robot commented Aug 23, 2016

k8s-github-robot commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

wojtek-t commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

timothysc commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

lavalamp commented Aug 24, 2016

hongchaodeng commented Aug 24, 2016

hongchaodeng commented Aug 24, 2016

hongchaodeng commented Aug 24, 2016

k8s-github-robot commented Aug 25, 2016

k8s-github-robot commented Aug 25, 2016

k8s-github-robot commented Aug 25, 2016

lavalamp commented Aug 25, 2016

wojtek-t commented Aug 26, 2016

timothysc commented Aug 26, 2016

timothysc commented Aug 30, 2016

TestWatch {etcd} #31262

TestWatch {etcd} #31262

Comments

k8s-github-robot commented Aug 23, 2016

sttts commented Aug 23, 2016

spxtr commented Aug 23, 2016

ncdc commented Aug 23, 2016

k8s-github-robot commented Aug 23, 2016

k8s-github-robot commented Aug 23, 2016

k8s-github-robot commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

wojtek-t commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

timothysc commented Aug 24, 2016

k8s-github-robot commented Aug 24, 2016

lavalamp commented Aug 24, 2016

hongchaodeng commented Aug 24, 2016

hongchaodeng commented Aug 24, 2016

hongchaodeng commented Aug 24, 2016

k8s-github-robot commented Aug 25, 2016

k8s-github-robot commented Aug 25, 2016

k8s-github-robot commented Aug 25, 2016

lavalamp commented Aug 25, 2016

wojtek-t commented Aug 26, 2016

timothysc commented Aug 26, 2016

timothysc commented Aug 30, 2016