GC changes for synchronous GC supports #38679

caesarxuchao · 2016-12-13T08:37:20Z

Split from #38676.
Design doc: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/synchronous-garbage-collection.md.

This PR won't compile. It's for review only.

Comments that will be addressed in follow-up PRs:

check if an update event changes DeletionTimestamp, OwnerReferences, or Finalizers, if not, don't send it via the graphChanges channel.
In attempToDelete, try to use the cache instead of communicating with the API server. One exception: We need to talk to the API server to check if the object represented by a "virtual" node exists.

TODO:

split the unit tests garbagecollector_test.go according to the components tested.
try if it's practically safe to use the cached object status in attempToDeleteItem(), after synchronous GC feature is stable. (Also see Add synchronous garbage collection #38676 (comment))
add blockOwnerDeletion for rs adoption GC changes for synchronous GC supports #38679 (comment)

cc @kubernetes/sig-api-machinery

k8s-reviewable · 2016-12-13T08:37:26Z

This change is

0xmichalis · 2016-12-13T11:25:23Z

pkg/controller/garbagecollector/garbagecollector.go

+		go monitor.controller.Run(stopCh)
+	}
+
+	wait.PollInfinite(10*time.Second, func() (bool, error) {


Use WaitForCacheSync

0xmichalis · 2016-12-13T11:25:56Z

pkg/controller/garbagecollector/garbagecollector.go

+	Register()
+	<-stopCh
+	glog.Infof("Garbage Collector: Shutting down")
+	gc.dirtyQueue.ShutDown()


You should defer these

0xmichalis · 2016-12-13T11:26:32Z

pkg/controller/garbagecollector/garbagecollector.go

+// solid: the owner exists, and is not "waiting"
+// dangling: the owner does not exist
+// waiting: the owner exists, its deletionTimestamp is non-nil, and it has
+// FianlizerDeletingDependents


0xmichalis · 2016-12-13T11:27:28Z

pkg/controller/garbagecollector/garbagecollector.go

+// FianlizerDeletingDependents
+// This function communicates with the server.
+func (gc *GarbageCollector) classifyReferences(item *node, latestReferences []metav1.OwnerReference) (
+	solid []metav1.OwnerReference,


Maybe solid, dangling, waiting []metav1.OwnerReference, ?

0xmichalis · 2016-12-13T11:30:42Z

pkg/controller/garbagecollector/garbagecollector.go

-				continue
+				glog.V(6).Infof("object %s's owner %s/%s, %s is not found", item.identity.UID, reference.APIVersion, reference.Kind, reference.Name)
+				dangling = append(dangling, reference)
+			} else {


Move this check on top

if !errors.IsNotFound(err) { return solid, dangling, waiting, err } gc.absentOwnerCache.Add(reference.UID) ...

0xmichalis · 2016-12-13T12:17:48Z

pkg/controller/garbagecollector/garbagecollector.go

+// orphanFinalizer dequeues a node from the orphanQueue, then finds its dependents
+// based on the graph maintained by the GC, then removes it from the
+// OwnerReferences of its dependents, and finally updates the owner to remove
+// the "Orphan" finalizer. The node is add back into the orphanQueue if any of


s/add/added/

caesarxuchao

Thanks, @Kargakis. Addressed.

caesarxuchao · 2016-12-13T18:19:17Z

pkg/controller/garbagecollector/garbagecollector.go

+		go monitor.controller.Run(stopCh)
+	}
+
+	wait.PollInfinite(10*time.Second, func() (bool, error) {


caesarxuchao · 2016-12-13T18:24:04Z

pkg/controller/garbagecollector/garbagecollector.go

+	Register()
+	<-stopCh
+	glog.Infof("Garbage Collector: Shutting down")
+	gc.dirtyQueue.ShutDown()


caesarxuchao · 2016-12-13T18:25:01Z

pkg/controller/garbagecollector/garbagecollector.go

+// FianlizerDeletingDependents
+// This function communicates with the server.
+func (gc *GarbageCollector) classifyReferences(item *node, latestReferences []metav1.OwnerReference) (
+	solid []metav1.OwnerReference,


caesarxuchao · 2016-12-13T18:25:50Z

pkg/controller/garbagecollector/garbagecollector.go

-				continue
+				glog.V(6).Infof("object %s's owner %s/%s, %s is not found", item.identity.UID, reference.APIVersion, reference.Kind, reference.Name)
+				dangling = append(dangling, reference)
+			} else {


caesarxuchao · 2016-12-13T18:26:12Z

pkg/controller/garbagecollector/garbagecollector.go

+// orphanFinalizer dequeues a node from the orphanQueue, then finds its dependents
+// based on the graph maintained by the GC, then removes it from the
+// OwnerReferences of its dependents, and finally updates the owner to remove
+// the "Orphan" finalizer. The node is add back into the orphanQueue if any of


deads2k · 2016-12-13T18:38:54Z

pkg/controller/garbagecollector/garbage_collector.go

+
+const ResourceResyncTime time.Duration = 0
+
+type monitor struct {


Reading from top to bottom, this needs doc.

The struct is removed. I added a comment to GarbageCollector.monitors.

deads2k · 2016-12-13T18:39:28Z

pkg/controller/garbagecollector/garbage_collector.go

+const ResourceResyncTime time.Duration = 0
+
+type monitor struct {
+	store      cache.Store


This is a very loose type. Any chance of using GenericLister instead? It looks like its filled via a reflector.

The struct is removed.

deads2k · 2016-12-13T18:41:12Z

pkg/controller/garbagecollector/garbage_collector.go

+	// clientPool uses the regular dynamicCodec. We need it to update
+	// finalizers. It can be removed if we support patching finalizers.
+	clientPool                       dynamic.ClientPool
+	dirtyQueue                       *workqueue.TimedWorkQueue


Seems like this could do with an interface.

Actually, is this type nothing but casting on the normal WorkQueue?

It adds the timing metrics to the workqueue. I think the normal workqueue supports timing metrics now, though missing some functionalities. I can try to migrate to use the normal workqueue in another PR.

It adds the timing metrics to the workqueue. I think the normal workqueue supports timing metrics now, though missing some functionalities. I can try to migrate to use the normal workqueue in another PR.

It would enhance my reading pleasure and allow you to wire up to a named rate limited queue. What's the delta on metrics being gathered?

Refactored to use the rate limited queue.

deads2k · 2016-12-13T18:42:11Z

pkg/controller/garbagecollector/garbage_collector.go

+	// clientPool uses the regular dynamicCodec. We need it to update
+	// finalizers. It can be removed if we support patching finalizers.
+	clientPool                       dynamic.ClientPool
+	dirtyQueue                       *workqueue.TimedWorkQueue


Having two queues is uncommon. Doc please.

deads2k · 2016-12-13T18:43:57Z

pkg/controller/garbagecollector/garbage_collector.go

+	dirtyQueue                       *workqueue.TimedWorkQueue
+	orphanQueue                      *workqueue.TimedWorkQueue
+	monitors                         []monitor
+	propagator                       *Propagator


This is a dependencyGraphBuilder, right?

dependencyGraphBuilder + workQueueBuilder

dependencyGraphBuilder + workQueueBuilder

After going through this on slack with you, I think I'd like to call this a dependencyGraphBuilder and give it an inbound and outbound queue (graphUpdates and attemptDeletes?)

graphUpdate will be a renaming of eventQueue.

Currently propagator has two outbounds queue, dirtyQueue and orphanQueue, I want to merge them into one, but the changes to the logic could be complex, so I want to do it in another PR.

I think attemptDeletes will just be a renaming of dirtyQueue.

Renamed the queues.

deads2k · 2016-12-13T18:44:20Z

pkg/controller/garbagecollector/garbage_collector.go

+	orphanQueue                      *workqueue.TimedWorkQueue
+	monitors                         []monitor
+	propagator                       *Propagator
+	clock                            clock.Clock


just for testing?

It's used to time how long an item spends in queues.

It's used to time how long an item spends in queues.

Yeah, that's a metric we gather.

deads2k · 2016-12-13T18:46:14Z

pkg/controller/garbagecollector/garbage_collector.go

+	monitors                         []monitor
+	propagator                       *Propagator
+	clock                            clock.Clock
+	registeredRateLimiter            *RegisteredRateLimiter


These are just for registering prometheus metrics or something?

Just for prometheus metrics.

Comments added.

deads2k · 2016-12-13T18:48:27Z

pkg/controller/garbagecollector/garbage_collector.go

+	registeredRateLimiter            *RegisteredRateLimiter
+	registeredRateLimiterForMonitors *RegisteredRateLimiter
+	// GC caches the owners that do not exist according to the API server.
+	absentOwnerCache *UIDCache


Seems like this would be discoveryable via a marker on nodes. The graph does it include these, right? Otherwise edge counting won't work how most people would expect.

The absentCache provides a slightly stronger guarantee than the graph does. If the object doesn't exist in the graph, it could be that the informer hasn't observed it yet. But if an object's UID is in the absentCache, the object couldn't exist.

deads2k · 2016-12-13T18:53:38Z

pkg/controller/garbagecollector/garbage_collector.go

+
+func (gc *GarbageCollector) monitorFor(resource schema.GroupVersionResource, kind schema.GroupVersionKind) (monitor, error) {
+	// TODO: consider store in one storage.
+	glog.V(6).Infof("create storage for resource %s", resource)


logging at this level gets caught up in RESTClient logging. I'd make it a five instead.

deads2k · 2016-12-13T18:56:06Z

pkg/controller/garbagecollector/garbage_collector.go

+		gc.dirtyQueue.Add(timedItem)
+		return
+	}
+	DirtyProcessingLatency.Observe(sinceInMicroseconds(gc.clock, timedItem.StartTime))


pretty sure this is now tracked by a named work queue automatically.

If I dequeue an object, process it, but an error occurs so I enqueue it once more, the normal work queue doesn't know it's the same object, right? I can try to expand the normal work queue to support such case.

If I dequeue an object, process it, but an error occurs so I enqueue it once more, the normal work queue doesn't know it's the same object, right?

If you add rate limited, it knows.

deads2k · 2016-12-13T18:56:44Z

pkg/controller/garbagecollector/garbage_collector.go

+	return monitor, nil
+}
+
+var ignoredResources = map[schema.GroupVersionResource]struct{}{


I know this is pre-existing, but make a note to pull this information from the discovery doc.

err, make an issue.

It's in the umbrella issue #26120.

Need a discovery mechanism to determine what resources GC should manage. For example, GC needs to determine if it should watch for extensions/v1beta1/job or batch/v1/job.

deads2k · 2016-12-13T18:57:55Z

pkg/controller/garbagecollector/garbage_collector.go

+				}
+				gc.propagator.eventQueue.Add(&workqueue.TimedWorkQueueItem{StartTime: gc.clock.Now(), Object: event})
+			},
+			UpdateFunc: func(oldObj, newObj interface{}) {


If the old object meta and new object meta have the same owner references, finalizers, and deletiontimestamp, there's no reason to issue the update to the graph builder, right?

Right. Currently the check is done in the propagator.

Right. Currently the check is done in the propagator.

This is the spot to do it. Here you have one thread per watch and you can run the checks concurrently. In the dependencyGraphBuilder, you've paid an enqueue, a dequeue, and you're single threaded on the consumption side.

There were some weird cases where the graph didn't already have the node even if it's an update event, thus this line. IMO this should never happen, maybe it's caused by an informer problem. So it's not safe to ignore some of the update events here.
Let's not introduce a possible source of flakes when adding new feature. I'll leave a TODO and try the suggested optimization in a follow up PR.

TODO added.

deads2k · 2017-01-03T16:31:09Z

pkg/controller/garbagecollector/graph.go

+}
+
+type concurrentUIDToNode struct {
+	*sync.RWMutex


These aren't usually pointers. Also, name it please.

deads2k · 2017-01-20T15:53:40Z

pkg/controller/garbagecollector/graph_builder.go

+	go wait.Until(gb.runProcessGraphChanges, 1*time.Second, stopCh)
+}
+
+var ignoredResources = map[schema.GroupVersionResource]struct{}{


TODO, many of these should be discoverable (cannot be deleted)

"TODO: GC should detect if a resource requires its attention via API discovery."

deads2k · 2017-01-20T16:02:42Z

pkg/controller/garbagecollector/graph_builder.go

+	}
+	for _, c := range changed {
+		if c.oldRef.BlockOwnerDeletion != nil && *c.oldRef.BlockOwnerDeletion &&
+			c.newRef.BlockOwnerDeletion != nil && !*c.newRef.BlockOwnerDeletion {


c.newRef.BlockOwnerDeletion == nil isn't treated as equivalent to false?

Nice catch. Updated.

deads2k · 2017-01-20T16:03:25Z

pkg/controller/garbagecollector/graph_builder.go

+func (gb *GraphBuilder) processTransitions(oldObj interface{}, newAccessor meta.Object, n *node) {
+	if startsWaitingForDependentsOrphaned(oldObj, newAccessor) {
+		glog.V(5).Infof("add %s to the attemptToOrphan", n.identity)
+		gb.attemptToOrphan.Add(n)


return at the end of this block to eliminate the else.

if return } if { }

k8s-github-robot · 2017-01-24T02:06:21Z

[APPROVALNOTIFIER] Needs approval from an approver in each of these OWNERS Files:

pkg/controller/OWNERS
test/OWNERS

We suggest the following people:
cc @derekwaynecarr @fejta
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

caesarxuchao · 2017-01-31T09:03:46Z

@deads2k @Kargakis comments addressed. PTAL. Thanks.

k8s-ci-robot · 2017-01-31T09:13:15Z

@caesarxuchao: The following test(s) failed:

Test name	Commit	Details	Rerun command
Jenkins GKE smoke e2e	`0fcc48f`	link	`@k8s-bot cvm gke e2e test this`
Jenkins GCI GKE smoke e2e	`0fcc48f`	link	`@k8s-bot gci gke e2e test this`
Jenkins GCE e2e	`0fcc48f`	link	`@k8s-bot cvm gce e2e test this`
Jenkins verification	`0fcc48f`	link	`@k8s-bot verify test this`
Jenkins GCE Node e2e	`0fcc48f`	link	`@k8s-bot node e2e test this`
Jenkins CRI GCE Node e2e	`0fcc48f`	link	`@k8s-bot cri node e2e test this`
Jenkins kops AWS e2e	`0fcc48f`	link	`@k8s-bot kops aws e2e test this`
Jenkins GCI GCE e2e	`0fcc48f`	link	`@k8s-bot gci gce e2e test this`
Jenkins unit/integration	`0fcc48f`	link	`@k8s-bot unit test this`
Jenkins Kubemark GCE e2e	`0fcc48f`	link	`@k8s-bot kubemark e2e test this`
Jenkins GCE etcd3 e2e	`0fcc48f`	link	`@k8s-bot gce etcd3 e2e test this`
Jenkins Bazel Build	`0fcc48f`	link	`@k8s-bot bazel test this`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

deads2k · 2017-02-03T20:20:24Z

@lavalamp there's a lot here. You should probably take a pass.

caesarxuchao · 2017-02-17T02:24:36Z

@deads2k do you have more comments? Or shall i just ask lavalamp to take another pass?

deads2k · 2017-02-17T13:22:53Z

@deads2k do you have more comments? Or shall i just ask lavalamp to take another pass?

I think it's @lavalamp's turn. I'd like another pass through when he's done, but I'm pretty happy with the shape.

caesarxuchao · 2017-02-22T06:48:12Z

pkg/controller/garbagecollector/garbage_collector.go

+	return ret
+}
+
+func (gc *GarbageCollector) attemptToDeleteItem(item *node) error {


@lavalamp this function is the main processing logic. The input the is object added to the queue by the watcher. This function takes different actions for the object that "needs to delete dependents in foreground/background", "needs to orphan dependents", "still has valid owner", or "is waiting for dependents to be deleted".

caesarxuchao · 2017-02-22T06:55:39Z

pkg/controller/garbagecollector/graph_builder.go

+}
+
+// Dequeueing an event from graphChanges, updating graph, populating dirty_queue.
+func (gb *GraphBuilder) processGraphChanges() bool {


@lavalamp this function is adapted from processEvent, which builds the dependency graph, adds object to attemptToOrphan or attemptToDelete queue. This function updated to handle objects that need foreground GC.

@lavalamp

Automatic merge from submit-queue (batch tested with PRs 38676, 41765, 42103, 41833, 41702) Add synchronous garbage collection Fix #29891. Split into five commits: 1. generated: don't need review 2. API: got reviewed in #38678, i addressed @lavalamp's comments there. 3. registry changes: @nikhiljindal could you help take a look? 4. gc changes: reviewed by @deads2k in #38679. It needs another pass. 5. tests: @lavalamp @deads2k could take a look? TODO: - [ ] Update doc. Note that the existing doc has been refactored in kubernetes/website#2488. - [ ] add an admission controller to check if a user can set OwnerReference.BlockOwnerDeletion - [ ] #38676 (comment) - [ ] split the unit tests garbagecollector_test.go according to the components tested. - [ ] try if it's practically safe to use the cached object status in attempToDeleteItem(), after synchronous GC feature is stable. (Also see #38676 (comment)) - [ ] add blockOwnerDeletion for rs adoption #38679 (comment) - [ ] https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/38676/pull-kubernetes-e2e-gce-etcd3/20101/ (improve the log message) ```release-note Added foreground garbage collection: the owner object will not be deleted until all its dependents are deleted by the garbage collector. Please checkout the [user doc](https://kubernetes.io/docs/concepts/abstractions/controllers/garbage-collection/) for details. deleteOptions.orphanDependents is going to be deprecated in 1.7. Please use deleteOptions.propagationPolicy instead. ```

anguslees · 2017-06-05T03:56:14Z

test/e2e/framework/metrics_util.go

-	"garbage_collector_orphan_processing_latency_microseconds",
+	"garbage_collector_event_queue_latency",
+	"garbage_collector_dirty_queue_latency",
+	"garbage_collector_orhan_queue_latency",


Typo: "orphan"

It's fixed in HEAD.

@lavalamp

Automatic merge from submit-queue (batch tested with PRs 38676, 41765, 42103, 41833, 41702) Add synchronous garbage collection Fix kubernetes/kubernetes#29891. Split into five commits: 1. generated: don't need review 2. API: got reviewed in #38678, i addressed @lavalamp's comments there. 3. registry changes: @nikhiljindal could you help take a look? 4. gc changes: reviewed by @deads2k in #38679. It needs another pass. 5. tests: @lavalamp @deads2k could take a look? TODO: - [ ] Update doc. Note that the existing doc has been refactored in kubernetes/website#2488. - [ ] add an admission controller to check if a user can set OwnerReference.BlockOwnerDeletion - [ ] kubernetes/kubernetes#38676 (comment) - [ ] split the unit tests garbagecollector_test.go according to the components tested. - [ ] try if it's practically safe to use the cached object status in attempToDeleteItem(), after synchronous GC feature is stable. (Also see kubernetes/kubernetes#38676 (comment)) - [ ] add blockOwnerDeletion for rs adoption kubernetes/kubernetes#38679 (comment) - [ ] https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/38676/pull-kubernetes-e2e-gce-etcd3/20101/ (improve the log message) ```release-note Added foreground garbage collection: the owner object will not be deleted until all its dependents are deleted by the garbage collector. Please checkout the [user doc](https://kubernetes.io/docs/concepts/abstractions/controllers/garbage-collection/) for details. deleteOptions.orphanDependents is going to be deprecated in 1.7. Please use deleteOptions.propagationPolicy instead. ``` Kubernetes-commit: 3afefae02a0588654693de27ef5387c1f9ab5f01

Chao Xu added 2 commits December 13, 2016 00:33

gc controller changes and a few rc/rs controller changes

c46cb68

e2e and integration tests

130377d

caesarxuchao added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Dec 13, 2016

caesarxuchao assigned deads2k Dec 13, 2016

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 13, 2016

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note-label-needed labels Dec 13, 2016

0xmichalis reviewed Dec 13, 2016

View reviewed changes

caesarxuchao commented Dec 13, 2016

View reviewed changes

Chao Xu added 2 commits December 13, 2016 10:30

to make review easier, move garbagecollector.go to garbage_collector.go

0d7a855

add comments

d6d693f

caesarxuchao force-pushed the sync-gc-controller branch from ef19668 to d6d693f Compare December 13, 2016 18:37

deads2k reviewed Dec 13, 2016

View reviewed changes

deads2k reviewed Jan 3, 2017

View reviewed changes

deads2k reviewed Jan 20, 2017

View reviewed changes

deads2k mentioned this pull request Jan 26, 2017

Make HandleError prevent hot-loops #40497

Merged

caesarxuchao mentioned this pull request Jan 30, 2017

[GC] controller manager gets error "unable to get REST mapping for kind" for ownerRefs to TPR and add-on APIs #39816

Closed

k8s-github-robot assigned madhusudancs and derekwaynecarr and unassigned deads2k and madhusudancs Jan 30, 2017

apelisse assigned deads2k and madhusudancs Jan 31, 2017

k8s-github-robot assigned krousey and mikedanese Jan 31, 2017

madhusudancs removed their assignment Jan 31, 2017

addressing comments

0fcc48f

deads2k assigned lavalamp Feb 3, 2017

caesarxuchao commented Feb 22, 2017

View reviewed changes

caesarxuchao mentioned this pull request Feb 23, 2017

Add synchronous garbage collection #38676

Merged

7 tasks

caesarxuchao closed this Feb 27, 2017

anguslees reviewed Jun 5, 2017

View reviewed changes


		const ResourceResyncTime time.Duration = 0

		type monitor struct {

GC changes for synchronous GC supports #38679

GC changes for synchronous GC supports #38679

Conversation

caesarxuchao commented Dec 13, 2016 • edited

k8s-reviewable commented Dec 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao Dec 14, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao Dec 13, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k Jan 20, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-github-robot commented Jan 24, 2017

caesarxuchao commented Jan 31, 2017

k8s-ci-robot commented Jan 31, 2017

deads2k commented Feb 3, 2017

caesarxuchao commented Feb 17, 2017

deads2k commented Feb 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao commented Dec 13, 2016 •

edited

caesarxuchao Dec 14, 2016 •

edited

caesarxuchao Dec 13, 2016 •

edited

deads2k Jan 20, 2017 •

edited