KEP of pod affinity/anti-affinity supports Gt and Lt operators #890

wgliang · 2019-03-12T01:56:37Z

As a follow-up of discussion kubernetes/kubernetes#72556 (comment)

/sig scheduling
/kind design
/priority important-longterm
cc/ @bsalamat @k82cn @ravisantoshgudimetla @misterikkit @resouer @Huang-Wei @liggitt

Huang-Wei

Please see my comments below.

Huang-Wei

Please see my comments below.

Huang-Wei · 2019-03-12T23:09:59Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+
+- if we use the original `Exists` operator to implement, then we will write affinity 
+like this:
+ ```go


change to "yaml" to make it indent well

Huang-Wei · 2019-03-12T23:10:13Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+```
+- if we use the original `In` operator to implement, then we will write affinity 
+like this:
+ ```go


change to "yaml" to make it indent well

Huang-Wei · 2019-03-12T23:10:18Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+ Both are not ideal solutions. A promising solution is to provide users with `Gt` and `Lt` 
+ operators, giving users the ability to specify the scope of the tag. Users can achieve 
+ this in this way:
+ ```go


change to "yaml" to make it indent well

Huang-Wei · 2019-03-12T23:10:54Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+      - labelSelector:
+          matchExpressions:
+          - key: foo
+            operator: Exists


Exists behaves like "whether a key exists". It has nothing to do with values.

So I think this sample can be deleted, just keep the latter "In" sample.

Sounds good.

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

wgliang · 2019-03-13T01:59:49Z

@Huang-Wei Thank you for your review, I have updated based on your comment.

ravisantoshgudimetla · 2019-03-13T05:52:47Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+ ### Non-Goals
+
+- Just add `Gt` and `Lt` operator functions without changing their original definition.
+- Not changing the behavior of other label selectors, such as `Replicases`, `Daemonsets`, etc.


typo: Replicases

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

ravisantoshgudimetla · 2019-03-13T06:18:25Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+ API of `PodSelector` is defined as below:
+
+ ```go
+type PodSelector struct {


Can we leverage most of node affinity structures here instead of using new ones? I know that the existing ones are hardcoded with prefix Node but I believe they're generic enough for example NodeSelector, NodeSelectorRequirements, NodeSelectorOperators have the same fields and types that we could use for pod affinity too.

Yes, @Huang-Wei also proposed before. The PodSelector was re-proposed to avoid confusion with NodeSelector. Because from the existing code, many checks and usages of LabelSelector and NodeSelector are inconsistent. I am worried about destroying its original usage habits.

But I think this is a reasonable choice, we can reuse the logic of NodeSelector from a new abstraction.

/cc @bsalamat needs some of your opinion.

I agree that the APIs are the same and act in the same way, but given that those have a "Node" prefix, we cannot use them for Pod selection. Otherwise, our API would be very confusing. I like the already proposed solution better that we can create a common LabelSelectorRequirement that is used in both Node and Pod selectors.

It is a backwards compatible change to change the name of the go structs, as those don't actually appear in user config (except for the top level objects like Pod). The field names cannot be changed.

If we change the name of go structs, the operators who is using it have to update the code accordingly, right?

ravisantoshgudimetla · 2019-03-13T06:24:00Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+
+ Along with this feature, the biggest risk should be performance. This may also be the reason 
+ why the `Gt` and `Lt` operators are not supported when Pod affinity/anti-affinity is 
+ first proposed. In order to eliminate the user's panic, carefully measure the impact of these operators and understand their impact is mandatory. So we will perform performance/benchmark tests on this change and backward compatibility test.


nit: s/In order to eliminate the user's panic, carefully measure the impact of these operators and understand their impact is mandatory/In order to understand the impact of these changes, we need to understand the their performance implications.

done, thanks.

bsalamat · 2019-03-13T23:32:14Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+
+ ### Non-Goals
+
+- Just add `Gt` and `Lt` operator functions without changing their original definition.


Is this really needed as a non-goal?

Yes, IMO, the definition of the operator is the same for Pod or Node, or any other entity object.
Do you think there is anything added?

I am also confused about why this needs to be stated. Who would expect us to change the definition of less than / greater than?

bsalamat · 2019-03-13T23:35:19Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+
+ Along with this feature, the biggest risk should be performance. This may also be the reason 
+ why the `Gt` and `Lt` operators are not supported when Pod affinity/anti-affinity is 
+ first proposed. In order to understand the impact of these changes, we need to understand the their performance implications.So we will perform performance/benchmark tests on this change and backward compatibility test.


s/.So we will/. We will/

bsalamat · 2019-03-13T23:40:03Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+ why the `Gt` and `Lt` operators are not supported when Pod affinity/anti-affinity is 
+ first proposed. In order to understand the impact of these changes, we need to understand the their performance implications.So we will perform performance/benchmark tests on this change and backward compatibility test.
+
+ As part of the mitigations, the good news is that the performance of the entire scheduler has been greatly improved (https://github.com/kubernetes/kubernetes/pull/74041#issuecomment-466191359), and the processing of Pod affinity/anti-affinity has been surprisingly Optimized(https://github.com/kubernetes/kubernetes/pull/67788).


Instead of "As part the mitigation..." you may want to say "Compared to the time that inter-pod affinity was introduced, performance of the scheduler has been greatly improved..."

Yeah, thanks.

bsalamat · 2019-03-13T23:43:13Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+ of `metav1.LabelSelector`:
+
+ ```go
+type PodAffinityTerm struct {


This needs an API review by our API experts.
I would personally prefer a generic approach that would work for other selectors as well, as opposed to a selector that works differently for a specific feature.

@kubernetes/api-reviewers @liggitt

SGTM, will update the design document.

bsalamat · 2019-03-13T23:44:10Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+
+ ### Test Plan
+
+ _To be filled until targeted at a release._


Please make sure to cover both correctness and performance tests in the plan.

wgliang · 2019-03-21T08:17:58Z

@ravisantoshgudimetla @bsalamat Thanks for your review, I have updated the content mentioned in the comments.
What needs special attention is to change the design and implementation of the API.

bsalamat

Thanks, @wgliang! Looks good to me. Given that this KEP proposes changes to existing API, we need an API review as well.

bsalamat · 2019-04-04T23:25:22Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+ API of `PodSelector` is defined as below:
+
+ ```go
+type PodSelector struct {


I agree that the APIs are the same and act in the same way, but given that those have a "Node" prefix, we cannot use them for Pod selection. Otherwise, our API would be very confusing. I like the already proposed solution better that we can create a common LabelSelectorRequirement that is used in both Node and Pod selectors.

bsalamat · 2019-04-04T23:28:02Z

@kubernetes/api-reviewers

wgliang · 2019-04-17T03:21:26Z

/ping @kubernetes/api-reviewers

lavalamp · 2019-04-17T15:09:52Z

Added to the api review board https://github.com/orgs/kubernetes/projects/13

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

lavalamp · 2019-04-17T15:14:51Z

/assign

lavalamp · 2019-04-17T15:34:00Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+  LabelSelectorOpGt           LabelSelectorOperator = "Gt"
+  LabelSelectorOpLt           LabelSelectorOperator = "Lt"
+)
+```


Last thought: whatever design you come up with has to maintain backwards compatibility. Existing pods must continue to work the same without any changes.

Yes, this has also clarified in the past that the definition of Pod will not be changed.

I don't understand that response, this proposal definitely looks like it is modifying types which are part of a Pod?

Backward compatibility is a requirement for this change. The intention of this KEP is that existing pod specs should work without needing any changes.

lavalamp · 2019-04-17T15:48:05Z

P.S. for better service from api reviewers next time, please see: https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md#mechanics

liggitt · 2019-04-19T13:09:03Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+- `NodeSelector` related tests
+- `PodSelector` related tests
+- `Gt` and `Lt` functional tests
+- `LabelSelectorRequirement` related tests


we also need tests ensuring that the new Gt/Lt operators are not permitted in "normal" selector fields (e.g. pod.spec.nodeSelector, etc)

Yes, this is for sure, pod.spec.nodeSelector(https://github.com/kubernetes/kubernetes/blob/22c67bb113dee2ce0858075cd5af5d110c255985/pkg/apis/core/types.go#L2599) is a standalone map structure.

Backwards compatibility - pods made with the new types should still be updatable if apiserver version is rolled back.
Forwards compatibility - all pods created today are wire-compatible (both proto and json) with the new api.

I agree with your comment. But what needs to be clarified is that because we have added Gt and Lt operators, this does not exist in the previous API.

wgliang · 2019-04-22T05:22:54Z

@liggitt @lavalamp @bsalamat @Huang-Wei
Thanks everyone for your comments, I have updated and replied to the comments. Any other questions, please comment here. Thanks.

As far as I know, the enhancement freeze is on April 30th. I hope to catch this train. :)

lavalamp · 2019-04-22T16:33:26Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+}
+```
+
+`LabelSelectorRequirement` is an abstract selector implementation of `NodeSelector` and `PodSelector`. And the `LabelSelectorRequirement` is defined as below:


I am worried about this name, I think it needs to be more specific, since it is only appropriate in these two places, and not for use in general label selectors everywhere?

It makes sense, I will change to NodeAndPodSelectorRequirement.

lavalamp · 2019-04-22T16:35:58Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+}
+```
+
+We will use `LabelSelectorRequirement` to replace the original `NodeSelectorRequirement` as the base selector of `NodeSelector`. So it looks like this:


What is the purpose of this change? Is anything wrong with the existing NodeSelectorRequirement?

Yes. Because we want to use a new selector(without a Node prefix) as the base selector for NodeSelector and PodSelector.

But you can do that without modifying the existing NodeSelector?

lavalamp · 2019-04-22T16:38:11Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+- `NodeSelector` related tests
+- `PodSelector` related tests
+- `Gt` and `Lt` functional tests
+- `LabelSelectorRequirement` related tests


Backwards compatibility - pods made with the new types should still be updatable if apiserver version is rolled back.
Forwards compatibility - all pods created today are wire-compatible (both proto and json) with the new api.

lavalamp · 2019-04-22T16:42:36Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+  LabelSelectorOpNotIn        LabelSelectorOperator = "NotIn"
+  LabelSelectorOpExists       LabelSelectorOperator = "Exists"
+  LabelSelectorOpDoesNotExist LabelSelectorOperator = "DoesNotExist"
+  LabelSelectorOpGt           LabelSelectorOperator = "Gt"


I think this name needs to indicate that the string-typed value will first be interpreted as a number; one could read this as meaning lexically greater--actually that's the natural meaning since label values are strings. Also, typically we request that things be spelled out in the API, although we should probably be consistent. :/ If this were starting from scratch I'd suggest the name "NumericallyGreater". You can still change the name of the constant to e.g. "LabelSelectorNumericallyGreater".

DONE, Thank you very much for your help and corrections.

lavalamp · 2019-04-23T16:56:24Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+We will use `NodeAndPodSelectorRequirement` to replace the original `NodeSelectorRequirement` as the base selector of `NodeSelector`. So it looks like this:
+ ```go
+type NodeSelectorTerm struct {
+  MatchExpressions []NodeAndPodSelectorRequirement


Name it by what it does, not where it's used. SchedulerSupportedSelectorRequirement? NumericAwareSelectorRequirement? I have to run, maybe @liggitt can think of a better name.

NodeAndPod won't work for this, the name can't be about the things being selected, it needs to be about the selection process. Otherwise people will probably use the type wrongly.

Sorry, please forgive my doubts. I don't particularly understand why mentioned numeric. Gt or Lt is not only able to compare numbers but also to compare letters, in other words it compares strings (including numbers).

I think SchedulerSupportedSelectorRequirement may be a possible choice, but inspired by you, I think SchedulerSelectorRequirement is more concise and appropriate. What do you think of it?

Gt or Lt is not only able to compare numbers but also to compare letters, in other words it compares strings (including numbers).

that is inconsistent with how these are used today:

https://github.com/kubernetes/kubernetes/blob/8ec6167f61463cbc6874514b02d0137d51b23542/staging/src/k8s.io/apimachinery/pkg/labels/selector.go#L151-L159

Are there compelling use cases for supporting lexical comparisons? It certainly makes the behavior less intuitive... e.g. what would the sort order of the following be?

1 1a 2 2a 10 20

This conversation is exactly why it needs to be prefixed with either "Numeric" or "Lexical" (or if it's some hybrid, something explaining that). We shouldn't make users work hard to figure this out.

Lexical comparison implies that "10" < "2", I do not think users will want that behavior.

Numeric comparison implies that error handling is needed if either of the label values are not numbers.

+1 to numeric comparison + error checking. I don't think there are many users who need lexical comparison in the API.

Ok. I mean, in fact, we use NumericXXX is not suitable, because there are still operators such as "In", "Equals" and so on.

Therefore, I agree that Gt and Lt only work on numbers. Summarize the above discussion:

Gt and Lt only work on numbers

NodeAndPodSelectorRequirement changed to SchedulerSelectorRequirement

What do you think of this?
@lavalamp @liggitt @bsalamat

In, Equals, etc all would behave the same whether the type is string or numeric. It's only ordered comparisons where the distinction becomes important.

"SchedulerSelectorRequirement" is not my favorite name, it's named for the component you expect to consume the thing. It is better than "NodeAndPod", but still: other components might eventually interpret this, the name isn't future-proof. It's also not at all obvious that "scheduler" here means "supports ordered comparisons on numeric values". That's really what you want the name to imply to people.

Understood.

Naming is really the hardest thing, I can't think of a better name than SchedulerSupportedNumericSelectorRequirement.
Do you have any other advice? @lavalamp @bsalamat

Some ideas:

NumericAwareSelectorRequirement

NumericallyOrderedSelectorRequirement

AffinitySelectorRequirement (only if this is going to be used for all and only affinity selection features)

Agree to use NumericAwareSelectorRequirement, I have updated. Thanks a lot for comments and your review.

lavalamp · 2019-04-23T16:58:51Z

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md

+}
+```
+
+`PodSelector` is our new selector. A pod selector represents the union of the results of one or more label queries over a set of pods. The of `PodSelector` is defined as below:


This is new, and you're not changing any existing selectors to use this type?

If so, I suggest calling it "NumericAwareSelector" or something else. That will make it more clear when it is appropriate to reuse it. It will also let you correctly make the documentation explain the selector--that it is applied to pods is really a consequence of where it is used, not something intrinsic to the type.

wgliang · 2019-04-30T02:22:46Z

@lavalamp @liggitt @bsalamat I don't know if this KEP can be merged before the end period. If it can, then it would be better.

lavalamp · 2019-04-30T03:36:58Z

LGTM from api review perspective now

k82cn · 2019-04-30T07:20:03Z

/lgtm
/approve

Mark the status according to the above discussion.

But I have a question that: if we change the name of go struct, the operators/customized-controllers has to change the code accordingly; do we need to keep such kind of backward compatibility? Golang support constructing nested struct without name, but it dependent on the coding style :).

k8s-ci-robot · 2019-04-30T07:20:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: k82cn, wgliang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-scheduling/OWNERS~~ [k82cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bsalamat · 2019-04-30T17:48:21Z

Given that the goal of this KEP is to add new operators to an existing feature, we must ensure that our implementation is fully backward compatible. So, we should avoid changing names or refactoring existing API structs if backward compatibility is broken.

lavalamp · 2019-05-01T01:55:05Z

But I have a question that: if we change the name of go struct, the operators/customized-controllers has to change the code accordingly; do we need to keep such kind of backward compatibility? Golang support constructing nested struct without name, but it dependent on the coding style :).

The wire format is the thing we make guarantees about. People are used to us breaking client-go compatibility, unfortunately. As I tried to make clear in earlier comments, I don't see a compelling reason to reuse and therefore rename the existing types. I think it's fine to have duplicate types, and somewhat prefer that. But this is only advice, if you all in the scheduling SIG think it's worth breaking client-go users for prettier types, I'm not going to block it, and I do at least agree that the new name is an improvement.

I hope a next step on the journey to fixing our deps that @liggitt is taking us on includes beginning to be a good golang citizen and keeping compatibility guarantees on the libraries we offer, at which point a change like this would not be permitted for that reason (or would have to be maintained differently in older client-go branches).

k82cn · 2019-05-01T23:49:26Z

if you all in the scheduling SIG think it's worth breaking client-go users for prettier types, I'm not going to block it

aha, "cyclic dependency" :)

we must ensure that our implementation is fully backward compatible

Agree with Bobby, we should keep backward compatible for this implementation. @wgliang , would you help to open a PR for that.

wgliang · 2019-05-02T09:56:07Z

@wgliang , would you help to open a PR for that.

Yeah, of course. And I will seriously consider the above comments.

k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. kind/design Categorizes issue or PR as related to design. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Mar 12, 2019

k8s-ci-robot requested review from bsalamat and k82cn March 12, 2019 01:56

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 12, 2019

Huang-Wei reviewed Mar 12, 2019

View reviewed changes

Huang-Wei requested changes Mar 12, 2019

View reviewed changes

wgliang force-pushed the master branch from 9992bf0 to 4ce75c6 Compare March 13, 2019 01:57

wgliang mentioned this pull request Mar 13, 2019

Pod affinity/anti-affinity support Gt and Lt operators kubernetes/kubernetes#72506

Closed

ravisantoshgudimetla reviewed Mar 13, 2019

View reviewed changes

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md Show resolved Hide resolved

ravisantoshgudimetla reviewed Mar 13, 2019

View reviewed changes

wgliang force-pushed the master branch from 4ce75c6 to b75b5d8 Compare March 13, 2019 06:54

bsalamat reviewed Mar 13, 2019

View reviewed changes

wgliang force-pushed the master branch 2 times, most recently from 0a62fce to defb6cb Compare March 21, 2019 08:15

bsalamat reviewed Apr 4, 2019

View reviewed changes

lavalamp added this to 1.15, Unassigned in API Reviews Apr 17, 2019

lavalamp reviewed Apr 17, 2019

View reviewed changes

keps/sig-scheduling/20190312-pod-affinity-and-anti-affinity-support-gt-and-lt-operators.md Show resolved Hide resolved

lavalamp moved this from 1.15, Unassigned to Assigned in API Reviews Apr 17, 2019

lavalamp reviewed Apr 17, 2019

View reviewed changes

lavalamp moved this from Assigned to Changes requested in API Reviews Apr 17, 2019

wgliang force-pushed the master branch 3 times, most recently from 579bf71 to 9e07fc3 Compare April 19, 2019 12:15

liggitt reviewed Apr 19, 2019

View reviewed changes

lavalamp reviewed Apr 22, 2019

View reviewed changes

wgliang force-pushed the master branch 2 times, most recently from 3dcf76b to 6e4cf7a Compare April 23, 2019 00:01

lavalamp reviewed Apr 23, 2019

View reviewed changes

wgliang force-pushed the master branch from 6e4cf7a to 45017d5 Compare April 28, 2019 11:50

KEP of pod affinity/anti-affinity supports Gt and Lt operators

b4eed78

wgliang force-pushed the master branch from 45017d5 to b4eed78 Compare April 30, 2019 02:16

k8s-ci-robot assigned k82cn Apr 30, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 30, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 30, 2019

k8s-ci-robot merged commit a65f0d1 into kubernetes:master Apr 30, 2019

liggitt moved this from Changes requested to Completed, 1.15 in API Reviews Apr 30, 2019

wgliang mentioned this pull request May 2, 2019

Add feature of pod affinity/anti-affinity supports Gt and Lt operators kubernetes/kubernetes#77341

Closed


		### Non-Goals

		- Just add `Gt` and `Lt` operator functions without changing their original definition.

KEP of pod affinity/anti-affinity supports Gt and Lt operators #890

KEP of pod affinity/anti-affinity supports Gt and Lt operators #890

Conversation

wgliang commented Mar 12, 2019

Huang-Wei left a comment • edited

Choose a reason for hiding this comment

Huang-Wei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgliang Mar 13, 2019 • edited

Choose a reason for hiding this comment

wgliang commented Mar 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgliang commented Mar 21, 2019

bsalamat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsalamat commented Apr 4, 2019

wgliang commented Apr 17, 2019

lavalamp commented Apr 17, 2019

lavalamp commented Apr 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lavalamp commented Apr 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgliang commented Apr 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgliang Apr 28, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wgliang commented Apr 30, 2019

lavalamp commented Apr 30, 2019

k82cn commented Apr 30, 2019

k8s-ci-robot commented Apr 30, 2019

bsalamat commented Apr 30, 2019

lavalamp commented May 1, 2019

k82cn commented May 1, 2019

wgliang commented May 2, 2019

Huang-Wei left a comment •

edited

wgliang Mar 13, 2019 •

edited

wgliang Apr 28, 2019 •

edited