Fix incorrectly set resource version in List #64150

wojtek-t · 2018-05-22T13:33:14Z

Fix incorrectly propagated ResourceVersion in ListRequests returning 0 items.

k8s-github-robot · 2018-05-22T13:35:47Z

[MILESTONENOTIFIER] Milestone Pull Request Labels Incomplete

@deads2k @liggitt @wojtek-t

Action required: This pull request requires label changes. If the required changes are not made within 3 days, the pull request will be moved out of the v1.11 milestone.

kind: Must specify exactly one of kind/bug, kind/cleanup or kind/feature.
priority: Must specify exactly one of priority/critical-urgent, priority/important-longterm or priority/important-soon.
sig owner: Must specify at least one label prefixed with sig/.

Help

liggitt · 2018-05-22T13:36:41Z

staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go

-	if err != nil {
-		return err
-	}
-	if len(getResp.Kvs) == 0 {


returning early with return s.versioner.UpdateList(listObj, uint64(getResp.Header.Revision), "") means a smaller diff and makes it clearer the exceptional case is handled

I don't agree with this comment - our goal shouldn't be having as small diff as possible.
I really think that with this change the code is much more understandable.

our goal shouldn't be having as small diff as possible

I assume we want to backport this... that was my thought about minimizing diff

Yes - I think we want to backport it.
but, it won't generate conflict on 1.10 and 1.9 and I think it's more important to have this code cleaner than to avoid conflict in 1.8. WDYT?

cleaner is in the eye of the beholder, but I don't feel that strongly

Yes - that's true.
But if you don't feel very strongly about this, I would prefer leaving as is.

liggitt · 2018-05-22T13:37:41Z

what did the v2 store do? would like to see a test for this against the etcd3, etcd2, and cacher store impls

wojtek-t · 2018-05-22T14:04:18Z

what did the v2 store do? would like to see a test for this against the etcd3, etcd2, and cacher store impls

I added tests.
In fact v2 store is also broken, but I don't really know how to fix it - for now added a TODO.

@liggitt - PTAL

liggitt · 2018-05-22T15:10:58Z

staging/src/k8s.io/apiserver/pkg/storage/etcd/etcd_helper.go

@@ -347,6 +347,7 @@ func (h *etcdHelper) GetToList(ctx context.Context, key string, resourceVersion
 	metrics.RecordEtcdRequestLatency("get", getTypeName(listPtr), startTime)
 	if err != nil {
 		if etcdutil.IsEtcdNotFound(err) {
+			// TODO: ResourceVersion should be set here.


in a not found case, the current index comes back with the error:

kubernetes/vendor/github.com/coreos/etcd/client/keys.go

Lines 57 to 62 in e85b81b

type Error struct {

Code int `json:"errorCode"`

Message string `json:"message"`

Cause string `json:"cause"`

Index uint64 `json:"index"`

}

$ curl http://localhost:2379/v2/keys/foo -v * Trying ::1... * TCP_NODELAY set * Connection failed * connect to ::1 port 2379 failed: Connection refused * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 2379 (#0) > GET /v2/keys/foo HTTP/1.1 > Host: localhost:2379 > User-Agent: curl/7.54.0 > Accept: */* > < HTTP/1.1 404 Not Found < Content-Type: application/json < X-Etcd-Cluster-Id: cdf818194e3a8c32 < X-Etcd-Index: 11 < Date: Tue, 22 May 2018 15:08:02 GMT < Content-Length: 70 < {"errorCode":100,"message":"Key not found","cause":"/foo","index":11}

note the index matches the X-Etcd-Index header, which is what is hoisted into response.Index in success cases:

kubernetes/vendor/github.com/coreos/etcd/client/keys.go

Lines 660 to 674 in e85b81b

func unmarshalSuccessfulKeysResponse(header http.Header, body []byte) (*Response, error) {

var res Response

err := codec.NewDecoderBytes(body, new(codec.JsonHandle)).Decode(&res)

if err != nil {

return nil, ErrInvalidJSON

}

if header.Get("X-Etcd-Index") != "" {

res.Index, err = strconv.ParseUint(header.Get("X-Etcd-Index"), 10, 64)

if err != nil {

return nil, err

}

}

res.ClusterID = header.Get("X-Etcd-Cluster-ID")

return &res, nil

}

Maybe it's a test issue, but in the test that I added, I'm getting "response=nil" in this case...

@liggitt - what etcd version you used in your experiment? Maybe that matters?

Actually - this seem to be an issue of etcd client library:
https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/coreos/etcd/client/keys.go#L654

In case of status different than OK and Created (so 200 and 201), the returned response is nil.

I'm getting "response=nil" in this case...

yes. the index is in the error object:

kubernetes/vendor/github.com/coreos/etcd/client/keys.go

Lines 57 to 62 in e85b81b

type Error struct {

Code int `json:"errorCode"`

Message string `json:"message"`

Cause string `json:"cause"`

Index uint64 `json:"index"`

}

Aaah ok - thanks. It works now. PTAL

wojtek-t · 2018-05-22T15:46:00Z

etcd v2 should also be fixed now. PTAL

liggitt · 2018-05-22T17:10:59Z

staging/src/k8s.io/apiserver/pkg/storage/etcd/etcd_helper.go

@@ -347,7 +347,8 @@ func (h *etcdHelper) GetToList(ctx context.Context, key string, resourceVersion
 	metrics.RecordEtcdRequestLatency("get", getTypeName(listPtr), startTime)
 	if err != nil {
 		if etcdutil.IsEtcdNotFound(err) {
-			return nil
+			etcdErr := err.(etcd.Error)


maybe be a little more fault tolerant... do a conditional cast, and only call UpdateList if the cast is successful and etcdErr.Index != 0?

Fair enough - fixed.

wojtek-t

@liggitt - PTAL

wojtek-t · 2018-05-23T06:06:53Z

staging/src/k8s.io/apiserver/pkg/storage/etcd/etcd_helper.go

@@ -347,7 +347,8 @@ func (h *etcdHelper) GetToList(ctx context.Context, key string, resourceVersion
 	metrics.RecordEtcdRequestLatency("get", getTypeName(listPtr), startTime)
 	if err != nil {
 		if etcdutil.IsEtcdNotFound(err) {
-			return nil
+			etcdErr := err.(etcd.Error)


Fair enough - fixed.

wojtek-t · 2018-05-23T06:07:16Z

staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go

-	if err != nil {
-		return err
-	}
-	if len(getResp.Kvs) == 0 {


Yes - that's true.
But if you don't feel very strongly about this, I would prefer leaving as is.

liggitt · 2018-05-23T06:31:21Z

/lgtm

k8s-ci-robot · 2018-05-23T06:31:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/pkg/storage/OWNERS~~ [liggitt,wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wojtek-t · 2018-05-23T06:47:55Z

/test pull-kubernetes-e2e-kops-aws

k8s-github-robot · 2018-05-23T11:53:19Z

Automatic merge from submit-queue (batch tested with PRs 64102, 63303, 64150, 63841). If you want to cherry-pick this change to another branch, please follow the instructions here.

…50-upstream-release-1.10 Automatic merge from submit-queue. Automated cherry pick of #64150 upstream release 1.10 Cherry pick of #64150 on release-1.10. #64150 : Fix incorrectly set resource version in List

jpbetz · 2018-05-30T18:44:51Z

/lgtm

…50-upstream-release-1.9 Automatic merge from submit-queue. Automated cherry pick of #64150 upstream release 1.9 Cherry pick of #64150 on release-1.9. #64150 : Fix incorrectly set resource version in List

wojtek-t assigned liggitt and deads2k May 22, 2018

k8s-ci-robot requested review from hongchaodeng and timothysc May 22, 2018 13:33

wojtek-t mentioned this pull request May 22, 2018

ResourceVersion is not returned from LIST requests with fieldSelector #64147

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 22, 2018

wojtek-t added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 22, 2018

wojtek-t added this to the v1.11 milestone May 22, 2018

k8s-github-robot added the milestone/incomplete-labels label May 22, 2018

liggitt reviewed May 22, 2018

View reviewed changes

timothysc removed their request for review May 22, 2018 13:48

wojtek-t force-pushed the fix_list_resource_version branch from 872296b to 6b9383d Compare May 22, 2018 14:02

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 22, 2018

liggitt reviewed May 22, 2018

View reviewed changes

wojtek-t force-pushed the fix_list_resource_version branch from 6b9383d to 6a470d1 Compare May 22, 2018 15:45

liggitt reviewed May 22, 2018

View reviewed changes

Fix incorrectly set resource version in List

a3578c8

wojtek-t force-pushed the fix_list_resource_version branch from 6a470d1 to a3578c8 Compare May 23, 2018 06:06

wojtek-t commented May 23, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2018

k8s-github-robot merged commit 9716f58 into kubernetes:master May 23, 2018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels May 28, 2018

This was referenced May 28, 2018

Automated cherry pick of #64150 upstream release 1.10 #64395

Merged

Automated cherry pick of #64150 upstream release 1.9 #64397

Merged

wojtek-t deleted the fix_list_resource_version branch July 3, 2018 12:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix incorrectly set resource version in List #64150

Fix incorrectly set resource version in List #64150

wojtek-t commented May 22, 2018 •

edited

Loading

k8s-github-robot commented May 22, 2018

liggitt May 22, 2018

wojtek-t May 22, 2018

liggitt May 22, 2018

wojtek-t May 22, 2018

liggitt May 22, 2018

wojtek-t May 23, 2018

liggitt commented May 22, 2018

wojtek-t commented May 22, 2018

liggitt May 22, 2018

wojtek-t May 22, 2018

wojtek-t May 22, 2018

wojtek-t May 22, 2018

liggitt May 22, 2018 •

edited

Loading

wojtek-t May 22, 2018

wojtek-t commented May 22, 2018

liggitt May 22, 2018 •

edited

Loading

wojtek-t May 23, 2018

wojtek-t left a comment

wojtek-t May 23, 2018

wojtek-t May 23, 2018

liggitt commented May 23, 2018

k8s-ci-robot commented May 23, 2018

wojtek-t commented May 23, 2018

k8s-github-robot commented May 23, 2018

jpbetz commented May 30, 2018

	type Error struct {
	Code int `json:"errorCode"`
	Message string `json:"message"`
	Cause string `json:"cause"`
	Index uint64 `json:"index"`
	}

	func unmarshalSuccessfulKeysResponse(header http.Header, body []byte) (*Response, error) {
	var res Response
	err := codec.NewDecoderBytes(body, new(codec.JsonHandle)).Decode(&res)
	if err != nil {
	return nil, ErrInvalidJSON
	}
	if header.Get("X-Etcd-Index") != "" {
	res.Index, err = strconv.ParseUint(header.Get("X-Etcd-Index"), 10, 64)
	if err != nil {
	return nil, err
	}
	}
	res.ClusterID = header.Get("X-Etcd-Cluster-ID")
	return &res, nil
	}

Fix incorrectly set resource version in List #64150

Fix incorrectly set resource version in List #64150

Conversation

wojtek-t commented May 22, 2018 • edited Loading

k8s-github-robot commented May 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented May 22, 2018

wojtek-t commented May 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt May 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wojtek-t commented May 22, 2018

liggitt May 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wojtek-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented May 23, 2018

k8s-ci-robot commented May 23, 2018

wojtek-t commented May 23, 2018

k8s-github-robot commented May 23, 2018

jpbetz commented May 30, 2018

wojtek-t commented May 22, 2018 •

edited

Loading

liggitt May 22, 2018 •

edited

Loading

liggitt May 22, 2018 •

edited

Loading