Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't store no-op updates in etcd. #20897

Merged
merged 1 commit into from
Feb 13, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 9 additions & 0 deletions pkg/registry/generic/etcd/etcd.go
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,15 @@ func (e *Etcd) Update(ctx api.Context, obj runtime.Object) (runtime.Object, bool
creating := false
out := e.NewFunc()
err = e.Storage.GuaranteedUpdate(ctx, key, out, true, func(existing runtime.Object, res storage.ResponseMeta) (runtime.Object, *uint64, error) {
// Since we return 'obj' from this function and it can be modified outside this
// function, we are resetting resourceVersion to the initial value here.
//
// TODO: In fact, we should probably return a DeepCopy of obj in all places.
err := e.Storage.Versioner().UpdateObject(obj, nil, resourceVersion)
if err != nil {
return nil, nil, err
}

version, err := e.Storage.Versioner().ObjectResourceVersion(existing)
if err != nil {
return nil, nil, err
Expand Down
59 changes: 59 additions & 0 deletions pkg/registry/generic/etcd/etcd_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,65 @@ func TestEtcdUpdate(t *testing.T) {

}

func TestNoOpUpdates(t *testing.T) {
server, registry := NewTestGenericEtcdRegistry(t)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if possible, can we also have a separate test that inserts an old incorrect record (containing resourceVersion) into etcd, makes sure a no-op update is ignored, and an actual update is persisted?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry - I'm afraid I don't understand:

  • what do you mean by "inserts an old incorrect record (containing resourceVersion) into etcd"?
  • actual updates are tested by a bunch of other tests, I don't think we need to test it here again

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the other tests are testing updates against new records made with the current version which will not set resourceVersion in etcd. I want to make sure when this fix runs against an etcd containing old incorrect records which contain resourceVersion, we get correct update behavior

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it - makes sense (although in that case no-op update is not no-op so it will be performed). But having such test makes sense.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add such test tomorrow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - that should be the case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm - it seems that there is no easy way to store object with ResourceVersion set in etcd after my fixes.
I manually checked that it works as expected and added a TODO for it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can, but we don't have such, so this would make this change much bigger (it will take some time to implement it).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defer server.Terminate(t)

newPod := func() *api.Pod {
return &api.Pod{
ObjectMeta: api.ObjectMeta{
Namespace: api.NamespaceDefault,
Name: "foo",
Labels: map[string]string{"prepare_create": "true"},
},
Spec: api.PodSpec{NodeName: "machine"},
}
}

var err error
var createResult runtime.Object
if createResult, err = registry.Create(api.NewDefaultContext(), newPod()); err != nil {
t.Fatalf("Unexpected error: %v", err)
}

createdPod, err := registry.Get(api.NewDefaultContext(), "foo")
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}

var updateResult runtime.Object
if updateResult, _, err = registry.Update(api.NewDefaultContext(), newPod()); err != nil {
t.Fatalf("Unexpected error: %v", err)
}

// Check whether we do not return empty result on no-op update.
if !reflect.DeepEqual(createResult, updateResult) {
t.Errorf("no-op update should return a correct value, got: %#v", updateResult)
}

updatedPod, err := registry.Get(api.NewDefaultContext(), "foo")
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}

createdMeta, err := meta.Accessor(createdPod)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
updatedMeta, err := meta.Accessor(updatedPod)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}

if createdMeta.GetResourceVersion() != updatedMeta.GetResourceVersion() {
t.Errorf("no-op update should be ignored and not written to etcd")
}
}

// TODO: Add a test to check no-op update if we have object with ResourceVersion
// already stored in etcd. Currently there is no easy way to store object with
// ResourceVersion in etcd.

type testPodExport struct{}

func (t testPodExport) Export(obj runtime.Object, exact bool) error {
Expand Down
2 changes: 1 addition & 1 deletion pkg/registry/serviceaccount/etcd/etcd_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ func TestUpdate(t *testing.T) {
// updateFunc
func(obj runtime.Object) runtime.Object {
object := obj.(*api.ServiceAccount)
// TODO: Update this serviceAccount
object.Secrets = []api.ObjectReference{{}}
return object
},
)
Expand Down
35 changes: 33 additions & 2 deletions pkg/storage/etcd/etcd_helper.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,24 @@ func (h *etcdHelper) Set(ctx context.Context, key string, obj, out runtime.Objec
if ctx == nil {
glog.Errorf("Context is nil")
}

version := uint64(0)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW - Set() method is used only in tests. I'm happy to remove this method from the interface if you agree it's not needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's open a follow up issue post 1.3 - I don't think we have to remove it now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #20963 for it.

if h.versioner != nil {
var err error
if version, err = h.versioner.ObjectResourceVersion(obj); err != nil {
return errors.New("couldn't get resourceVersion from object")
}
if version != 0 {
// We cannot store object with resourceVersion in etcd, we need to clear it here.
if err := h.versioner.UpdateObject(obj, nil, 0); err != nil {
return errors.New("resourceVersion cannot be set on objects store in etcd")
}
}
}
// TODO: If versioner is nil, then we may end up with having ResourceVersion set
// in the object and this will be incorrect ResourceVersion. We should fix it by
// requiring "versioner != nil" at the constructor level for 1.3 milestone.

var response *etcd.Response
data, err := runtime.Encode(h.codec, obj)
if err != nil {
Expand All @@ -200,7 +218,7 @@ func (h *etcdHelper) Set(ctx context.Context, key string, obj, out runtime.Objec

create := true
if h.versioner != nil {
if version, err := h.versioner.ObjectResourceVersion(obj); err == nil && version != 0 {
if version != 0 {
create = false
startTime := time.Now()
opts := etcd.SetOptions{
Expand Down Expand Up @@ -558,6 +576,16 @@ func (h *etcdHelper) GuaranteedUpdate(ctx context.Context, key string, ptrToType
ttl = *newTTL
}

// Since update object may have a resourceVersion set, we need to clear it here.
if h.versioner != nil {
if err := h.versioner.UpdateObject(ret, meta.Expiration, 0); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is RV the only field like this? How about SelfLink, can we clear that too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can, but then we can't use versioner for it. And since we will need to use meta.Accessor, then there is no need to use versioner at all. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smarterclayton - sorry, I'm afraid I don't understand.
You suggest leaving as is or changing (to not use versioner and just work on meta.Accessor)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with punting on a self link fix. That'd only be a problem with objects available from multiple URLs, and we won't have many for 1.2.

return errors.New("resourceVersion cannot be set on objects store in etcd")
}
}
// TODO: If versioner is nil, then we may end up with having ResourceVersion set
// in the object and this will be incorrect ResourceVersion. We should fix it by
// requiring "versioner != nil" at the constructor level for 1.3 milestone.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smarterclayton - I browsed through the code, and basically there is no easy way to construct etcdHelper with nil versioner (all constructors set it to ApiObjectVersioner{}) and we don't have setters to change it.
So what do you think about fixing this TODO in a followup PR now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data, err := runtime.Encode(h.codec, ret)
if err != nil {
return err
Expand All @@ -580,7 +608,10 @@ func (h *etcdHelper) GuaranteedUpdate(ctx context.Context, key string, ptrToType
}

if string(data) == origBody {
return nil
// If we don't send an update, we simply return the currently existing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes. Can you add a test case for this condition so we don't regress?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell if you added one or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I didn't. I will add tomorrow.
@smarterclayton Are you fine with me self-applying "lgtm" label after adding a test?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, a test for this would be good--subtle problem.

// version of the object.
_, _, err := h.extractObj(res, nil, ptrToType, ignoreNotFound, false)
return err
}

startTime := time.Now()
Expand Down
1 change: 0 additions & 1 deletion pkg/storage/etcd/etcd_helper_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,6 @@ func TestSetNilOutParam(t *testing.T) {
server := etcdtesting.NewEtcdTestClientServer(t)
defer server.Terminate(t)
helper := newEtcdHelper(server.Client, testapi.Default.Codec(), etcdtest.PathPrefix())
helper.versioner = nil
err := helper.Set(context.TODO(), "/some/key", obj, nil, 3)
if err != nil {
t.Errorf("Unexpected error %#v", err)
Expand Down