[store] Change the Restore action on objects to update instead of delete/create

[cyli]

Since if the object already exists, the event produced should be an update and not a delete/create.

This is probably the cause of moby/moby#33541, where an unlock key change is sometimes is not picked up by a snapshot restore from another node.

cc @aaronlehmann

Also, :( generics

[codecov]

Codecov Report

Merging #2281 into master will increase coverage by 0.58%.
The diff coverage is 82.6%.

@@            Coverage Diff             @@
##           master    #2281      +/-   ##
==========================================
+ Coverage   60.38%   60.96%   +0.58%     
==========================================
  Files         125      126       +1     
  Lines       20412    20373      -39     
==========================================
+ Hits        12326    12421      +95     
+ Misses       6699     6581     -118     
+ Partials     1387     1371      -16

[aaronlehmann]

I tried to simplify the Restore function. This is what I came up with:

                Restore: func(tx Tx, snapshot *api.StoreSnapshot) error {
                        clusters, err := FindClusters(tx, All)
                        if err != nil {
                                return err
                        }
                        updated := make(map[string]struct{})
                        for _, n := range snapshot.Clusters {
                                if err := UpdateCluster(tx, n); err == ErrNotExist {
                                        if err := CreateCluster(tx, n); err != nil {
                                                return err
                                        }
                                } else if err != nil {
                                        return err
                                } else {
                                        updated[n.ID] = struct{}{}
                                }
                        }
                      for _, n := range clusters {
                                if _, ok := updated[n.ID]; !ok {
                                        if err := DeleteCluster(tx, n.ID); err != nil {
                                                return err
                                        }
                                }
                        }
                        return nil
                },

I'm not sure it's really better, to be honest. What do you think?

Another idea I have is to have the type-specific Restore functions convert the slice to []api.StoreObject and call a type-independent helper function that uses tx methods:

func RestoreTable(tx Tx, table string, newObjects []api.StoreObject) error {
        checkType := func(by By) error {
                return nil
        }
        var oldObjects []api.StoreObject
        appendResult := func(o api.StoreObject) {
                oldObjects = append(oldObjects, o)
        }

        err := tx.find(table, All, checkType, appendResult)
        if err != nil {
                return nil
        }

        updated := make(map[string]struct{})

        for _, o := range newObjects {
                if existing := tx.lookup(table, indexID, o.GetID()); existing != nil {
                        if err := tx.update(table, o); err != nil {
                                return err
                        }
                        updated[o.GetID()] = struct{}{}
                } else {
                        if err := tx.create(table, o); err != nil {
                                return err
                        }
                }
        }
        for _, o := range oldObjects {
                if _, ok := updated[o.GetID()]; !ok {
                        if err := tx.delete(table, o.GetID()); err != nil {
                                return err
                        }
                }
        }
        return nil
}

I used my algorithm here as a proof of concept, but I'd be fine with using yours instead as long as it's only implemented in one place. Yours might be a bit more memory-efficient.

The downsides of this approach are that it needs an extra slice copy, and that it skips the validation in the type-specific Create* / Update* wrappers. But any objects inside a snapshot should already have passed validation. Ultimately I like this solution because it removes the bulk of the duplicated code.

[cyli]

@aaronlehmann I really like the idea of de-duplication - if we get more objects (such as generic resources) it will make the store code much easier to maintain. I like your algorithm better - I think it's actually the more memory efficient one, since it only creates one map, and a bit easier to read. Thanks!

[aaronlehmann]

Let's call GetID a single time for each loop iteration. I was sloppy when I wrote the PoC.

[cyli]

Happy to make this change, but out of curiosity, is the function call particularly expensive? All the objects just seem to return the ID field, and there doesn't seem to be much indirection?

[aaronlehmann]

No, it's not expensive.

[cyli]

Why is it better practice to only make the function call once? I don't particularly have an opinion either way, I was just wondering if it was a style reason or some other reason?

[aaronlehmann]

I think it makes the code a little easier to read and it has a very marginal performance impact because it avoids a redundant function call, but that's so insignificant I hesitate to bring it up.

I guess I don't care either. Just disagreeing with myself, reviewing my own code.

[aaronlehmann]

Let's call GetID a single time for each loop iteration. I was sloppy when I wrote the PoC.

[aaronlehmann]

Almost LGTM. extensions.go and resources.go also need these changes.

[cyli]

@aaronlehmann Question about adding this to extensions - currently extensions are immutable - if we restore, we'd emit an EventUpdateExtension. Technically I guess we aren't updating anything, we are just restoring from a snapshot, but I just wanted to check that this was ok?

[aaronlehmann]

LGTM