Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding optional revision bump to snapshot restore #16029

Merged
merged 1 commit into from
Jul 3, 2023

Conversation

dusk125
Copy link
Contributor

@dusk125 dusk125 commented Jun 7, 2023

This PR adds the flags --bump-revision and --mark-compacted to etcdutl snapshot restore for #16028.

I chose to handle this after the database is loaded from the snapshot, so as to not modify the snapshot, I didn't want to risk corrupting the snapshot as a part of this optional step. If for some reason the bump fails, a user would still be able to restore the snapshot again without the bump and restart the watchers manually.

@dusk125 dusk125 force-pushed the revision-bump branch 5 times, most recently from 9cd9656 to 64cefc6 Compare June 7, 2023 18:24
@serathius
Copy link
Member

To clarify by bumping revision I didn't mean take transform whole change history by shifting revisions of all modifications, but to just bump revision of the next transaction.

@dusk125
Copy link
Contributor Author

dusk125 commented Jun 12, 2023

With this latest update, I've changed the revision-bump flag to only increase the latest revision by the amount so that clients shouldn't see decreasing revisions.
Also, I added the other flag, mark-compacted, to set the latest revision as a scheduled "in progress" compaction so that watchers will probably fail and resync.

etcdutl/snapshot/util.go Outdated Show resolved Hide resolved
etcdutl/snapshot/util.go Outdated Show resolved Hide resolved
etcdutl/snapshot/util.go Outdated Show resolved Hide resolved
etcdutl/snapshot/util.go Outdated Show resolved Hide resolved
Copy link
Contributor

@tjungblu tjungblu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm (nonbinding)

@serathius
Copy link
Member

cc @ahrtr @ptabor @spzala @mitake

@ahrtr ahrtr self-requested a review June 23, 2023 15:02
etcdutl/etcdutl/snapshot_command.go Outdated Show resolved Hide resolved
etcdutl/etcdutl/snapshot_command.go Outdated Show resolved Hide resolved
etcdutl/snapshot/v3_snapshot.go Outdated Show resolved Hide resolved
)

latest.main += amount
k := [17]byte{}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially agree with @tjungblu . We should consider moving all revision related common functions (i.e. revToBytes, bytesToRev, newRevBytes) into package pkg (or api), so that they can be reused.

In this case, we should try to reuse newRevBytes. Please raise a followup ticket.

etcdutl/snapshot/v3_snapshot.go Outdated Show resolved Hide resolved
@dusk125
Copy link
Contributor Author

dusk125 commented Jun 27, 2023

I think that test failure may be a flake, running locally they succeeded. Can someone relaunch that test please?

return latest
}

func (s *v3Manager) unsafeMarkRevisionCompacted(tx backend.BatchTx, latest revision) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional nit: it might be friendly if https://github.com/etcd-io/etcd/blob/main/server/storage/mvcc/kvstore.go#L201 can have a comment like "similar logic can be found in etcdctl side, this should be synced with the command"? Currently they share UnsafeSetScheduledCompact so it's not easy to make them diverged. If we add additional method calls to server side, the comment will be helpful for not forgetting about the etcdutl side logic (such change might be quite unlikely though).

Copy link
Member

@ahrtr ahrtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @dusk125

@ahrtr
Copy link
Member

ahrtr commented Jun 29, 2023

Two left actions:

  • Please raise a followup ticket for moving some revision related common functions to package "pkg", see Adding optional revision bump to snapshot restore #16029 (comment)
  • Please raise a followup ticket for documentation for the flag "--mark-compact".
    • It isn't allowed to mark compact when users don't bump revision;
    • If users bump revision, then users must specify the flag. The purpose is to make sure users are explicitly aware of the operation (mark compaction). (Note that if we receive too many negative feedback from users, we may consider to remove the flag, and go for 2.1 in future).

@dusk125
Copy link
Contributor Author

dusk125 commented Jun 30, 2023

Two left actions:

  • Please raise a followup ticket for moving some revision related common functions to package "pkg", see Adding optional revision bump to snapshot restore #16029 (comment)

  • Please raise a followup ticket for documentation for the flag "--mark-compact".

    • It isn't allowed to mark compact when users don't bump revision;
    • If users bump revision, then users must specify the flag. The purpose is to make sure users are explicitly aware of the operation (mark compaction). (Note that if we receive too many negative feedback from users, we may consider to remove the flag, and go for 2.1 in future).

Tickets have been raised. I'll be on holiday for the next two weeks, so I can handle them when I get back (July 17th).

@tjungblu
Copy link
Contributor

tjungblu commented Jul 3, 2023

While @dusk125 is out, I can take over the 3.5 backport. Can anyone hit the merge button? Thanks.

@ahrtr ahrtr merged commit c6fd719 into etcd-io:main Jul 3, 2023
27 checks passed
wenjiaswe added a commit to wenjiaswe/etcd that referenced this pull request Jul 7, 2023
Signed-off-by: Wenjia Zhang <wenjiazhang@google.com>
wenjiaswe added a commit to wenjiaswe/etcd that referenced this pull request Jul 7, 2023
Signed-off-by: Wenjia Zhang <wenjiazhang@google.com>
wenjiaswe added a commit to wenjiaswe/etcd that referenced this pull request Jul 7, 2023
Signed-off-by: Wenjia Zhang <wenjiazhang@google.com>
serathius added a commit that referenced this pull request Jul 7, 2023
@serathius serathius mentioned this pull request Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants