br/cmd: added operator checksum-as|force-flush by YuJuncen · Pull Request #58801 · pingcap/tidb

YuJuncen · 2025-01-08T10:06:56Z

What problem does this PR solve?

Issue Number: close #58798 #58799

Problem Summary:
See the issue description.

What changed and how does it work?

This PR added two new subcommands to br operator:

checksum-as: this generates rewrite rule from a storage and performs checksum.
force-flush: this allows user to force trigger a flush.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
I used these utilities when executing some internal tests. They works fine as far as I go.
The scenarios including:

Checksum in a cluster with log backup enabled. (Upstream)
Restore point in a new cluster. (Downstream)
Run checksum-as where -s is the --full-backup-storage in the restore point command.
Compare the checksum.

(With the compatibility solution between log backup and ingesting, which doesn't present in master branch yet)

Run checksum-as after restoring to a cluster, where -s is the restored backup.
After a point in time restore to another cluster, run checksum-as over the tables restored where -s is the restored backup in 1..

The force-flush part was used in integration tests.

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: hillium <yu745514916@live.com>

tiprow · 2025-01-08T10:07:16Z

Hi @YuJuncen. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-01-08T11:03:26Z

Codecov Report

Attention: Patch coverage is 25.49020% with 228 lines in your changes missing coverage. Please review.

Project coverage is 74.7206%. Comparing base (87867eb) to head (547de59).
Report is 4 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #58801        +/-   ##
================================================
+ Coverage   73.1124%   74.7206%   +1.6081%     
================================================
  Files          1678       1695        +17     
  Lines        464111     464746       +635     
================================================
+ Hits         339323     347261      +7938     
+ Misses       103907      95904      -8003     
- Partials      20881      21581       +700

Flag	Coverage Δ
integration	`46.2655% <25.4902%> (?)`
unit	`72.3398% <10.1307%> (+0.0348%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.4146% <25.4902%> (+15.6585%)`	⬆️

YuJuncen · 2025-01-09T01:23:11Z

/retest-required

tiprow · 2025-01-09T01:23:32Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: hillium <yu745514916@live.com>

YuJuncen · 2025-01-09T02:35:40Z

/retest-required

tiprow · 2025-01-09T02:36:02Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

YuJuncen · 2025-01-09T02:43:22Z

merge: 476f860c09e8b068b9b5ebffc2bf28a134bfa221 - not something we can merge ...?

/test build

tiprow · 2025-01-09T02:43:43Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

merge: 476f860c09e8b068b9b5ebffc2bf28a134bfa221 - not something we can merge ...?

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

YuJuncen · 2025-01-09T03:54:11Z

/test build

tiprow · 2025-01-09T03:54:32Z

@YuJuncen: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…e-new-opeartor-commands

BornChanger · 2025-01-13T08:25:52Z

+type ForceFlushConfig struct {
+	task.Config
+
+	StoresPattern *regexp.Regexp


please give a simple sample

3pointer

rest LGTM

3pointer · 2025-01-14T07:23:27Z

+	log.Info("About to start force flushing.", zap.Stringer("stores-pattern", cfg.StoresPattern))
+	for _, s := range tikvs {
+		if !cfg.StoresPattern.MatchString(s.Address) || engine.IsTiFlash(s) {
+			log.Info("Skipping TiFlash or not matched TiKV.",


miss continue.

ti-chi-bot · 2025-01-14T09:11:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, BornChanger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [3pointer,BornChanger]
~~br/OWNERS~~ [3pointer,BornChanger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-01-14T09:11:24Z

[LGTM Timeline notifier]

Timeline:

2025-01-14 06:59:33.142729419 +0000 UTC m=+15444.597775568: ☑️ agreed by BornChanger.
2025-01-14 09:11:23.475521968 +0000 UTC m=+23354.930568117: ☑️ agreed by 3pointer.

close pingcap#58798

…58801) (pingcap#239)

…58801) X-Original-Commit: 7b6e626

YuJuncen added 2 commits January 8, 2025 16:50

added operator checksum-as|force-flush

45d0c5b

added skip tiflash and utilized for integration test

94c29ae

Signed-off-by: hillium <yu745514916@live.com>

ti-chi-bot Bot added do-not-merge/invalid-title release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 8, 2025

YuJuncen changed the title ~~added operator checksum-as|force-flush~~ br/cmd: added operator checksum-as|force-flush Jan 8, 2025

ti-chi-bot Bot removed the do-not-merge/invalid-title label Jan 8, 2025

YuJuncen added 3 commits January 8, 2025 18:09

revert irrelative changes

292280b

make bazel_prepare

6f38e25

fix build

728145e

fix build

476f860

Signed-off-by: hillium <yu745514916@live.com>

YuJuncen closed this Jan 9, 2025

YuJuncen reopened this Jan 9, 2025

YuJuncen added 2 commits January 9, 2025 13:44

Merge branch 'master' of https://github.com/pingcap/tidb into add-som…

fd56324

…e-new-opeartor-commands

make linter happy

15664c5

BornChanger reviewed Jan 10, 2025

View reviewed changes

Comment thread br/pkg/task/operator/force_flush.go

BornChanger reviewed Jan 13, 2025

View reviewed changes

BornChanger approved these changes Jan 14, 2025

View reviewed changes

ti-chi-bot Bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 14, 2025

Merge branch 'master' into add-some-new-opeartor-commands

c0b96ea

3pointer reviewed Jan 14, 2025

View reviewed changes

YuJuncen added 2 commits January 14, 2025 15:34

fix build

40dfca5

address comments

547de59

3pointer approved these changes Jan 14, 2025

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 14, 2025

ti-chi-bot Bot merged commit 0c6681f into pingcap:master Jan 14, 2025

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Jan 16, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

a76c982

close pingcap#58798

YuJuncen mentioned this pull request Jan 16, 2025

br: Release 8.5 20250114 v8.5.0 #58981

Merged

13 tasks

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Jan 17, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

2ce950b

close pingcap#58798

YuJuncen mentioned this pull request Feb 19, 2025

br: Cherry pick 58981 to release 8.5 #59638

Closed

13 tasks

zeminzhou pushed a commit to zeminzhou/tidb that referenced this pull request May 6, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

54528c1

close pingcap#58798

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Dec 1, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

04d2858

close pingcap#58798

YuJuncen mentioned this pull request Dec 2, 2025

br: pick commits from master to release-8.5; part 1 #64803

Closed

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Dec 8, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

a7026b7

close pingcap#58798

YuJuncen mentioned this pull request Dec 8, 2025

br: pick "log backup full backup compactibility" to release-8.5 #64910

Merged

13 tasks

YuJuncen added a commit to YuJuncen/tidb that referenced this pull request Dec 9, 2025

br/cmd: added operator checksum-as|force-flush (pingcap#58801)

8ed369e

close pingcap#58798

AvaisP added a commit to AvaisP/tidb that referenced this pull request Mar 2, 2026

Cherry-pick: br/cmd: added operator checksum-as|force-flush (pingcap#…

d051bd0

…58801) (pingcap#239)

AvaisP added a commit to AvaisP/tidb that referenced this pull request Mar 2, 2026

Cherry-pick: br/cmd: added operator checksum-as|force-flush (pingcap#…

1ef7142

…58801) (pingcap#239)

AvaisP added a commit to AvaisP/tidb that referenced this pull request Mar 4, 2026

Cherry-pick: br/cmd: added operator checksum-as|force-flush (pingcap#…

6a3967d

…58801) (pingcap#239)

ClamChowderTiDB added a commit to ClamChowderTiDB/tidb that referenced this pull request Mar 31, 2026

Cherry-pick: br/cmd: added operator checksum-as|force-flush (pingcap#…

5c2978e

…58801) X-Original-Commit: 7b6e626

Conversation

YuJuncen commented Jan 8, 2025

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Jan 8, 2025

Uh oh!

codecov Bot commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

YuJuncen commented Jan 9, 2025

Uh oh!

tiprow Bot commented Jan 9, 2025

Uh oh!

YuJuncen commented Jan 9, 2025

Uh oh!

tiprow Bot commented Jan 9, 2025

Uh oh!

YuJuncen commented Jan 9, 2025

Uh oh!

tiprow Bot commented Jan 9, 2025

Uh oh!

YuJuncen commented Jan 9, 2025

Uh oh!

tiprow Bot commented Jan 9, 2025

Uh oh!

Uh oh!

BornChanger Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

3pointer left a comment

Choose a reason for hiding this comment

Uh oh!

3pointer Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Jan 14, 2025

Uh oh!

ti-chi-bot Bot commented Jan 14, 2025

[LGTM Timeline notifier]

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jan 8, 2025 •

edited

Loading