Skip to content

log restore: filter backupmeta file by ts to speed up pitr#61347

Merged
ti-chi-bot[bot] merged 7 commits intopingcap:masterfrom
3pointer:filter_backupmeta
Jun 3, 2025
Merged

log restore: filter backupmeta file by ts to speed up pitr#61347
ti-chi-bot[bot] merged 7 commits intopingcap:masterfrom
3pointer:filter_backupmeta

Conversation

@3pointer
Copy link
Copy Markdown
Contributor

@3pointer 3pointer commented May 27, 2025

What problem does this PR solve?

Issue Number: ref #61318

Problem Summary:

  1. As described in the issue.

What changed and how does it work?

  1. Improved timestamp-based filtering logic for pitr .
  2. Ensured compatibility with previous and future versions of TiKV by gracefully handling unexpected or legacy formats.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Support filtering log backup meta files by timestamp to improve PITR performance.

@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented May 27, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 27, 2025
@tiprow
Copy link
Copy Markdown

tiprow Bot commented May 27, 2025

Hi @3pointer. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels May 27, 2025
Signed-off-by: 3pointer <luancheng@pingcap.com>
@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 27, 2025
@3pointer 3pointer changed the title filter backupmeta file by ts to speed up pitr log restore: filter backupmeta file by ts to speed up pitr May 28, 2025
@3pointer 3pointer marked this pull request as ready for review May 28, 2025 09:39
@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 28, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented May 28, 2025

Codecov Report

Attention: Patch coverage is 90.09901% with 10 lines in your changes missing coverage. Please review.

Project coverage is 74.7513%. Comparing base (998279f) to head (198637b).
Report is 218 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #61347        +/-   ##
================================================
+ Coverage   73.1815%   74.7513%   +1.5698%     
================================================
  Files          1726       1729         +3     
  Lines        478595     489546     +10951     
================================================
+ Hits         350243     365942     +15699     
+ Misses       106890     101173      -5717     
- Partials      21462      22431       +969     
Flag Coverage Δ
integration 46.3428% <81.1881%> (?)
unit 72.3567% <82.1782%> (-0.0957%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.7804% <ø> (ø)
parser ∅ <ø> (∅)
br 60.2289% <90.0990%> (+12.6802%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@YuJuncen YuJuncen requested a review from Copilot May 30, 2025 06:23
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves PITR performance by introducing timestamp‐based filtering for backup meta files. Key changes include new filtering logic via FilterPathByTs, corresponding updates in metadata scanning functions, and new unit tests verifying the filtering behavior.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
br/pkg/stream/stream_misc_test.go Added comprehensive tests for the new FilterPathByTs function.
br/pkg/stream/stream_mgr.go Introduced FilterPathByTs for timestamp-based filtering and updated FastUnmarshalMetaData usage.
br/pkg/stream/stream_metas.go Adjusted FastUnmarshalMetaData invocation to match the new function signature.
br/pkg/restore/log_client/log_file_manager.go Updated meta file processing to incorporate the new filtering logic and removed a redundant parameter.
br/pkg/restore/log_client/BUILD.bazel Added a dependency on the tikv oracle package.

Comment thread br/pkg/stream/stream_mgr.go Outdated
Comment on lines +384 to +386
minTs, _ := strconv.ParseUint(matches[1], 10, 64)
maxTs, _ := strconv.ParseUint(matches[2], 10, 64)
minDefaultTs, _ := strconv.ParseUint(matches[3], 10, 64)
Copy link

Copilot AI May 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider handling potential errors from strconv.ParseUint instead of ignoring them, to prevent unintended behavior if the filename format is malformed.

Suggested change
minTs, _ := strconv.ParseUint(matches[1], 10, 64)
maxTs, _ := strconv.ParseUint(matches[2], 10, 64)
minDefaultTs, _ := strconv.ParseUint(matches[3], 10, 64)
minTs, err := strconv.ParseUint(matches[1], 10, 64)
if err != nil {
log.Error("failed to parse minTs from filename", zap.String("file", path), zap.Error(err))
return path
}
maxTs, err := strconv.ParseUint(matches[2], 10, 64)
if err != nil {
log.Error("failed to parse maxTs from filename", zap.String("file", path), zap.Error(err))
return path
}
minDefaultTs, err := strconv.ParseUint(matches[3], 10, 64)
if err != nil {
log.Error("failed to parse minDefaultTs from filename", zap.String("file", path), zap.Error(err))
return path
}

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor

@YuJuncen YuJuncen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

Comment thread br/pkg/stream/stream_mgr.go Outdated
streamBackupGlobalCheckpointPrefix = "v1/global_checkpoint"
)

var metaPattern = regexp.MustCompile(`^(\d+)-(\d+)-(\d+)-[^/]+$`)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can encode the filename as hex so they can be compared byte by byte?

@ti-chi-bot ti-chi-bot Bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 30, 2025
@3pointer
Copy link
Copy Markdown
Contributor Author

/hold

@ti-chi-bot ti-chi-bot Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 30, 2025
@3pointer
Copy link
Copy Markdown
Contributor Author

3pointer commented Jun 3, 2025

/unhold

@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 3, 2025
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented Jun 3, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leavrth, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 3, 2025
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented Jun 3, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-05-30 07:10:20.947834629 +0000 UTC m=+175642.009852235: ☑️ agreed by YuJuncen.
  • 2025-06-03 09:48:22.04454465 +0000 UTC m=+348788.246889915: ☑️ agreed by Leavrth.

@ti-chi-bot ti-chi-bot Bot merged commit c8e7dd4 into pingcap:master Jun 3, 2025
23 checks passed
@ti-chi-bot ti-chi-bot Bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Jul 1, 2025
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jul 1, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Copy Markdown
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #62115.
But this PR has conflicts, please resolve them!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants