Skip to content

[fix](backup) Strip replica info from backup meta#65321

Open
Ryan19929 wants to merge 1 commit into
apache:masterfrom
Ryan19929:backup-strip-replica-info
Open

[fix](backup) Strip replica info from backup meta#65321
Ryan19929 wants to merge 1 commit into
apache:masterfrom
Ryan19929:backup-strip-replica-info

Conversation

@Ryan19929

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Historical Context

Backup metadata has stored the full Table object since the original backup/restore implementation was introduced in #224. As a result, Replica objects were included implicitly through the catalog object hierarchy:

OlapTable -> Partition -> MaterializedIndex -> Tablet -> Replica

They were not added as a dedicated backup metadata object.

A later restore enhancement, #11942, introduced the reserve_replica property. That feature preserves the original replication allocation when requested, but it does not require preserving individual serialized Replica objects. During restore, Doris still recreates tablets and replicas with new ids and selected backend ids.

Background

Large BACKUP jobs can generate very large backup metadata because Doris currently persists all Replica objects for every backed up tablet. For tables with many tablets and replicas, this increases both backup metadata size and FE memory usage.

Root Cause

Backup metadata currently stores a detached copy of the full table metadata. Since Tablet metadata contains Replica objects, all replicas are serialized into backup metadata as part of the table object graph.

This is unnecessary for restore. Restore only needs the table structure, partition/index/tablet topology, version information, and replica allocation. The actual Replica objects are recreated during restore with new replica ids and selected backend ids.

Changes

This PR strips replica information from the detached table copy used for backup metadata.

A mutable FE config backup_meta_reserve_replica_info is added as a fallback. The default value is false, which omits unused replica information from backup metadata. Setting it to true preserves the old behavior.

Follow-up Work

This PR only addresses replica-info amplification in backup metadata. Follow-up PRs will handle larger serialization optimizations:

  • Reduce RestoreJob JSON serialization/deserialization memory usage by streaming large RestoreJob fields such as snapshotInfos and restoredVersionInfo.
  • Further reduce BACKUP job memory usage by introducing streaming JSON write/read paths for large backup metadata.

Compatibility

Restore compatibility is kept because RESTORE does not depend on serialized Replica objects. The previous behavior can still be restored with backup_meta_reserve_replica_info.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#63

Problem Summary: Backup meta currently persists every Replica object of each backed up tablet. For large tables this amplifies the metadata size and FE memory footprint, while restore does not use those replica objects because replicas are recreated from restore replica allocation and selected backends. Add a mutable config to keep the old behavior when needed, and strip replica info from detached backup table copies by default.

### Release note

Reduce FE memory usage and backup meta size for BACKUP jobs by omitting unused replica information from backup metadata by default.

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.catalog.OlapTableTest#testSelectiveCopyBackupReplicaStripping
    - git diff --check
- Behavior changed: Yes (backup meta no longer persists replica info by default; controlled by backup_meta_reserve_replica_info)
- Does this need documentation: No
@Ryan19929 Ryan19929 requested a review from gavinchou as a code owner July 7, 2026 09:08
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Ryan19929

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 60.00% (6/10) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants