HBASE-28957: Adding support for continuous Backup and Point-in-Time Recovery

[anmolnar]

Summary

This PR introduces Continuous Backup and Point-in-Time Recovery (PITR) support in HBase.
The feature enables continuous WAL archival and recovery to any specific point in time, providing stronger data protection and recovery flexibility.

Key Highlights

• Enables continuous backup instead of batch-based incremental backups.
• Supports precise point-in-time recovery to mitigate data loss due to corruption or accidental updates.
• Simplifies WAL management and backup lifecycle handling.
• Improves reliability and reduces storage overhead on the source cluster.

For detailed design and implementation, refer to the design document:
[Continuous Backup and Point-in-Time Recovery]

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 31s		master passed
+1 💚	compile	8m 48s		master passed
+1 💚	checkstyle	2m 21s		master passed
+1 💚	spotbugs	12m 34s		master passed
+1 💚	spotless	0m 55s		branch has no errors when running spotless:check.
-0 ⚠️	patch	1m 25s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 10s		the patch passed
+1 💚	compile	8m 45s		the patch passed
+1 💚	cc	8m 45s		the patch passed
-0 ⚠️	javac	8m 45s	/results-compile-javac-root.txt	root generated 12 new + 1896 unchanged - 0 fixed = 1908 total (was 1896)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 22s	/results-checkstyle-root.txt	root: The patch generated 50 new + 3 unchanged - 0 fixed = 53 total (was 3)
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	spotbugs	13m 8s		the patch passed
+1 💚	hadoopcheck	12m 53s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	hbaseprotoc	4m 54s		the patch passed
+1 💚	spotless	0m 57s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		87m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7445/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7445
JIRA Issue	HBASE-28957
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac xmllint hadoopcheck compile spotbugs checkstyle hbaseanti cc buflint bufcompat hbaseprotoc
uname	Linux a4010f884649 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 6aa212f
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	191 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-server hbase-mapreduce hbase-backup . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7445/2/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 31s		master passed
+1 💚	compile	2m 14s		master passed
+1 💚	javadoc	2m 58s		master passed
+1 💚	shadedjars	6m 17s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	6m 47s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 5s		the patch passed
+1 💚	compile	2m 12s		the patch passed
+1 💚	javac	2m 12s		the patch passed
-0 ⚠️	javadoc	0m 13s	/results-javadoc-javadoc-hbase-backup.txt	hbase-backup generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
-0 ⚠️	javadoc	1m 56s	/results-javadoc-javadoc-root.txt	root generated 4 new + 210 unchanged - 0 fixed = 214 total (was 210)
+1 💚	shadedjars	6m 8s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	290m 6s		root in the patch passed.
		329m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7445/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7445
JIRA Issue	HBASE-28957
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 488b63b537ab 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 6aa212f
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7445/2/testReport/
Max. process+thread count	7246 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-server hbase-mapreduce hbase-backup . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7445/2/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache9]

I still do not think we need to introduce this new class...

The current implementation will update the wal position every time when we finish sending out a batch, in general I think this could be changed, FWIW, we can increase the batch to reduce the time we persist the log position, so it is not a big problem to not always persist the position.

In this way, we could add a callback when we actually want to persist the log position, to commit the data to external storage. I think in this way the logic more clear.

[anmolnar]

I still do not think we need to introduce this new class...

The current implementation will update the wal position every time when we finish sending out a batch, in general I think this could be changed, FWIW, we can increase the batch to reduce the time we persist the log position, so it is not a big problem to not always persist the position.

I'm not sure I understand this. We're streaming the data to the S3 object, but we don't want to persist the WAL position every time a batch is shipped, only when we close the stream and open a new one.

In order to persist WAL position every time a batch is shipped, the batch size must be equal to desired S3 object size.

In this way, we could add a callback when we actually want to persist the log position, to commit the data to external storage. I think in this way the logic more clear.

I think this could work. The callback is passed to the shipper which will call it when it wants to persist. @vinayakphegde @ankitsol wdyt?

[Apache9]

What I mean is that, now our mechanism is to persist the log position after every batch right? But since we can change the batch size to tune the frequency of persisting log position, so logically there is no hard limit that we 'must' persist the log position after every batch.

I think we can change the condition on whether we need to log position.

1. Every time or a fixed number of times(a number)
2. Uncommitted size
3. Time after last commit
4. Every time when moving to a new file(a flag)

This can be a more general framework and do not need to introduce more concept.

WDYT?

[vinayakphegde]

Hi @Apache9, if I understand correctly what you're saying, taking your first option:

Every time or a fixed number of times (a number)

I assume that if the fixed number is 10, for the first 9 times we push the batch, the endpoint consumes it and returns true, but we don’t move the offset. On the 10th time, the endpoint should mandatorily persist the file and then return true. Once the endpoint returns true, we assume it has persisted, and we move the offset.
With this approach, we can stick to using only true/false without needing to pass the replication source to the endpoint for persistence.

We’ll take a look into this and see if there could be any issues with this approach. Thanks!

[wchevreuil]

Sorry, as this is quite a large PR, I have a few (maybe dumb) questions about it:

1. Is this a per-table configuration?
2. Do we need to define a specific replication peer for this?
3. If we have normal replication enabled (a normal replication peer), will all CFs with REPLICATION_SCOPE = 1 also be targeted for this PITR?
4. Do we keep pitr wal files indefinitely?
5. Should the pitr endpoint replicate method consider if the underlying FS supports sync before deciding the result to return?
6. Does it support pause (disable_peer)?
7. Can we include doc changes within this PR?

[kgeisz]

@wchevreuil I have answers to some of your questions:

1. Is this a per-table configuration?

Backups need to be enabled in the HBase conf file via hbase.backup.enable. For continuous backups, a backup WAL directory needs to be set via hbase.backup.continuous.wal.dir. Backups can be done on a per-table basis.

2. Do we need to define a specific replication peer for this?

A replication peer is defined by default as continuous_backup_replication_peer using the CONTINUOUS_BACKUP_REPLICATION_PEER constant.

4. Do we keep pitr wal files indefinitely?

No, WALs outside of the PITR window get cleaned up automatically. They are also removed when the backup is deleted. This is tracked in HBASE-28992

7. Can we include doc changes within this PR?

I believe that is the plan. It's still a work in progress.

[anmolnar]

3. If we have normal replication enabled (a normal replication peer), will all CFs with REPLICATION_SCOPE = 1 also be targeted for this PITR?

Negative. Continuous backup needs a special replication endpoint to be set for the peer ContinuousBackupReplicationEndpoint. Normal replication peers are not affected and work fine side-by-side.

[anmolnar]

5. Should the pitr endpoint replicate method consider if the underlying FS supports sync before deciding the result to return?

The PITR endpoint assumes that sync() is not supported by the filesystem which is usually the case with object stores like S3. The WAL pointer gets persisted only when the stream is closed.

[anmolnar]

6. Does it support pause (disable_peer)?

This is a very good question. Given that it's a standard replication endpoint, I would say sure, why not. This is something that we should validate sooner rather than later..

[anmolnar]

7. Can we include doc changes within this PR?

I suggest to finish reviewing and merging this patch first. Proper documentation and integration testing should come afterwards in separate PRs.

[vinayakphegde]

6. Does it support pause (disable_peer)?

This is a very good question. Given that it's a standard replication endpoint, I would say sure, why not. This is something that we should validate sooner rather than later..

@anmolnar @wchevreuil We actually use that. When cleaning up WAL files, if all the files are removed (which happens when deleting the last full backup — the bare minimum required for PITR), we disable the peer.

More details: BackupCommands.java#L900

[wchevreuil]

3. If we have normal replication enabled (a normal replication peer), will all CFs with REPLICATION_SCOPE = 1 also be targeted for this PITR?

Negative. Continuous backup needs a special replication endpoint to be set for the peer ContinuousBackupReplicationEndpoint. Normal replication peers are not affected and work fine side-by-side.

How does the PITR Endpoint avoids backing up entries for CFs with REPLICATION_SCOPE set to true (for normal replication), but which are not primarily wanted to be in PITR backups? Is it possible to "opt out" from PITR at CF table/level?

[wchevreuil]

@wchevreuil I have answers to some of your questions:

1. Is this a per-table configuration?

Backups need to be enabled in the HBase conf file via hbase.backup.enable. For continuous backups, a backup WAL directory needs to be set via hbase.backup.continuous.wal.dir. Backups can be done on a per-table basis.

Will this trigger backups on all existing tables?

2. Do we need to define a specific replication peer for this?

A replication peer is defined by default as continuous_backup_replication_peer using the CONTINUOUS_BACKUP_REPLICATION_PEER constant.

Can you elaborate further how this peer is defined?

4. Do we keep pitr wal files indefinitely?

No, WALs outside of the PITR window get cleaned up automatically. They are also removed when the backup is deleted. This is tracked in HBASE-28992

Ok, so we can only do actually PITR within that window period. Can you point me to the code in this PR where this is implemented?

[vinayakphegde]

3. If we have normal replication enabled (a normal replication peer), will all CFs with REPLICATION_SCOPE = 1 also be targeted for this PITR?

Negative. Continuous backup needs a special replication endpoint to be set for the peer ContinuousBackupReplicationEndpoint. Normal replication peers are not affected and work fine side-by-side.

How does the PITR Endpoint avoids backing up entries for CFs with REPLICATION_SCOPE set to true (for normal replication), but which are not primarily wanted to be in PITR backups? Is it possible to "opt out" from PITR at CF table/level?

@wchevreuil That is determined when starting the continuous backup.

For example, if you want to start continuous backup for table1, you initiate a full backup with the continuous backup option enabled. This performs the full backup and adds the table to the continuous_backup_replication_peer (the default replication peer used for continuous backup).

If the column families don’t already have replication scope enabled, it’s also set to true at this stage.
Please refer to the following methods for details:

hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/FullTableBackupClient.java

Line 269 in 6aa212f

private long startContinuousWALBackup(Admin admin) throws IOException {

enableTableReplication()
updateContinuousBackupReplicationPeer()
addContinuousBackupReplicationPeer()

So, if you manually change the replication scope to true for a random table, it will not be replicated as part of the continuous backup, since it’s not included in the continuous_backup_replication_peer.

[vinayakphegde]

Can you elaborate further how this peer is defined?

hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/FullTableBackupClient.java

Line 269 in 6aa212f

private long startContinuousWALBackup(Admin admin) throws IOException {

this contains the whole logic.

Ok, so we can only do actually PITR within that window period. Can you point me to the code in this PR where this is implemented?

here is the PITR logic is defined. validateAndRestore() method.

hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/AbstractPitrRestoreHandler.java

Line 68 in 6aa212f

public abstract class AbstractPitrRestoreHandler {

And this is where the clean-up logic is defined — how backed-up WALs are cleaned up as part of continuous backup.

hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupCommands.java

Line 900 in 6aa212f

private void cleanUpUnusedBackupWALs() throws IOException {

[kgeisz]

@wchevreuil I have answers to some of your questions:

1. Is this a per-table configuration?

Backups need to be enabled in the HBase conf file via hbase.backup.enable. For continuous backups, a backup WAL directory needs to be set via hbase.backup.continuous.wal.dir. Backups can be done on a per-table basis.

Will this trigger backups on all existing tables?

@wchevreuil You trigger a backup by running the hbase backup create full command on the desired table(s).
For example: hbase backup create full table1,table2,...,tableN

[wchevreuil]

@wchevreuil I have answers to some of your questions:

1. Is this a per-table configuration?

Backups need to be enabled in the HBase conf file via hbase.backup.enable. For continuous backups, a backup WAL directory needs to be set via hbase.backup.continuous.wal.dir. Backups can be done on a per-table basis.

Will this trigger backups on all existing tables?

@wchevreuil You trigger a backup by running the hbase backup create full command on the desired table(s). For example: hbase backup create full table1,table2,...,tableN

Ok, so only after a full backup is created on a given table through the described command is when that table is added to the pitr replication peer. Is there any action that may remove a given table from the list of tables tracked by the pitr replication peer?

[kgeisz]

Is there any action that may remove a given table from the list of tables tracked by the pitr replication peer?

@wchevreuil The only action I can think of is forcefully deleting the full backup (i.e. using --force-delete), as that is the foundation of the backup and thus the PITR. If the last full backup for the table is forcefully deleted, then all of the continuous backup metadata, associated WAL files, and the replication peer are cleaned up. If the user tries to delete the final full backup for that table without using --force-delete, then the delete attempt fails since that particular backup is important for the backup's integrity.

@vinayakphegde @ankitsol @anmolnar Can one of you please confirm?

[vinayakphegde]

Is there any action that may remove a given table from the list of tables tracked by the pitr replication peer?

@wchevreuil The only action I can think of is forcefully deleting the full backup (i.e. using --force-delete), as that is the foundation of the backup and thus the PITR. If the last full backup for the table is forcefully deleted, then all of the continuous backup metadata, associated WAL files, and the replication peer are cleaned up. If the user tries to delete the final full backup for that table without using --force-delete, then the delete attempt fails since that particular backup is important for the backup's integrity.

@vinayakphegde @ankitsol @anmolnar Can one of you please confirm?

Correct. related code:

hbase/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupCommands.java

Line 900 in 6aa212f

private void cleanUpUnusedBackupWALs() throws IOException {

[wchevreuil]

LGTM, all my questions have been addressed.

Regarding the wal tracking approach, I don't have a strong opinion. @Apache9 , do you think this is a blocker for this merge? Or an alternative could be worked as a follow-up PR?

[anmolnar]

LGTM, all my questions have been addressed.

Regarding the wal tracking approach, I don't have a strong opinion. @Apache9 , do you think this is a blocker for this merge? Or an alternative could be worked as a follow-up PR?

We're working on a pull request currently which would give us some insight on the alternative approach and lets us decide which way to move forward. Hopefully will be ready in a few days.

[Apache9]

I suggest we try the approach to not introduce the new 'ReplicationResult' concept to see if it could work before merging this back. Once the new concept is there, it will be difficult to remove it in the future, as later code may rely on it too...