New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… #2451
Conversation
…hen hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.meta.replica.replication to target a different endpoint implementation). Set hbase.region.meta.replica.replication.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The HBaseMetaReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the HBaseMetaReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the HBaseMetaReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so HBaseMetaReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an HBaseMetaReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALEdit.java Add an isFlushMarker utility. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseMetaReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseMetaReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests.
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
…hen hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.meta.replica.replication to target a different endpoint implementation). Set hbase.region.meta.replica.replication.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The HBaseMetaReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the HBaseMetaReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the HBaseMetaReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so HBaseMetaReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an HBaseMetaReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALEdit.java Add an isFlushMarker utility. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseMetaReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseMetaReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests.
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
Start reading, will be back. |
CatalogReplicationSource
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
I ran the failed tests locally and they passed. Rerunning. @Apache9 Mind taking a look sir? This the direction you suggested? |
🎊 +1 overall
This message was automatically generated. |
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
Outdated
Show resolved
Hide resolved
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
@huaxiangsun I added HBASE-25145 "WALReader quits if nothing to replicate (and won't restart)" to look at the issue you found above sir. |
🎊 +1 overall
This message was automatically generated. |
Good. |
Great! I also confused a bit as I thought most of queueStorage access is from Source.
Thanks for looking into the details! Yeah, we can do refactoring later. |
Yeah, that is what I read from the code. A follow-on is good as we can create a unitest to show if the issue really exists.
Agreed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we are on a feature branch so I think it is fine to go. Can polish later.
* verifying async wal replication replays the edits to the secondary region in various scenarios. | ||
* @see TestRegionReplicaReplicationEndpoint | ||
*/ | ||
@Category({FlakeyTests.class, LargeTests.class}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why FlakeyTests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copy/Paste. Let me fix. Thanks.
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
apache#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
apache#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
#2451) * HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
…hen hbase:meta assigned to RS
Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'.
HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta
Table; hbase:meta Table could only do Phase 1 Region Replicas reading
the primary Regions' hfiles. Here we add 'async WAL Replication' to
hbase:meta so Replicas can be more current with the primary's changes.
Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates
all edits to the configured in-cluster endpoint (Defaults to the
RegionReadReplicaEndpoint.class -- set hbase.region.meta.replica.replication to
target a different endpoint implementation).
Set hbase.region.meta.replica.replication.enabled to enable async WAL
Replication for hbase:meta region replicas. Its off by default.
The HBaseMetaReplicationSource for async WAL Replication of hbase:meta does
NOT need to keep up WAL offset or a queue of WALs-to-replicate in the
replication queue store as is done in other ReplicationSource implementations;
the HBaseMetaReplicationSource is for Region Replicas only. General
Replication does not replicate hbase:meta. hbase:meta Region Replicas reset
on crash of the primary replica so there is no need to 'recover'
replication that was running on the crashed server.
Because it so different in operation, the HBaseMetaReplicationSource is bolted
on to the side of the ReplicationSourceManager. It is lazily
instantiated to match the lazy instantiation of the hbase:meta
WALProvider, created and started on the open of the first Region of an
hbase:meta table. Thereafter it stays up till the process dies, even if
all hbase:meta Regions have moved off the server, in case a hbase:meta
Region is moved back (Doing this latter simplifies the implementation)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Read configuration to see if we need to wait on setting a Region read-enabled
(if so, replicas will only flip to enable read after confirming a
flush of the primary so they for sure are a replica of a known point)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
If configured, on open of hbase:meta, ask the ReplicationSourceManager
to add a ReplicationSource (if it hasn't already).
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
Edit log message.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
If configured, on close of hbase:meta, update ReplicationSourceManager
that a source Region has closed.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
javadoc and make constructor private.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
Add logPositionAndCleanOldLogs w/ default of the old behavior so
HBaseMetaReplicationSource can bypass updating store with WAL position,
etc.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
Add creation and start of an HBaseMetaReplicationSource.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
Go via ReplicationSource when calling logPostionAndCleanOldLogs.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
Javadoc.
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
Add utility for reading configurations for hbase:meta region replicas.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALEdit.java
Add an isFlushMarker utility.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
Javadoc.
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
Use define.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseMetaReplicationSource.java
Specical version of ReplicationSource for Region Replicas on
hbase:meta.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseMetaReplicationSourcePeer.java
Needs a special peer too (peers are baked into replication though we
don't use 'peers' here)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java
Tests.