-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-27085 Create REPLICATION_SINK_TRACKER table to persist marker rows coming from source cluster #4486
HBASE-27085 Create REPLICATION_SINK_TRACKER table to persist marker rows coming from source cluster #4486
Conversation
… rows coming from source cluster
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
jdk8 tests succeeds locally.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left few nits, looks good overall
boolean replicationSinkTrackerEnabled = conf.getBoolean(REPLICATION_SINK_TRACKER_ENABLED_KEY, | ||
REPLICATION_SINK_TRACKER_ENABLED_DEFAULT); | ||
if (!replicationSinkTrackerEnabled) { | ||
LOG.info("replication sink tracker requests logging to system table " |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about DEBUG log and with placeholder {}
?
return; | ||
} | ||
if (!masterServices.getTableDescriptors().exists(REPLICATION_SINK_TRACKER_TABLE_NAME)) { | ||
LOG.info(REPLICATION_SINK_TRACKER_TABLE_NAME_STR + " table not found. Creating."); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: we can use placeholder
Thank you @virajjasani for the review feedback. Addressed couple of comments in the latest patch. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
I think this is good to go soon as positive QA results are posted. |
The failed test succeeds locally
|
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
1 similar comment
💔 -1 overall
This message was automatically generated. |
Oops, I guess the rebase coincided with the ongoing build. |
Just this last one I guess and we should be good https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4486/10/ |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
@shahrs87 could you once pull in latest changes from |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
3ccbd47
into
apache:HBASE-26913-replication-observability-framework
Changes done in this PR.
ReplicationMarkerChore
. This chore will create a marker wal edit with family asMETAFAMILY
and column qualifier asHBASE::REPLICATION_MARKER
(newly introduced).This chore is enabled by
hbase.regionserver.replication.marker.enabled
config key (default to false). When enabled, it will create marker rows everyhbase.regionserver.replication.marker.chore.duration
seconds (default to 30 seconds).REPLICATION.SINK_TRACKER
with replication scope 0. This table is NOT replicated. Introduced config keyhbase.regionserver.replication.sink.tracker.enabled
to determine if sink cluster is configured to accept marker rows from peer cluster.ReplicationSourceWALActionListener
andReplicationSourceWALReader
), sink side replication code (ReplicationSink
).ReplicationSourceWALReader
will process all the edits and determine if it is a marker row. If it is found to be a marker row, it will update the edit with region server name, wal name, offset within current wal and replicate the cell.ReplicationSink
on the sink side will handle this special marker row. When sink encounters this marker row, it will de-serialize ReplicationMarkerDescriptor object and convert it to Put request with table REPLICATION.SINK_TRACKER and the above columns.Introduced 2 new config keys which will control the following:
hbase.regionserver.replication.marker.enabled
(default to false): This will create ReplicationMarkerChore on the region server. If not enabled, it will NOT create any marker rows.hbase.regionserver.replication.sink.tracker.enabled
(default to false): If enabled, regionserver acting as a sink cluster will create REPLICATION.SINK_TRACKER table and will process the special marker rows. If not enabled, it will not create REPLICATION.SINK_TRACKER table and IGNORE the special marker rows coming from source cluster which has ReplicationMarkerChore enabled.@virajjasani Please review.