-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-27698 Migrate meta locations from zookeeper to master data may … #5167
base: branch-2
Are you sure you want to change the base?
Conversation
…not always possible if we migrate from 1.x HBase
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
The test case failures are not related to this change. @taklwu could you please review the change. Thanks. |
Why change from throwing an exception to returning false? |
@Apache9 Here I have mentioned how the upgrade went smooth with my patch. |
Thank you @chrajeshbabu. It seems the changes make sense from the upgrade viewpoint. On the other hand, if this rare scenario were to happen on a healthy 2.x cluster, does this change make any difference?
If this happens on 2.x cluster, what would now be different? Only that meta would be assigned on any random server, correct? |
I have tried two scenarios in healthy 2.x cluster with the change
By throwing exception also we need to delete master data because of failed init meta procedure not bring the master up and need to follow one of these steps
|
Got it, thanks. This makes sense, +1 from my side. |
@Apache9 does this look good to you? |
@Apache9 will commit it if it's fine for you. Could you please confirm. |
I still need to check the code. This is a very critical part, if we run InitMetaProcedure when meta exists, it could cause serious data loss problem... We should try to prevent scheduling the InitMetaProcedure, instead of just letting it go and hoping it would work... |
After chekcing the code, I think the correct way for fixing this problem is to do the work in HMaster.tryMigrateMetaLocationsFromZooKeeper. We can some more code in this method, to check whether the meta directory is already there and if it is, if we can make sure that this is an upgrading from 1.x(by trying to read something on the filesystem? Is this possible?), then insert a record to the master region so we can skip scheduling the InitMetaProcedure. |
we can check the column families in meta table to detect whether it's from 1.x or current versions. |
And do we have read replica support on 1.x? Do we also need to insert the record for secondary replicas? |
We can insert record but we may not know the state and location which leads to InitMetaProcedure again. |
Checked the code, the condition for whether to schedule an InitMetaProcedure is
So I think it only needs we have a record in master region for meta, and do not need to know its state and location? And we will first try to migrate from zookeeper, and for 1.x, if the cluster shutdown gracefully, there is no znode for meta, then we can enter the logic described above, so in this case, I think the state for meta should be CLOSED? |
That's correct let me check and update the patch accordingly. |
@chrajeshbabu but isn't meta state unknown in your case of migration from 1.x to 2.x? Is the plan to read meta CF from filesystem directly as part of |
@Apache9 I have tried your suggestion of creating put entry of meta table in master region which gets filled in regionstates and helps to avoid calling init meta procedure there is problem with this approach. Here is the code I have tried.
|
Better post the full patch somewhere so I can check it? And maybe we need to manually schedule a TRSP to bring meta region online in this case... |
…not always possible if we migrate from 1.x HBase