Utility to rebuild interleaved storage index files #1642
Conversation
We came across a case where the a ledger had been deleted from zookeeper accidently. It was possible to recover the ledger metadata from the zookeeper journal and old snapshots, but the bookies had deleted the indices by this time. However, even if the index is deleted, the data still exists in the entrylog. This utility scans the entrylog to rebuild the index, thereby making the ledger available again.
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
Outdated
Show resolved
Hide resolved
...tion/smoke/src/test/java/org/apache/bookkeeper/tests/integration/TestBookieShellCluster.java
Outdated
Show resolved
Hide resolved
...tion/smoke/src/test/java/org/apache/bookkeeper/tests/integration/TestBookieShellCluster.java
Outdated
Show resolved
Hide resolved
...tion/smoke/src/test/java/org/apache/bookkeeper/tests/integration/TestBookieShellCluster.java
Outdated
Show resolved
Hide resolved
...tion/smoke/src/test/java/org/apache/bookkeeper/tests/integration/TestBookieShellCluster.java
Outdated
Show resolved
Hide resolved
...r-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedStorageRegenerateIndexOp.java
Show resolved
Hide resolved
...r-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedStorageRegenerateIndexOp.java
Outdated
Show resolved
Hide resolved
...r-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedStorageRegenerateIndexOp.java
Outdated
Show resolved
Hide resolved
| if (!ledgerCache.ledgerExists(ledgerId)) { | ||
| // set dummy master key and set fenced. | ||
| // master key is only used on write operations | ||
| ledgerCache.setMasterKey(ledgerId, new byte[0]); |
There was a problem hiding this comment.
hmm. this would cause a problem, if we recover a ledger, but the writer comes in later and attempts to write entries, all the writes will be failing with auth failure. and the reader can't actually read them, right? do we have a test case to cover this negative case?
we might need to fix it or just add messages in the help messages of the tool, so people is aware of this situation.
There was a problem hiding this comment.
Hmm, yes. I had been thinking of adding it as required a passed in option.
There was a problem hiding this comment.
add messages in the help messages of the tool
I think we need to pick up your effort to revamp the the tools, and 1) move them all into the same framework, 2) add tests & 3) add docs for each one.
Right now, bookie shell is a mess. There's even multiple tools that do exactly the same thing.
There was a problem hiding this comment.
There's a little bit more hidden complexity here that I'm not sure how to resolve.
If a idx file has been deleted, but ledger metadata exists, when a client tries to open it and run recovery, it will see that it cannot read any entries, and close the ledger as empty.
I know DLog has a force option for this, but I think it would be better to provide another cli tool to allow users to mark the metadata from Closed -> In recovery, and then run the recovery again.
There was a problem hiding this comment.
It looks like we don't actually check the password on fencing, we just blindly accept what the client passes in :/
There was a problem hiding this comment.
I think we need to pick up your effort to revamp the the tools, and 1) move them all into the same framework, 2) add tests & 3) add docs for each one.
let's pick up the tasks for #1000
If a idx file has been deleted, but ledger metadata exists, when a client tries to open it and run recovery, it will see that it cannot read any entries, and close the ledger as empty.
yeah. I think this is a rare corner case that we actuallly can do nothing for it.
I know DLog has a force option for this, but I think it would be better to provide another cli tool to allow users to mark the metadata from Closed -> In recovery, and then run the recovery again.
sgtm
It looks like we don't actually check the password on fencing, we just blindly accept what the client passes in :/
not sure we really need password for fencing, since it doesn't really append entries. and I think in general password should be removed eventually.
There was a problem hiding this comment.
Hmm, yes, I'll push my other updates for now, and leave out the password stuff, since it has no effect.
I don't think we need to cherry-pick this to 4.8. I think branch-4.8 is already cut and waiting for bug fixes only, so any feature-like issues should go to 4.9. |
|
rerun bookkeeper-server replication tests |
|
rerun pr validation |
|
rerun bookkeeper-server replication tests |
We came across a case where the a ledger had been deleted from
zookeeper accidently. It was possible to recover the ledger metadata
from the zookeeper journal and old snapshots, but the bookies had
deleted the indices by this time. However, even if the index is
deleted, the data still exists in the entrylog.
This utility scans the entrylog to rebuild the index, thereby making
the ledger available again.