Issue #2305: Bookies should not be allowed to come up without a cookie in metadata store.

[Ghatage]

Motivation

Currently we allow bookies to come up even if they don't have a cookie in the metadata store.
The cookie is an identity of a registered bookie and a bookie should not be allowed to come up without it.

Changes

Update handling of bookie boot up. Add test.
This scenario is rarely expected but can happen in case of corruption or errant deletion of znodes for cookies

[Ghatage]

Fix for #2305.
@eolivelli

[eolivelli]

LGTM

[eolivelli]

Please check CI

[Ghatage]

Hi @eolivelli,
Thanks for the review, I fixed the errors.
Now it seems like a Github Actions error in the Integration tests.

[ERROR] The command '/bin/sh -c /opt/bookkeeper/scripts/install-python-client.sh' returned a non-zero code: 1
[ERROR] The command '/bin/sh -c /opt/bookkeeper/scripts/install-python-client.sh' returned a non-zero code: 1
[ERROR] Failed to execute goal com.spotify:dockerfile-maven-plugin:1.4.13:build (default) on project current-version-image: Could not build image: The command '/bin/sh -c /opt/bookkeeper/scripts/install-python-client.sh' returned a non-zero code: 1 -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :current-version-image
##[error]Process completed with exit code 1.

[merlimat]

My concern here is that currently bookie is relying on this scenario for valid cases.

For example, the cookie is first written on local disk and then on ZK. If there's a failure in between, the bookie will currently be able to recover on next restart.

After this change, wouldn't this require manual intervention to fix?

[Ghatage]

Hi @merlimat!

Yes you are right. If there is a failure in between writing the cookie to the disk and then to ZK, the bookie won't be able to restart as it will now throw an InvalidCookieException.

My concern here is that currently bookie is relying on this scenario for valid cases.

If there is no cookie in ZK, any operations which involve reading the cookie from ZK, like decommissioning a bookie via decommission command or recovering a bookie via recover command will throw NPE / fail

[Ghatage]

@ivankelly Any comments?

[eolivelli]

@ivankelly @merlimat @jiazhai @sijie PTAL