Bootup cookie validation considers an empty journal to signify a new bookie

[ivankelly]

This is wrong, and can lead to split brain.

From the dev list:
We check all ledger, journal and index directories for a cookie. If it
doesn't exist, it gets added to a missingCookieDirs list. We then
iterate over this directory. If any directory in missingCookieDirs
isn't listed as a ledger directory in the journal dir cookies, or
isn't empty, we fail to start.

The issue is that a journal dir could be emptied and we wouldn't
detect it. It would be great if someone else could eyeball the code
and tell me I'm wrong. The code is in Bookie#checkEnvironment.

This breaks correctness. Imagine we have a ledger on b1, b2, b3.
Writer w1 is writing to the ledger.
The state of the ledger on the bookies is:

b1: e0 Fenced: false, LAC: -
b2: e0 Fenced: false, LAC: -
b3: e0 Fenced: false, LAC: -

w1 gets partitioned from network. w2 tries to recover the ledger, it
tries to fence on all bookies. The message to b3 gets lost. b1 and b2
acknowledge the fencing, so w2 continues to recover and close the
ledger with e0 as the last entry.

b1: e0 Fenced: true, LAC: e0
b2: e0 Fenced: true, LAC: e0
b3: e0 Fenced: false, LAC: -

If w1 became unpartitioned at this point, it wouldn't be able to add a
new entry to the ledger as any quorum would see fenced on b1 or b2.

However, imagine that the fenced message is only in the journal on b2,
b2 crashes, something wipes the journal directory and then b2 comes
back up. The new state of the ledger on the bookies will be.

b1: e0 Fenced: true, LAC: e0
b2: e0 Fenced: false, LAC: -
b3: e0 Fenced: false, LAC: -

Now w1 can write a new entry, e1, and b2 & b3 would both acknowledge
it, even though the end of the ledger is e0.