ZOOKEEPER-3306: Fixing node not accessible issue due the inconsistent ACL reference map after SNAP sync

[lvfangmin]

Please check the Jira for more details of the bug.

Need more opinion on this fix to see if there is other corner cases or side effect we didn't handle properly.

[anmolnar]

@lvfangmin Huh, this is not easy. Another edge case captured by FB. :)
The fix seems fine to me, but not sure about the implications. You essentially touch the ACL to prevent garbage collecting it by incrementing the ref counter, right?

[lvfangmin]

@anmolnar currently, we cannot tell if the ACL reference is already added or not during fuzzy snapshot sync and replaying a txn for a already exist node, this code change may count twice, but it's not a big deal since this is a rare case only happen during SNAP syncing, and we'll clean the ACL map if there is no node using it when deserializing the DataTree from disk.

[lvfangmin]

We've released this fix onto prod for more than 2 weeks, haven't hit this corner case so far, but it says this is a really corner case which we may not even happen in 2 weeks before this fix.

But this should be safe enough to merge in master. @anmolnar would you like to take another look?

[anmolnar]

@lvfangmin Where is the point in the code where we clear the ACL cache eventually?

[lvfangmin]

@anmolnar it's here: https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java#L1290.

[anmolnar]

+1 lgtm

[anmolnar]

Committed to master. Thanks @lvfangmin !