HDDS-11253. Handle corrupted merkle tree files

[errose28]

What changes were proposed in this pull request?

Summary

A merkle tree file may be corrupted during write or by some other error. This should only be a temporary problem since the scanner will rewrite it on its next run. Until that happens failures should be handled accordingly:

• Clients calling the read API should be able to cleanly fail on a corrupted file. The server does not deserialize it before sending.
• Any operation that updates the file should regenerate a local checksum file if it is corrupted, as if it does not exist.
• File write can be done by renaming a temp file into place to avoid corruption from partial writes. The file should either be present or absent if a write fails, but not partially written.

Details

• When a checksum file is corrupted, we overwrite it with a new copy. This will lose any block delete information present in the file.
  • This is not a problem because the merge algorithm that will be implemented in HDDS-10928 will pick up this information again from other replicas.
  • If no other replicas have the delete information then all replicas match and there is no problem.
• When a checksum file does not exist on the datanode, an error is returned to the client. This prevents accidental errors that may occur from using a default value protobuf or null value instead.
• The read checksum API now automatically fails on unclosed containers. These containers will never have the files generated.
• Exception handling for the current read API was handled incorrectly since the request was not marked readonly in HddsUtils#isReadOnly. Tests in this PR caught the issue and it is now fixed.
• The API was renamed from getContainerMerkleTree to getContainerChecksumInfo to match the proto value returned.
  • We have protos called ContainerMerkleTree and ContainerChecksumInfo. ContainerChecksumInfo contains a ContainerMerkleTree along with other metadata.
  • The API was called getContaienrMerkleTree but returned a Bytestring that actually deserialized into a ContainerChecksumInfo, which was unintuitive. Calling getContainerMerkleTree and deserializing as a ContainerMerkleTree would fail.
  • Since the API needs to return a ContainerChecksumInfo to capture all information about the container (the merkle tree alone will not suffice), it is intuitive to rename the API getContainerChecksumInfo.

What is the link to the Apache JIRA

HDDS-11253

How was this patch tested?

• Unit tests added for handling file corruption within the datanode.
• Integration tests added to test error propagation from server to client when reading the file.

[aswinshakil]

Thanks for working on this @errose28. I have a minor nit, apart from that I'm good with the changes.

[kerneltime]

First round.

[kerneltime]

Nit: Simpler signature than passing a massive object.

Suggested change

	public static File getTmpContainerChecksumFile(ContainerData data) {
	public static File getTmpContainerChecksumFile(String path, long containerID) {

[errose28]

This signature matches getContainerChecksumFile for consistency. Changing them both as this suggests would require the caller to know more internal details like where in the container the file(s) belong and whether the path is relative to the container root or not. Also in practice callers already have ContainerData to refer to the container in the first place so this ends up being less verbose because callers do not need to unpack the object themselves.

[errose28]

All should be addressed in the latest commits.

[errose28]

All tests are expected to pass after the rebase. TestContainerCommandReconciliation is passing locally for me after the updates. Waiting for CI on my fork before taking this out of draft.