HDDS-10200. OM may terminate due to NPE in S3SecretValue proto conversion

[adoroszlai]

What changes were proposed in this pull request?

Revoked S3 secret is stored in the cache as S3SecretValue with null secret value. Since the object is mutable, other thread could access the same instance before revocation, and might encounter NPE while trying to convert to protobuf.

2024-01-24 09:49:09,063 [om1-OMDoubleBufferFlushThread] ERROR ratis.OzoneManagerDoubleBuffer (ExitUtils.java:terminate(133)) - Terminating with exit status 2: During flush to DB encountered error in OMDoubleBuffer flush thread om1-OMDoubleBufferFlushThread
org.apache.ratis.util.ExitUtils$ExitException: During flush to DB encountered error in OMDoubleBuffer flush thread om1-OMDoubleBufferFlushThread when handling OMRequest: cmdType: SetS3Secret
traceID: ""
success: true
status: OK
SetS3SecretResponse {
  accessId: "testUser1accessId1"
  secretKey: "testsecret"
}

	at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:141)
	at org.apache.ratis.util.ExitUtils.terminate(ExitUtils.java:151)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.terminate(OzoneManagerDoubleBuffer.java:559)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:413)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:345)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:319)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:286)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$S3Secret$Builder.setAwsSecret(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ozone.om.helpers.S3SecretValue.getProtobuf(S3SecretValue.java:100)
	at org.apache.hadoop.hdds.utils.db.DelegatedCodec.toCodecBuffer(DelegatedCodec.java:85)
	at org.apache.hadoop.hdds.utils.db.Codec.toDirectCodecBuffer(Codec.java:73)
	at org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:171)
	at org.apache.hadoop.ozone.om.OmMetadataManagerImpl$1.addWithBatch(OmMetadataManagerImpl.java:1949)
	at org.apache.hadoop.ozone.om.response.s3.security.OMSetSecretResponse.addToDBBatch(OMSetSecretResponse.java:80)
	at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:66)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$9(OzoneManagerDoubleBuffer.java:407)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:244)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:406)
	... 4 more

was seen in TestMultiTenantVolume (4 / 1000 runs).

This PR changes S3SecretValue to be immutable.

https://issues.apache.org/jira/browse/HDDS-10200

How was this patch tested?

TestMultiTenantVolume passed 10x100:
https://github.com/adoroszlai/ozone/actions/runs/7645630287

Regular CI:
https://github.com/adoroszlai/ozone/actions/runs/7645622076

[adoroszlai]

@ArafatKhan2198 can you please also take a look?

[myskov]

LGTM

[sadanand48]

Thanks @adoroszlai the patch, LGTM

[szetszwo]

@adoroszlai , thanks for working on this! Just a few minor comments inlined.

[szetszwo]

An unrelated comment: It is not a good idea store a secret in a plaintext String and write it to db.

The usual practice is to encrypt it in byte[] and erase it after use.

[szetszwo]

+1 the change looks good.

[adoroszlai]

Thanks @myskov, @sadanand48, @szetszwo for the review.