Make some DataONE APIs into transactions in Metacat

[taojing2002]

Recently we got some reports that the Metacat instances only had partial records on objects. It is warning sign that uploading objects to Metacat is not a transaction.

Create, update, and delete are the methods which should be transactions - they should either fully succeed or rollback properly.

As we move Metacat instances to k8s , we should expect interruptions during those procedures more frequently.

So we need to design a system to do this job.

[mbjones]

@artntek @taojing2002 one way to determine if this will be a real issue on k8s deployments would be to develop a test for CRUD API operations that take some time and force a pod evictions at key points in the middle of what should be an atomic operation. For that to happen, we might need to test with a long-running operation, such as a large data upload. During that "transaction", we could then use the k8s Eviction API to evict the metacat pod and make k8s spin it up on another node. This is what would happen if, for example, a hardware failure happens or if Nick cordons and drains a node for maintenance. The example in the docs they give for the API call is:

curl -v -H 'Content-type: application/json' https://your-cluster-api-endpoint.example/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json

We should be able to automate that in a test, albeit with some complications due to authentication against a k8s cluster -- its a complicated integration test. Maybe its sufficient for us to just run this run this test manually at first to gauge if there are even any issues here to be addressed.

[taojing2002]

In this ticket: we found:

• the metacat code saves the metadata to Hazelcast (which, in turn, saves it to the database systemmetadata table), BEFORE checking the accessPolicy fields.
• the Chrome bug causes the accessPolicy section to be messed up Cannot submit or edit datasets from Chrome version 120.0.6099.71 metacatui#2235 (comment), which in turn causes metacat to throw an exception at this point (...InvalidSystemMetadata: The Permission shouldn't be null....), and fails to save the data object.
• Because the above steps are not part of a transaction, we now have the metadata saved, but not the data.

Matthew commented:
actually, on develop, this code needing a transaction has now moved to SystemMetadataManager.java -> updateSystemMetadata()

Matthew and I discussed the issue - now the code is fine.