Globus Bug Fixes

[qqmyers]

Several issues have been discovered in testing the new Globus capabilities in prep for the MOC demo and DCM 2024 meeting:

• Remote stores:error in creating exports and Dataset getThumbnail() failure - both caused by trying to treat a Dataset storageIdentifier as a path when it isn't like a DataFile one
• Obsolete/undocumented dataverse-inaccessible setting still used in one place - replaced with the documented files-not-accessible-by-dataverse setting
• Previewers/Ext. tools showing when files are not dataverse accessible - previewers/file tools now only show if files are accessible or they require auxFileAccess (since aux files are accessible via the backing store).
• The direct and Globus upload APIs only checked the file size/quota limits after the file transfer had happened - the direct upload API now won't send signedURLs for files that are too big and the Globus API will report both the fileSizeLimit and remainingQuota to the caller. (Since Globus doesn't enforce size limits, we still have to trust that users won't try transferring files that they've been told will be rejected/not added to the dataset.)

PR is draft until the latest round of testing is done.

Which issue(s) this PR closes:

Closes #10344
Closes #10231

Special notes for your reviewer:

Suggestions on how to test this:

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

coverage: 20.585% (+0.01%) from 20.574%
when pulling 3be8cac on GlobalDataverseCommunityConsortium:globusstore
into e26fabe on IQSS:develop.

[landreev]

Testing with NESE Demo Tape pool - a managed store with files inaccessible to Dataverse - the following in the log must be coming from the calculateDetails() method in the Globus service bean, so it appears that it attempts to calculate the checksum once the upload is complete. It's not fatal - it gives up after 3 tries and sets the hash to "Not available in Dataverse", as it should. But, is it a problem that it even tries, on a store with files-not-accessible-by-dataverse?

[2024-03-04T15:32:48.367-0500] [Payara 6.2023.8] [SEVERE] [] [] [tid: _ThreadID=1770 _ThreadName=pool-217-thread-1] [timeMillis: 1709584368367] [levelValue: 1000] [[
  java.lang.NullPointerException: Cannot invoke "java.io.InputStream.read(byte[])" because "in" is null
	at edu.harvard.iq.dataverse.util.FileUtil.calculateChecksum(FileUtil.java:735)
	at edu.harvard.iq.dataverse.globus.GlobusServiceBean.calculateDetails(GlobusServiceBean.java:1138)
	at edu.harvard.iq.dataverse.globus.GlobusServiceBean.lambda$calculateDetailsAsync$11(GlobusServiceBean.java:1112)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
]]

[2024-03-04T15:32:48.368-0500] [Payara 6.2023.8] [INFO] [] [edu.harvard.iq.dataverse.globus.GlobusServiceBean] [tid: _ThreadID=1770 _ThreadName=pool-217-thread-1] [timeMillis: 1709584368368] [levelValue: 800] [[
  Cannot invoke "java.io.InputStream.read(byte[])" because "in" is null]]

[2024-03-04T15:32:53.374-0500] [Payara 6.2023.8] [SEVERE] [] [] [tid: _ThreadID=1770 _ThreadName=pool-217-thread-1] [timeMillis: 1709584373374] [levelValue: 1000] [[
  java.lang.NullPointerException: Cannot invoke "java.io.InputStream.read(byte[])" because "in" is null
	at edu.harvard.iq.dataverse.util.FileUtil.calculateChecksum(FileUtil.java:735)
	at edu.harvard.iq.dataverse.globus.GlobusServiceBean.calculateDetails(GlobusServiceBean.java:1138)
	at edu.harvard.iq.dataverse.globus.GlobusServiceBean.lambda$calculateDetailsAsync$11(GlobusServiceBean.java:1112)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
]]

[2024-03-04T15:32:53.374-0500] [Payara 6.2023.8] [INFO] [] [edu.harvard.iq.dataverse.globus.GlobusServiceBean] [tid: _ThreadID=1770 _ThreadName=pool-217-thread-1] [timeMillis: 1709584373374] [levelValue: 800] [[
  Cannot invoke "java.io.InputStream.read(byte[])" because "in" is null]]

[2024-03-04T15:32:58.379-0500] [Payara 6.2023.8] [SEVERE] [] [] [tid: _ThreadID=1770 _ThreadName=pool-217-thread-1] [timeMillis: 1709584378379] [levelValue: 1000] [[
  java.lang.NullPointerException: Cannot invoke "java.io.InputStream.read(byte[])" because "in" is null
	at edu.harvard.iq.dataverse.util.FileUtil.calculateChecksum(FileUtil.java:735)
	at edu.harvard.iq.dataverse.globus.GlobusServiceBean.calculateDetails(GlobusServiceBean.java:1138)
	at edu.harvard.iq.dataverse.globus.GlobusServiceBean.lambda$calculateDetailsAsync$11(GlobusServiceBean.java:1112)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
]]

[2024-03-04T15:32:58.379-0500] [Payara 6.2023.8] [INFO] [] [edu.harvard.iq.dataverse.globus.GlobusServiceBean] [tid: _ThreadID=1770 _ThreadName=pool-217-thread-1] [timeMillis: 1709584378379] [levelValue: 800] [[
  Cannot invoke "java.io.InputStream.read(byte[])" because "in" is null]]

[landreev]

Testing a remote managed store upload workflow, aside from the cast-to-int problem mentioned on slack, the issue persists where the transfer successfully completes on the globus side, but then dataverse fails to save the datafile on its end.
The failure happens inside the /addFiles api, with the following exception stack trace:

java.lang.NullPointerException: Cannot invoke "java.lang.Long.longValue()" because the return value of "edu.harvard.iq.dataverse.FileMetadata.getVersion()" is null
	at edu.harvard.iq.dataverse.util.json.JsonPrinter.json(JsonPrinter.java:672)
	at edu.harvard.iq.dataverse.util.json.JsonPrinter.json(JsonPrinter.java:656)
	at edu.harvard.iq.dataverse.util.json.JsonPrinter.jsonFileMetadatas(JsonPrinter.java:526)
	at edu.harvard.iq.dataverse.util.json.JsonPrinter.jsonDataFileList(JsonPrinter.java:486)
	at edu.harvard.iq.dataverse.datasetutility.AddReplaceFileHelper.getSuccessResultAsJsonObjectBuilder(AddReplaceFileHelper.java:1904)
	at edu.harvard.iq.dataverse.datasetutility.AddReplaceFileHelper.addFiles(AddReplaceFileHelper.java:2091)
	at edu.harvard.iq.dataverse.api.Datasets.addFilesToDataset(Datasets.java:4055)
... 

The above is likely a symptom, not the cause of the failure (the exception is thrown in AddReplaceFileHelper.getSuccessResultAsJsonObjectBuilder() - probably called after a failure to save that didn't get detected (?).
Attached is a fine server log output; starting from receiving a confirmation from globus about the completion of the upload.

Looking at the following lines: https://github.com/GlobalDataverseCommunityConsortium/dataverse/blob/6d3769f2a26b70e7108c18b92f39c1838ad8cd28/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java#L820-L852 it looks like when a failure occurs during this /addFiles api call it is not recorded as such (countError not incremented). Also, the edit lock is left on the dataset, because the removeDatasetLock() call is inside the catch() statement, and no exception is thrown in this case.

It's not immediately clear to me from the server log why the actual failure in /addFiles occurs. But has to be something simple - ?

server_log.txt

[poikilotherm]

/push-image thx

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:globusstore

ghcr.io/gdcc/configbaker:globusstore

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[landreev]

I have confirmed that the above (datafile not created, dataset stuck with an "edit in progress" lock) is happening consistently not just on my own dev. instance, but also on other servers, like dataverse-internal. Same exception stack trace.
I'm wondering if it's something very straightforward; like some necessary fix missing in the PR branch. Or maybe some JVM option, or setting that I am missing in my config?

[landreev]

Finally occurred to me to check - but it looks like it's not just the Globus uploads!
Direct uploads via DvUploader are similarly broken (same exception). I.e. /addFiles appears to be broken in develop (not just this branch).

This is from beta, which is running the develop branch build:

PROCESSING(F): /Users/landreev/NetBeansProjects/dataverse/test_685.txt
               Does not yet exist on server.
              UPLOADED as: MD5:ab3b010ddaaf87bc09944ad210e5d6eb
CURRENT TOTAL: 1 files :685 bytes
Error response when processing files in Individual files on command line : Bad Request
{"status":"ERROR","message":"Cannot invoke \"java.lang.Long.longValue()\" because the return value of \"edu.harvard.iq.dataverse.FileMetadata.getVersion()\" is null"}

[landreev]

It's really simple too: all it appears to be is that this JsonPrinter line in 6.1:

.add("version", fmd.getVersion())

is working because the JsonObjectBuilder methods are overridden by our own NullSafeJsonBuilder, and in develop we somehow lost that, and the .add methods from the native class are used.

[landreev]

(there was a wrong code line copy-and-pasted in the last comment; corrected)

[landreev]

It is a very recent change, and it is this line, 660 in JsonPrinter.json:

   JsonObjectBuilder builder = jsonObjectBuilder();

jsonObjectBuilder() creates a NullSafeJsonBuilder, but it is cast back to the native JsonObjectBuilder. ☹️

[landreev]

With the fix for the main issue merged in develop (#10366) and with the fix for file size>2GB committed, I can now upload files - yay.

I'm inclined to go ahead and merge this PR. With the acknowledgement that everything about Globus is rather convoluted, and I have been testing one simple workflow - uploading files to one specific endpoint (NESE test/demo tape), using Borealis dataverse-globus app. So, apologies if I missed something in the process.

[landreev]

Just noticed the "size 3" label - lol.