GDCC/8605-add-archival-status-support

[qqmyers]

What this PR does / why we need it: To support more sophisticated Archivers (i.e. those that can provide status feedback and may have multistep internal processes), this PR adds support for managing this status. Specifically it changes the archivalCopyLocation from being a null/String (originally intended as a URL identifying/providing a landing page for the archival copy in the archiving system) to being a json object that contains a 'status' of 'success'/'pending'/'failure' and a 'message' that is again a string. In the success case, the message is again intended as an identifier/landing page URL whereas for failure and pending, the message can be an informative string.
As noted in the issue, this work is supported as part of the Harvard Data Commons project (3A) for use specifically with the DRS Archiver. However, the PR includes updates to the other existing archivers to use the same format (although these currently only have success and failure status, no pending states.)

Which issue(s) this PR closes:

• Closes HDC 3A: support handling archival status updates #8605

Special notes for your reviewer:

• Could rename the db column as it is no longer a location.
• The API calls follow the original naming convention of the admin API submitDatasetVersionToArchive format which doesn't fit as well with the /api/datasets convention of having the next. These could be changed - would require changes in the DataCommons service that calls them - and presumably we/d want to align the existing admin call and batch call in TDL/7493 Batch Archiving #8610 - let me know a decision.
• Also note that the flyway script also handles the 'Attempted' state introduced in GDCC/8604 Improve archiver error handling #8612. Nominally this should only be in development databases and at TDL where this was added to avoid rerunning the archiving for failed datasets when doing batch uploads. That will be superseded/replaced by this PR.
• FWIW: This API was added to the /datasets endpoint because the intent is for remote archiving systems (like DRS) to report their status updates and putting it in admin would restrict it to localhost or require changing to the unblock-key policy. The API is limited to superuser use.

Suggestions on how to test this: The new API supports get/set/delete of the status values. The simplest test would be to configure an archiver, such as the Local file archiver and use the API to retrieve the status and verify the success message. (I think misconfiguration of that, e.g. pointing to a directory where the archiver can't write, should allow viewing a failure status as well.
Also note that another PR will be coming that will show the archival status in the versions table - more opportunity to test the api with that.

Does this PR introduce a user interface change? If mockups are available, please link/include them here: No - this is db/api only

Is there a release notes update needed for this change?: part of #8611

Additional documentation:

[coveralls]

Coverage decreased (-0.03%) to 19.736% when pulling 7410c5b on GlobalDataverseCommunityConsortium:GDCC/8605-add-archival-status into 567e506 on IQSS:develop.

[pdurbin]

I didn't run the code yet but here's some initial feedback.

[pdurbin]

Any reason not to use getDatasetVersionOrDie here (and in the other two calls to findByFriendlyVersionNumber in this PR)?

[qqmyers]

Not sure I saw it but looking now, getDatasetVersionOrDie doesn't support the friendlyVersionNumber syntax which is a ~requirement here (that's the convention used in the Bag naming and metadata that the archiver gets). I can go ahead and add parsing for that which would have the presumably useful side effect of letting other datasetversion api calls support the friendly version number as well.

[pdurbin]

It should. I'm seeing handleSpecific(long major, long minor). It's used by https://guides.dataverse.org/en/5.11/api/native-api.html#get-version-of-a-dataset which has a "friendly" example of "1.0".

[qqmyers]

Yep - you're right. I missed the string parsing in handleVersion(). I'll update the PR to use it.

[qqmyers]

Hmm - calls to this are counted with MakeDataCounts. I guess since these are API calls they should count? (although they are clearly system-level interactions and not end-user interaction with the data). In any case, I went ahead for now.

[pdurbin]

I dunno. I'd leave this out of Make Data Count. Like you said, these are systems setting and retrieving archival status messages. The spirit of Make Data Count is views/investigations and downloads/requests. People and machines looking at data.

[pdurbin]

submitDatasetVersionToArchive is a weird name. submitDataVersionToArchive (Data instead of Dataset) is under /api/admin and documented under installation/config.html

[qqmyers]

Yes. So far it ~mirrors the /api/admin/submitDatasetVersionToArchive call (name changed to say 'Dataset' in #8610 which hasn't merged yet), which seemed reasonable when it was a single call. With the status calls, I initially had them in /api/admin as well, but eventually decided they should move to /api/datasets (see the comment about superuser being required on those). With that, they could be renamed - e.g. to /api/datasets/<id>/<version>/archivalStatus .

[pdurbin]

I like the new name ending with /archivalStatus. Thanks.

[pdurbin]

If this script is only needed by TDL as suggested in the PR description, perhaps we don't need it.

[qqmyers]

UPDATE datasetversion SET archivalCopyLocation = CONCAT('{"status":"success", "message":"', archivalCopyLocation,'"}') where archivalCopyLocation is not null is needed for standard instances (those that have used archiving and therefore have non-null entries). The and not archivalCopyLocation='Attempted'; and the second line handle the case that TDL deployed which was in the initial PR #8610 which has gotten passed by this PR.

[pdurbin]

Ok, I guess my understanding is that both lines are needed or at least won't hurt anything.

[pdurbin]

Looks good. I played around with the tests in DatasetsIT. I didn't test the SQL upgrade script.