CVOC : allow flexible params in retrievalUri (Ontoportal integration)

[luddaniel]

@qqmyers Following agreements on next steps of #10145 here is a way to add flexibility in retrievalUri.
We can now add named fields (child or siblings) of term-uri-field and capacity to encodeUri value when usefull.
Example below :

cvoc_conf.json contains :

"field-name": "keyword",
"term-uri-field": "keywordTermURL",
"retrieval-uri": "https://data.agroportal.lirmm.fr/ontologies/{keywordVocabulary}/classes/{encodeUrl:keywordTermURL}",
"managed-fields": {
        "vocabularyName": "keywordVocabulary",
        "termName": "keywordValue",
        "vocabularyUri": "keywordVocabularyURI"
    },

Resulting in retrieval-uri = https://data.agroportal.lirmm.fr/ontologies/AGROVOC/classes/http%3A%2F%2Faims.fao.org%2Faos%2Fagrovoc%2Fc_331529

This code was tested with a classic skomos https://skosmos.loterre.fr/rest/v1/data?uri={0} and it's still working.

Refactoring has been made regarding the use of registerExternalTerm function around dataset imports / updates methods where it was used while parsing json fields meaning imcomplete related fields. Now registerExternalTerm is called after dataset metadata are ready to be commited and directly as step of Command of dataset create and update.

[coveralls]

Changes unknown
when pulling 2774bc9 on Recherche-Data-Gouv:9276-allow-flexible-params-in-retrievaluri-cvoc
into ** on IQSS:develop**.

[qqmyers]

Looks good. Nice to have the register part out of the DatasetPage and in the commands (where it can be run with the SPA).

It would be good to add a release note (could be a combined one for this and #10331). It would also be helpful to have some testing instructions and/or a test: nominally existing scripts should still work, so those could be tested with the examples in the https://github.com/gdcc/dataverse-external-vocab-support/ repo. If there isn't an ontoportal example yet, perhaps just verifying that the substitution works in replaceRetrievalUriParam in a unit test would be helpful.

I'll also note that this is matched by a docs update at gdcc/dataverse-external-vocab-support#20.

[luddaniel]

Hello @qqmyers Thanks for the review,

• Release note has been added for both PR here (CVOC : allow flexible params in retrievalUri (Ontoportal integration) #10404 + CVOC : add HTTP request headers (Ontoportal integration) #10331) .
• PR synchronized with develop
• Update doc - Allow flexible params in retrievalUri (Ontoportal integration) gdcc/dataverse-external-vocab-support#20 updated

For testing / unit tests / integration tests, we decided to go for unit tests of methods of DatasetFieldServiceBean, coming in another PR. Other kind of testing looks complicated for now. I'll talk to you on slack for this topic.

[luddaniel]

Decided with @qqmyers, the unit tests for all PR for Ontoportal / CVOC will be pushed after the merging of PR as it will require merged codes.

[qqmyers]

@luddaniel - There's 1 test failing with this PR - a 500 error returned instead of a 200 response at edu.harvard.iq.dataverse.api.EditDDIIT.testUpdateVariableMetadata(EditDDIIT.java:153) which is caused ultimately by an attempt to save a DatasetVersion that doesn't have a createTime set. From the lengthy stack trace, it looks like that is due to the call to ctxt.dsField().registerExternalVocabValues(df). That calls em.flush() if an external val is found - perhaps that or something else in the call is causing the DatasetVersion itself to be flushed to the db. (I also see @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) on registerExternalTerm().)

This all happens before the date gets set in UpdateDatasetVersionCommand line ~261 (setting the last update time also sets the createTime if it was null) so anything causing the new DatasetVersion to be written to the db would trigger the problem since createTime is defined as non-nullable.

If that's all correct, I think editing/perhaps having to add a CVOC value to a published version (causing a new version to be created) in the UI would trigger the issue. Assuming you can repeat the problem, it might be that we no longer need the flush(), or that setting the date can be moved earlier in the method, etc.

[qqmyers]

Suggested change:
You can now use managed-fields field names as well as the term-uri-field field name as parameters in the 'retrieval-uri' when configuring an external vocabulary service. '{0}' as an alternative to using the 'term-uri-field' name is still supported for backward compatibility.