RIOXX v3 OAI profile support

[amgciadev]

Description

This PR implements a RIOXX v3 compliant OAI profile. RIOXX stands for Research Outputs Metadata Schema and the latest version of the specification is available at https://www.rioxx.net/profiles/v3-0-final/

This PR also adds support for exposing in OAI resource policies for bitstreams to address the issues / use cases described in #9075 and #9050 - see references below.

References

Add references/links to any related issues or PRs. These may include:

• Fixes OAI: Resource Policies for every bitstream at XOAI #9075
• Fixes OpenAIRE v4 Crosswalk uses item-level dc.rights metadata field rather than bitstream-level metadata #9050

Instructions for Reviewers

This PR implements a new RIOXX OAI context available at /oai/rioxx.
The solution adds a method in the XOAI ItemUtils.java file, that is called when the item is processed. It adds the extra XML elements to the XOAI bitstream, that expose the defined resource policies related to access level (group Anonymous + action READ).

To test all RIOXX elements are populated (in particular those that rely on related entities), follow similar instructions to those for OpenAire 4 as Rioxx reuses a number of Openaire relationships (https://wiki.lyrasis.org/display/DSDOC7x/OpenAIRE4+Guidelines+Compliancy):

1. Deploy PR
2. Enable entities (load both relationships.xml and openaire4-relationships.xml)

   cd [dspace]/bin
   ./dspace initialize-entities -f ../config/entities/relationship-types.xml
   ./dspace initialize-entities -f ../config/entities/openaire4-relationships.xml
   

3. Replace virtual-metadata.xml with contents of virtual-metadata.xml.openaire4

   cd [dspace]/config/spring/api/
   mv virtual-metadata.xml.openaire4 virtual-metadata.xml
   

4. Restart Tomcat container
5. run the OAI import from the command line with the cleaning cache parameter to reload all data to OAI

   [dspace]/bin/dspace oai import -c
   

List of changes in this PR:

• ItemUtils.java: expose resource policy information at the bitstream level
• Addition of rioxx.xsl (metadataFormats and associated rioxx.xsl transformer)
• Configuration of context in xoai.xml
• Addition of filter ItemsWithBitstreamFilter to support context filtering to only include items with bitstreams associated with them
• Addition of item.hasbitstream field to OAI solr schema (used in the RIOXX OAI context to determine whether an item has files in ORIGINAL Bundle)
• Addition of virtual-metadata.xml.rioxx3 and rioxx3-relationships.xml to model required relationships to expose relevant funding and project entities information

Include guidance for how to test or review your PR. This may include: steps to reproduce a bug, screenshots or description of a new feature, or reasons behind specific changes.

Checklist

This checklist provides a reminder of what we are going to look for when reviewing your PR. You need not complete this checklist prior to creating your PR (draft PRs are always welcome). If you are unsure about an item in the checklist, don't hesitate to ask. We're here to help!

• My PR is small in size (e.g. less than 1,000 lines of code, not including comments & integration tests). Exceptions may be made if previously agreed upon.
• My PR passes Checkstyle validation based on the Code Style Guide.
• My PR includes Javadoc for all new (or modified) public methods and classes. It also includes Javadoc for large or complex private methods.
• My PR passes all tests and includes new/updated Unit or Integration Tests based on the Code Testing Guide.
• If my PR fixes an issue ticket, I've linked them together.

[github-actions]

Hi @amgciadev,
Conflicts have been detected against the base branch.
Please resolve these conflicts as soon as you can. Thanks!

[amgciadev]

The overhead added to OAI was mentioned as a side effect of processing and including resource policies in the OAI metadata. I have run a full OAI reindex using this PR on a DSpace 7.6.1 instance for a large repository - 325,000 items:

• First run allocating 4GB of mem to the process and it did not complete. It errored close to the end (at around 310,000 items) with a java heap error
• Second run with 8GB mem (it probably does not need that much) and it has finished successfully in ~3 hours, which is a good time compared to previous runs in our repository

[tdonohue]

👍 Thanks @amgciadev . Gave this a test today and it works well. I've verified it fixes both the bugs listed in the description. I've also verified the new 'rioxx" context in OAI-PMH works along with the "rioxx" metadata format. (NOTE: I haven't verified that this OAI-PMH content is valid according to Rioxx itself...but it looks right at a glance).

The code also looks good except for a few very minor comments inline below. I'll go ahead and add my +1 vote, but I'll wait to merge until you have a chance to review the inline comments below.

[tdonohue]

@amgciadev : I meant to add that after merging this, we'll need to create some clear documentation for Rioxx support in DSDOC8x similar to the documentation for OpenAIRE v4: https://wiki.lyrasis.org/display/DSDOC8x/OpenAIRE4+Guidelines+Compliancy

Specifically, I think we'll need to make it clear when sites should use Rioxx versus OpenAIRE v4. It sounds like they are mostly "compatible" but you may only be able to support one or the other? (Or maybe I'm wrong and there's a way to support both?) In any case we need to add / enhance documentation here.

[amgciadev]

@tdonohue thanks so much for the review and the +1. You've made very useful comments re the documentation, absolutely agree re adding the page for Rioxx similar to the one for Openaire, and I am happy to work on that one too.

You've made a really important point re compatibility of Openaire and Rioxx: they should be completely compatible and in fact it is highly important that you are able to have them both enabled at the same time. @paulo-graca hopefully can help confirming this one / adding his thought given his knowledge of the Openaire work. The only area that may need a bit of reworking is the virtual-metadata files: virtual-metadata.xml.openaire4 and virtual-metadata.xml.rioxx3 to allow for the requirements of both to co-exist. I don't think they are incompatible.

I'll work on the minor comments asap. Thanks a lot!

[tdonohue]

@amgciadev : Thanks for clarifying that it should be possible to support both Rioxx v3 and OpenAIRE v4.

One challenge that I see to that in DSpace is that we appear to have 3 different virtual-metadata.xml files:

1. Default virtual-metadata.xml for Configurable Entities
2. virtual-metadata.xml.openaire4 for OpenAIRE v4 support
3. virtual-metadata.xml.rioxx3 for Rioxx v3 support (added by this PR)

Which file should be used if you want to support both Rioxx & OpenAIRE?

Currently, it's confusing to me how to achieve that. But, maybe you or @paulo-graca would have suggestions on how to do this. (Do we need a combined virtual-metadata.xml for both Rioxx and OpenAIRE?)

[amgciadev]

@tdonohue let me review virtual-metadata.xml.rioxx3 as what I tried to do with this one was to actually combine the default entities virtual-metadata file with the additions needed for rioxx re project fundind and awards (which are very close to those specified in openaire).

It may be that we don't need the added virtual-metadata.xml.rioxx3. Likewise, we don't need the separate entities/rioxx2-relationships.xml as the contents are the same as those in openaire-relationships.xml (I only added it to add the relevant comments for the metadata fields that they are used for as those are different between the two profiles). It may likely be that we could re-use relationships and virtual metadata for openaire and what we need to do is be very clear in the docs?

[tdonohue]

@amgciadev : That all sounds reasonable. If we find that Rioxx support just requires using the OpenAIRE configs, that's fine with me. As you said, we can just be clear in the docs. We could even say that Rioxx support depends on first enabling OpenAIRE support, if we find that is the best approach.

In any case, I'll wait to merge this until we have a final direction. Overall though, this works great & looks good. I can give it a final test again once it is ready.

[amgciadev]

@tdonohue I have performed more tests and wanted to raise the following:

1. Removed virtual-metadata.xml.rioxx3 as the relationships to populate related entities metadata for openaire4 can be reused for Rioxx
2. Kept rioxx3-relationships.xml eventhough the relationships are the same as for openaire, the file has useful docs as to what rioxx fields rely on what relationships (happy to remove if you feel that documenting this in the wiki suffices)
3. I think I have encountered a bug in virtual-metadata.xml.openaire which was using the incorrect direction of the relationship in publicationProject_projectFunderId_related

Without the change in 3. I have confirmed that oaire:funderIdentifier and oaire.funderName metadata from related entities was not being populated. This was also affecting the equivalent rioxxterms:grant elements which also includes a reference to the funder name and funder id. I didn't think it was a bug originally, which is why I ended up having a different virtual-metadata.xml file for Rioxx but I now believe is a bug.

With my changes in this last commit I can confirm that both openaire and rioxx metadata for funder names and funder identifiers are exposed in the following scenarios

1. A publication is linked with a project
2. That project is linked with a funder (orgUnit entity)
3. The funder has org name and identifier metadata

Example of the generated metadata:
Openaire:
<oaire:fundingReferences> <oaire:fundingReference> <oaire:awardTitle>CERJ Research Project</oaire:awardTitle> <oaire:funderName>UK Research and Innovation (UKRI)</oaire:funderName> <oaire:funderIdentifier funderIdentifierType="Crossref Funder ID">https://ror.org/001aqnf71</oaire:funderIdentifier> <oaire:awardNumber awardURI="https://diamond-f-dev.lib.cam.ac.uk/handle/1811/202">cerj-0001</oaire:awardNumber> </oaire:fundingReference>

Rioxx:

<rioxxterms:project>https://diamond-f-dev.lib.cam.ac.uk/handle/1811/202</rioxxterms:project> <rioxxterms:grant funderName="UK Research and Innovation (UKRI)" funder_id="https://ror.org/001aqnf71">cerj-0001</rioxxterms:grant>

I think this PR should be good to go now.

[tdonohue]

👍 Thanks @amgciadev ! I re-reviewed the latest updates and re-tested today. It is still working, and I appreciate the changes to just use the existing virtual-metadata.xml.openaire4 for Rioxx as well.

I'll wait to merge this until @paulo-graca has a chance to glance at it, as I'd appreciate any feedback he may have. But, as far as I'm concerned this looks ready to merge. Thanks again!

[paulo-graca]

@amgciadev : Thanks for clarifying that it should be possible to support both Rioxx v3 and OpenAIRE v4.

One challenge that I see to that in DSpace is that we appear to have 3 different virtual-metadata.xml files:

1. Default virtual-metadata.xml for Configurable Entities
2. virtual-metadata.xml.openaire4 for OpenAIRE v4 support
3. virtual-metadata.xml.rioxx3 for Rioxx v3 support (added by this PR)

Which file should be used if you want to support both Rioxx & OpenAIRE?

Currently, it's confusing to me how to achieve that. But, maybe you or @paulo-graca would have suggestions on how to do this. (Do we need a combined virtual-metadata.xml for both Rioxx and OpenAIRE?)

Sorry for my late feedback on this. I'm being swamped with local stuff.

We are currently using OpenAIRE's file configuration. To use it, we rename the existing file before the build process. However, I would prefer not to do it and instead I would prefer to rely on a configuration setting, possibly in my local.cfg, to use that speicific virtual metadata. What @amgciadev has done is the only way to have support for different virtual metadata on DSpace. In this case, we are incorporating RIOXX schema into DSpace, which requires some addional fields that aren't present in default virtual metadata. Additionally, since every .xml file in the /spring/api folder is interpreted, a non-xml file must be present.

[paulo-graca]

My comment has on hold for a while and I just saw recent commits. I'm glad that @amgciadev could use existing virtual metadata.

[amgciadev]

@paulo-graca thanks for the comments. Since you implemented the Openaire v4 integration, could you have a look at my changes in the last commit (in virtual-metadata.xml.openaire4) as per the below:

3. I think I have encountered a bug in virtual-metadata.xml.openaire4 which was using the incorrect direction of the relationship in publicationProject_projectFunderId_related

Without the change in 3. I have confirmed that oaire:funderIdentifier and oaire.funderName metadata from related entities was not being populated. This was also affecting the equivalent rioxxterms:grant elements which also includes a reference to the funder name and funder id. I didn't think it was a bug originally, which is why I ended up having a different virtual-metadata.xml file for Rioxx but I now believe is a bug.

With my changes in this last commit I can confirm that both openaire and rioxx metadata for funder names and funder identifiers are exposed in the following scenarios

1. A publication is linked with a project
2. That project is linked with a funder (orgUnit entity)
3. The funder has org name and identifier metadata

If you could confirm that this is a bug, that would be great. Thanks!

[paulo-graca]

Thank you @amgciadev for your contribution. I think it will help bring more DSpace installations. I think we need to ease the way to have virtual metadata configurations on DSpace.
Weirdly both configurations (yours and the old one) worked for me, I've tested them on my server and got the same expected result. So It's a +1. Thank you very much!

[tdonohue]

Merging as this is now at +2. Thanks again @amgciadev !

This now needs documentation to be added to the DSpace 8 documentation space similar to the OpenAIRE v4 docs in that space: https://wiki.lyrasis.org/display/DSDOC8x/OpenAIRE4+Guidelines+Compliancy Once those docs are added, the needs documentation label can be removed from this PR.

[amgciadev]

@tdonohue can you give me permissions to add/copy/edit pages in confluence? I tried copied the openaire page but got a permissions error while logged in.

[tdonohue]

@amgciadev : I've just updated your wiki permissions to have full permissions to the DSpace documentation spaces. You should now be able to edit/add pages.

[amgciadev]

@tdonohue docs added now: https://wiki.lyrasis.org/display/DSDOC8x/Rioxx+v3+schema+compliance

[tdonohue]

Thanks @amgciadev ! Docs look good to me. I'll take off the "needs documentation" label.