[DS-4205] Upgrade to Handle Server v9

[ilittle-cnri]

https://jira.duraspace.org/browse/DS-4205

This PR is to upgrade the version of the Handle Server software included in DSpace to the latest release (v9.1.0).

Note: Building this PR will require the updated Handle.Net library (v9.1.0) to be uploaded to Maven Central (for now, under the org.dspace namespace). I've sent the relevant files to @tdonohue for that process. CNRI intends to maintain the library under its own namespace in the future, and DSpace will be able to point at that library instead.

Changes to the install procedure

There are no changes to the setup instructions needed for this update.

Changes to the upgrade procedure

There are no changes to the automatic part of the upgrade process. There is one manual change required: In the existing Handle server config.dct file, "enable_txn_queue" = "no" must be added to the server_config section.

This change is required because, by default, the Handle v9 transaction queue uses a version of BerkleyDB that is only available from the Oracle maven repository. Since DSpace doesn't actually use this queue, I thought it made more sense to turn off that feature and not have to pull in the extra dependency from another repository. However, if you'd prefer to have a completely invisible upgrade, I can modify this PR to add that repository and pull in the needed library instead.

What to test

New Install: Should be able to setup a new DSpace instance with Handle server using the current instructions. When objects are created, handles should be assigned in DSpace and resolvable through the Handle server interface.

Existing Install: Should be able to follow the standard upgrade instruction, with the addition of making the above mentioned config.dct change before starting the Handle server. Existing handles should continue to resolve correctly, and new objects should get resolvable handles with the same prefix. If the server uses a globally-resolvable prefix, it should still be possible to resolve handles using hdl.handle.net.

Note: I was not able to actually test the standard upgrade process, because the in-progress Solr changes seem to have broken things at the moment. I did, however, copy the handle-server directory to an existing DSpace 7 install from a DSpace 6 install, and verified that Handle v9 in DSpace 7 was able to read and use that directory correctly.

What's New in Handle v9.1.0

1. Support for IPv6 endpoint at Handle.Net server interface.
2. Enables updated cryptographic primitives including use of SHA-256 and AES.
3. Support for HTTP JSON API, which may be useful for developing applications using Handles.
4. Leverages Java 8.
5. Prefixes can now be requested from [multiple parties including CNRI] (https://www.dona.net/mpas).
6. Additional improvements and features available when a standalone Handle.Net server is setup. See Release Notes for more details.

Additional notes

An additional optional upgrade step might be suggested to users running Handle servers. Users could update their site information to note that they are using a newer version of the Handle protocol. The site information update will cut some of the runtime negotiation related to protocol versioning. Details for performing this optional step are in the Handle Technical Manual, Chapter 2. Admins can also contact CNRI about this if they have questions at hdladmin@cnri.reston.va.us.

[tdonohue]

@ilittle-cnri : I've released the Handle JAR (v9.1.0) to Maven Central, as you can see here. Unfortunately, though, this PR still fails to build, as the Handle JAR's POM declares a dependency on net.cnri:cnriutil:jar:2.0 (i.e. cnriutil.jar version 2.0). As that dependency is not found in Maven Central, the Maven build of this PR fails.

Is cnriutil.jar a required dependency of handle.jar? If so, we may need to release it to Maven Central as well (along with any other required dependencies that are not already there). If not, I will need to push a new POM to Maven Central to update the list of dependencies.

[ilittle-cnri]

It is required. I'll send you an email with a link to download the jar and pom files needed (including the source and java doc jars). I guess you'll also need to modify the Handle POM to point at cnriutil under the dspace namespace.

[ilittle-cnri]

Actually, instead of an email, I've attached the files to the JIRA ticket. cnriutils should use the same license information as the Handle library for now.

[terrywbrady]

@tdonohue , @ilittle-cnri ,

I have a question about handle server testing.

How useful would it be to package the handle server as a docker container that could be run with DSpace? Does this make sense to attempt?

Currently, I only run the handle server on my production server. The only way that I no to test it is to resolve an hdl.handle.net handle.

[tdonohue]

@terrywbrady : just for clarity, in DSpace 7 the handle server will still be embedded as part of DSpace (in the same way that it was in DSpace 6 and below). So, in Docker, I'd expect it may need to be run on the container that has the DSpace configuration, REST API, etc. I'm not sure that it'd be possible to run from a separate container until the Handle Server is "extracted" and run standalone (alongside DSpace), which isn't likely to happen until DSpace 8 at the earliest.

So, to sum up, DSpace 7 will run the Handle Server in the same way as DSpace 6 (and below). It'll just run the latest version of the Handle Server.

[terrywbrady]

Thanks @tdonohue . If we choose to make the handle server available in a container, it would need to be a separate container. It could use the same code, but it would have a different hostname.

[tdonohue]

@ilittle-cnri : Just dropping you a note to let you know I released both Handle.jar and CNRIUtil.jar to Maven Central (I appended a release date on each as I cannot re-release the same version twice)

• Handle: http://central.maven.org/maven2/org/dspace/handle/9.1.0.v20190416/
• CNRIUtil: http://central.maven.org/maven2/org/dspace/cnriutil/2.0.v20190416/

After some minor fixes to this PR (see last two commits), this is now building & passing all tests! So, it's ready for review!

@mwoodiupui : as you previously volunteered to give this a look/test, I've added you as a reviewer.

[mwoodiupui]

I think we need to review our installation instructions. I followed them and was led through a number of questions that I did not expect. The resulting config.dct lacked the "enable_txn_queue" = "no" directive. The instructions as written seem to bypass the bin/make-handle-config shell script.

Once that was corrected, the resolver was able to start. I was able to browse it at http://[hostname]:8000. I was able to use that page to query for the Site's Handle record (123456789/0). The instance has as yet no content, so this is as far as I got. It's looking very well.

[benbosman]

Verified that the handle server resolving still works when:

• Upgrading the config by adding "enable_txn_queue" = "no"
• Creating a new config by executing make-handle-config

I did not connect it to the official handle.net, but only verified in the UI available on port 8000. I have no production environment on DSpace 7 to verify this

[mwoodiupui]

We definitely need to update the current instructions. They say:

[dspace]/bin/dspace make-handle-config [dspace]/handle-server

The launcher is configured such that bin/dspace make-handle-config runs the class net.handle.server.SimpleSetup instead of the bin/make-handle-config script. This takes the user through an interview which is not described in the DSpace manual. We should probably just drop this command from the launcher (and the manual), and remove the instructions to manually set storage_type and storage_class and replace instances of 300:0.NA/YOUR_NAMING_AUTHORITY.

[mwoodiupui]

+1 so long as we fix the DSpace documentation.
I also have no spare prefixes for testing. After building and installing DSpace and running bin/make-handle-config, bin/start-handle-server completes without error and the server is running on the expected ports. I see no errors in the generated configuration.

[ilittle-cnri]

Awesome, thanks. It's on my task list to go through and update the documentation. I'll bump that up in priority now that this has two approvals.

[tdonohue]

@ilittle-cnri : now that this is at +2 votes, it's lined up to be merged soon (thanks for your patience here!). However, I notice that this PR has two (seemingly minor) merge conflicts (in two POM files). Would you be able to resolve those merge conflicts so that this is ready to merge? (We might be able to merge this as soon as this week if the merge conflicts are easily resolved)

[ilittle-cnri]

Everything looks good now. Thanks for your work on this!

[tdonohue]

Merging, as this is at +2 and looks good to me.

I will keep the associated JIRA ticket open (https://jira.duraspace.org/browse/DS-4205) until the Documentation has been updated per the above notes. Thanks again @ilittle-cnri for your contributions!