Add DOI to icat publishing

[mantid-builder]

This issue was originally TRAC 8538

Original Reporter: Jay Rainey

This ticket is blocked by :

• TRAC9208
• TRAC9223

This ticket is blocks : TRAC9186

This will need to be discussed further with Tom.

---

Keywords: ICAT, publish

[mantid-builder]

Jay Rainey (2014-01-28T16:32:24):
Having spoken with Tom, the following was concluded:

1. The DOI API is developed by the SCP, which is unlikely to be released by the code-freeze.
2. Generating a DOI for a datafile is optional.
3. When a user publishes a datafile they are automatically made private. Tom suggested that I should implement a checkbox to provide the user with an optional to make data private, and by default make it public.

   ---

@NickDraper (2014-02-14T10:59:50):
Bulk move of tickets out of triage (new) to assigned at the introduction of the triage state

---

Jay Rainey (2014-04-04T10:51:50):
To add DOI generation support for publishing I need to:

1. Generate gSoap files against the DOI generator end-point, which are:
   • https://data4.isis.stfc.ac.uk/doi/DOIService?wsdl // Develop
   • https://sig-03.esc.rl.ac.uk:8181/doi/DOIService?wsdl // Production
2. Add the end-point (dev for now until changes below are made) to Facilities.xml.
3. Update ICatalogInfoService to include a new method (generateDOIForDatafile) that makes use of the gSoap methods above (specifically: registerDatafileDOI) and returns the generated DOI (to output on LOG).
4. Add and implement the functionality of the new method above in ICat4Catalog.
   • This includes obtaining the end-point from Facilities.xml t oaccomdate other facilities using th
5. Add two new properties to CatalogPublish:
   • InvestigationID, "",`The database row ID of the investigation that you want to publish to. (Note: this is required to generate a DOI.)
   • GenerateDOI, true, "Generates a DOI for the datafile being published. Once a DOI is generated the datafile will be made public."
6. Add logic to exec in CatalogPublish to make use of the properties and call generateDOIForDatafile from the catalog. '''Note:''' if a file fails to upload (e.g. already exists etc...) a DOI '''must not''' be generated.
7. Update CatalogPublishDialog to have a checkbox for GenerateDOI. InvestigationID will be hidden on the GUI and obtained from the table generated getPublishInvestigations (once http://trac.mantidproject.org/mantid/ticket/9223 is complete).
   • The generateDOI (make public) checkbox on the GUI should be checked by default.
   • To prevent users accidentally generating DOIs that they may not have wanted to, a QMessageBox should be generated IF the user has checked generateDOI to confirm this choice.

     ---

Jay Rainey (2014-04-08T14:47:34):
Added DOI endpoint to Facilities.xml. Refs http://trac.mantidproject.org/mantid/ticket/8538.

ecea7bc

---

Jay Rainey (2014-04-08T14:47:35):
Added tests for doi in CatalogInfo. Refs http://trac.mantidproject.org/mantid/ticket/8538.

• Also added missing testing for external download url (IDS).

866ff0f

---

Jay Rainey (2014-04-08T14:47:35):
Added doiEndPoint method to CatalogInfo. Refs http://trac.mantidproject.org/mantid/ticket/8538.

bb5a48f

---

Jay Rainey (2014-04-08T14:47:35):
Added gSoap generated code for endpoint. Refs http://trac.mantidproject.org/mantid/ticket/8538.

b0e3af7

---

Jay Rainey (2014-04-08T14:47:35):
Add registerDatafileDOI to interface & inherited classes. Refs http://trac.mantidproject.org/mantid/ticket/8538.

78723d5

---

Jay Rainey (2014-04-08T14:47:35):
Refactor CatalogPublish error handling. Refs http://trac.mantidproject.org/mantid/ticket/8538.

f968514

---

Jay Rainey (2014-04-08T14:47:35):
Merge 'origin/master' into feature/8538_icat_publish_dois. Refs http://trac.mantidproject.org/mantid/ticket/8538.

85af503

---

Jay Rainey (2014-04-08T14:47:36):
Add properties and logic to catalogPublish algorithm. Refs http://trac.mantidproject.org/mantid/ticket/8538.

c8c27c2

---

Jay Rainey (2014-04-08T14:47:36):
Use getRef for ease of use. Refs http://trac.mantidproject.org/mantid/ticket/8538.

edf82a7

---

Jay Rainey (2014-04-08T14:47:36):
Rename parameter on registerDOI. Refs http://trac.mantidproject.org/mantid/ticket/8538.

df97d12

---

Jay Rainey (2014-04-08T14:47:36):
Make SSLContext method generic via templates. Refs http://trac.mantidproject.org/mantid/ticket/8538.

• This means we can make use of this for ICATProxy and the DOIProxy objects.

81fabd7

---

Jay Rainey (2014-04-08T14:47:36):
Make throwErrorMessage generic via templates. Refs http://trac.mantidproject.org/mantid/ticket/8538.

e228227

---

Jay Rainey (2014-04-08T14:47:36):
Renamed method. Refs http://trac.mantidproject.org/mantid/ticket/8538.

ff5abaf

---

Jay Rainey (2014-04-08T14:47:37):
Added DOI catalog implementation. Refs http://trac.mantidproject.org/mantid/ticket/8538.

3b19191

---

Jay Rainey (2014-04-08T14:47:37):
Added DatabaseID to investigation workspace. Refs http://trac.mantidproject.org/mantid/ticket/8538.

• Removed investigationID check as it's handled on the DOI server.

bc1936c

---

Jay Rainey (2014-04-08T14:47:37):
Use specified namespace instead of ns1. Refs http://trac.mantidproject.org/mantid/ticket/8538.

d0ce080

---

Jay Rainey (2014-04-08T14:47:37):
Updated readme to include namespace generation. Refs http://trac.mantidproject.org/mantid/ticket/8538.

• The -n option overrides the ns namespace prefix. This is required to prevent warnings if several end-points have been generated without a namespace set.

a1a7698

---

Jay Rainey (2014-04-08T14:47:37):
Add make public checkbox to publish GUI. Refs http://trac.mantidproject.org/mantid/ticket/8538.

ae0b062

---

Jay Rainey (2014-04-08T16:11:37):
Set DatabaseID property of Publish algorithm via GUI. Refs http://trac.mantidproject.org/mantid/ticket/8538.

b18ec0d

---

Jay Rainey (2014-04-08T16:18:41):
Fix include. Refs http://trac.mantidproject.org/mantid/ticket/8538.

dc3d16e

---

Russell Taylor (2014-04-08T19:28:04):
The system test that validates of the Facilities file against its schema is failing because "doiendpoint" does not feature in the schema.

---

@martyngigg (2014-04-09T07:21:51):
There are quite a lot of new cppcheck [http://download.mantidproject.org/jenkins/view/Static%20Analysis/job/is_cppcheck_develop/517/cppcheckResult/ warnings] for the generated code. I think we will need to add another EXCLUDE for the new gsoap-generated code in Code/Mantid/Build/CMake/CppCheckSetup.cmake.

---

Jay Rainey (2014-04-09T08:53:22):
Add CPPEXCLUDE for DOI gSoap generated code. Refs http://trac.mantidproject.org/mantid/ticket/8538.

0cf95f1

---

Jay Rainey (2014-04-09T09:14:40):
Added doiendpoint to FacilitiesScheme. Refs http://trac.mantidproject.org/mantid/ticket/8538.

629055c

---

Jay Rainey (2014-04-09T09:14:40):
Added doi field to datafile table. Refs http://trac.mantidproject.org/mantid/ticket/8538.

b990be3

---

Jay Rainey (2014-04-10T13:38:19):
Restrict doiendpoint as facilities may not support. Refs http://trac.mantidproject.org/mantid/ticket/8538.

4521af6

---

Jay Rainey (2014-04-25T11:20:31):
Not sure why, but several commits I just made didn't make their way onto track. They were:

1. [https://github.com/mantidproject/mantid/commit/9ac89d8f619f741ca83bba8f5d18fafd374f5d0c Merged master into branch after http://trac.mantidproject.org/mantid/ticket/9223 was completed.]
2. [https://github.com/mantidproject/mantid/commit/c1c6557702e4038f6d07a8d391cccbff343266f4 Refactor ICAT SOAP calls to improve readability.]
3. [https://github.com/mantidproject/mantid/commit/4f19b94c2827d2b5d582e6faf6f0221bb6bc11d1 Get the error message from soap, rather than do it manually.]
4. [https://github.com/mantidproject/mantid/commit/d97a5f1e82121f792de45bd4bf7be916dd8575ab Output the gSOAP error code to debug - very useful.]

   ---

Jay Rainey (2014-05-16T13:17:40):
Moved keepAlive above IDS methods. Refs http://trac.mantidproject.org/mantid/ticket/8538.

22d2990

---

Jay Rainey (2014-05-16T13:17:40):
Updated doiendpoint (still using dev). Refs http://trac.mantidproject.org/mantid/ticket/8538.

63ea25c

---

Jay Rainey (2014-05-16T13:17:41):
Updated doiendpoint to new development url. Refs http://trac.mantidproject.org/mantid/ticket/8538.

0f951ee

---

Jay Rainey (2014-05-16T13:17:41):
Merge 'origin/master' into feature/8538_icat_publish_dois. Refs http://trac.mantidproject.org/mantid/ticket/8538.

Conflicts:

• Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

4b1e897

---

Jay Rainey (2014-05-16T13:17:41):
Add better logging messages for publishing. Refs http://trac.mantidproject.org/mantid/ticket/8538.

e1b0fc6

---

Jay Rainey (2014-05-16T13:24:53):
The functionality is complete, but am having issues with the ICAT DOI development server (It's throwing an error when DOI generation should be valid). I've got in touch with the DOI developer and once this is sorted this can be tested.

---

Jay Rainey (2014-06-09T11:04:02):
Updated DOI wsdl

---

Jay Rainey (2014-06-10T09:25:31):
'''The functionality for this ticket is complete.''' However, there are permission issues in ICAT that are preventing DOIs being generated, which Tom is looking into. Once these have been addressed this ticket can be opened for testing. If it does not pass testing, any additional changes should occur on [https://github.com/mantidproject/mantid/compare/master...feature;8538_icat_publish_dois the published branch].

= Background =

This ticket deals with generating a DOI when the user publishes a datafile to the archives and has checked the '''"Make public"''' option on the CatalogPublish dialog.

= Testing =

== Development settings in Facilities.xml ==

You '''must''' update the '''url''' of ''soapendpoint'', ''externaldownload'' and ''doiendpoint'' to be the following to ensure only test DOIs are generated:

 `<soapendpoint url="https://icatdev.isis.cclrc.ac.uk/ICATService/ICAT">``</soapendpoint>`
 `<externaldownload url="https://isisoxfordvmsrv.isis.cclrc.ac.uk/ids/">``</externaldownload>`
 `<doiendpoint url="https://topcatdev.isis.cclrc.ac.uk/doi/DOIWebService">``</doiendpoint>`

=== Generating a DOI ===

To be thorough I have attached a python script that will generate a DOI against the test server. This should be used prior to testing it within Mantid. This is a faster way to observe if there are any errors on ICAT when generating DOIs.

==== Generating a DOI with the provided script ====

'''Note: '''If the DOI has been generated, then this proves that a DOI can be generated, and saved into ICAT successfully.

'''Remember to change your Facilities.xml file to the developer options.'''

1. Run the attached script. Enter your username & password.
2. Select f for the Object type as we want to generate a DOI for a datafile.
3. Enter '''32493044''' into the Object id field. '''Note:''' This is the ''ID'' of a datafile in an EMU investigation. The title of this investigation is: Magnetic ionic liquid surfactants.
4. You will notice that a DOI has been generated. If an error occurs (DOI not created, response code: 400), then this ICAT permissions is is still not fixed! Contact Tom about this.
5. Now that a DOI has been generated we need to see if it saved correctly to ICAT.
6. Log into the catalog and open the Search GUI.
7. In the Investigation name field enter: Magnetic ionic liquid surfactants and click Search.
8. Double click on the investigation and the Datafiles information table will be populated.
9. Observe that a DOI now exists in the related row for that datafile.

==== Generating a DOI within Mantid ====

'''Remember to change your Facilities.xml file to the developer options.'''

1. Log into the catalog.
2. Left click Publish in the Catalog menu. The publishing GUI will open.
3. Select Make public. This option ensures a DOI is generated when the datafile is published.
4. Select a datafile to upload to ICAT.
5. Enter a name in the Name in catalog field. Copy that name to your clipboard.
6. Select an investigation where you would like to publish the datafile.
7. Click Run. If the file is uploaded successfully then a DOI will be generated.
8. Open the catalog search GUI.
9. Click Advanced search and paste the above file name into the Datafile name field.
10. Click Search and double click the investigation that is returned.

=== Code review ===

As generating a DOI is ''irreversible'' it is essential to perform a ''thorough'' code review. I recommend reading comment:8 to gain an idea of what the desired logic for this ticket was.

'''Note:''' once this ticket has been tested the ICAT DOI endpoint (doiendpoint) should be updated to that of the production server, and not the testing server.

---

@NickDraper (2014-12-08T10:25:31):
Moved to the backlog at the code freeze of R3.3

[mantid-builder]

http://trac.mantidproject.org/mantid/raw-attachment/ticket/8538/generate_doi.py
(uploaded by Jay Rainey on 2014-06-09T11:04:02)

---

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you feel this is incorrect please comment to keep it alive, with a reason why.

To prevent closure, e.g. for long-term planning issues, add the "Never Stale" label.

[stale]

This issue has been closed automatically. If this still affects you please re-open this issue with a comment so we can look into resolving it.