OpenAIRE guidelines for Data Archives #4664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

kcondon merged 43 commits into IQSS:develop from 4Science:openaire

May 7, 2019

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -2,6 +2,9 @@ nb-configuration.xml @@
     target
     infer-out
     nbactions.xml
+    .settings
+    .classpath
+    .project
     michael-local
     GPATH
     GTAGS
@@ Expand Down @@

doc/sphinx-guides/source/admin/metadataexport.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -7,14 +7,7 @@ Metadata Export @@
     Automatic Exports
     -----------------
-    Publishing a dataset automatically starts a metadata export job, that will run in the background, asynchronously. Once completed, it will make the dataset metadata exported and cached in all the supported formats:
-    - Dublin Core
-    - Data Documentation Initiative (DDI)
-    - DataCite 4
-    - native JSON (Dataverse-specific)
-    - OAI_ORE
-    - Schema.org JSON-LD
+    Publishing a dataset automatically starts a metadata export job, that will run in the background, asynchronously. Once completed, it will make the dataset metadata exported and cached in all the supported formats listed under :ref:`Supported Metadata Export Formats <metadata-export-formats>` in the :doc:`/user/dataset-management` section of the User Guide.
     A scheduled timer job that runs nightly will attempt to export any published datasets that for whatever reason haven't been exported yet. This timer is activated automatically on the deployment, or restart, of the application. So, again, no need to start or configure it manually. (See the "Application Timers" section of this guide for more information)
@@ Expand Down @@

doc/sphinx-guides/source/api/native-api.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -291,7 +291,7 @@ Export Metadata of a Dataset in Various Formats @@
         GET http://$SERVER/api/datasets/export?exporter=ddi&persistentId=$persistentId
-    .. note:: Supported exporters (export formats) are ``ddi``, ``oai_ddi``, ``dcterms``, ``oai_dc``, ``schema.org`` , ``OAI_ORE`` , ``Datacite`` and ``dataverse_json``.
+    .. note:: Supported exporters (export formats) are ``ddi``, ``oai_ddi``, ``dcterms``, ``oai_dc``, ``schema.org`` , ``OAI_ORE`` , ``Datacite``, ``oai_datacite`` and ``dataverse_json``.
     Schema.org JSON-LD
     ^^^^^^^^^^^^^^^^^^
@@ Expand Down @@

doc/sphinx-guides/source/user/dataset-management.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -20,7 +20,20 @@ A dataset contains three levels of metadata: @@
     For more details about what Citation and Domain Specific Metadata is supported please see our :ref:`user-appendix`.
-    Note that once a dataset has been published its metadata may be exported. A button on the dataset page's metadata tab will allow a user to export the metadata of the most recently published version of the dataset. Currently supported export formats are DDI, Dublin Core, Datacite 4, OAI_ORE, Schema.org JSON-LD, and Dataverse's native JSON format.
+    .. _metadata-export-formats:
+    Supported Metadata Export Formats
+    ---------------------------------
+    Once a dataset has been published its metadata is exported in a variety of formats. A button on the dataset page's metadata tab will allow a user to export the metadata of the most recently published version of the dataset. Currently supported export formats are:
+    - Dublin Core
+    - DDI (Data Documentation Initiative)
+    - DataCite 4
+    - JSON (native Dataverse format)
+    - OAI_ORE
+    - OpenAIRE
+    - Schema.org JSON-LD
     Adding a New Dataset
     ====================
@@ Expand Down Expand Up @@
     .. |file-upload-prov-window| image:: ./img/prov1.png
        :class: img-responsive
     .. |image-file-tree-view| image:: ./img/file-tree-view.png
-       :class: img-responsive
+       :class: img-responsive

pom.xml

-Original file line number
+Diff line change
@@ Expand Up / @@ -599,6 +599,12 @@ @@
                 <artifactId>tika-parsers</artifactId>
                 <version>1.19</version>
             </dependency>
+            <!-- Named Entity Recognition -->
+            <dependency>
+                <groupId>org.apache.opennlp</groupId>
+                <artifactId>opennlp-tools</artifactId>
+                <version>1.9.1</version>
+            </dependency>
         </dependencies>
         <build>
             <!--        <testResources>
@@ Expand Down Expand Up / @@ -632,6 +638,7 @@ @@
                     <includes>
                         <include>**/*.sql</include>
                         <include>**/*.xml</include>
+                        <include>**/firstNames/*.*</include>
                     </includes>
                 </resource>
             </resources>
@@ Expand Down @@

src/main/java/edu/harvard/iq/dataverse/S3PackageImporter.java

-Original file line number
+Diff line change
@@ Expand Up / @@ -21,7 +21,7 @@ @@
     import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
     import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException;
     import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
-    import edu.harvard.iq.dataverse.util.FileUtil;;
+    import edu.harvard.iq.dataverse.util.FileUtil;
     import java.io.BufferedReader;
     import java.io.IOException;
     import java.io.InputStreamReader;
@@ Expand Down @@

src/main/java/edu/harvard/iq/dataverse/api/dto/DataFileDTO.java

-Original file line number
+Diff line change
@@ Expand Up / @@ -12,6 +12,7 @@ public class DataFileDTO { @@
         private String id;
         private String storageIdentifier;
         private String contentType;
+        private Long filesize;
         private String filename;
         private String originalFileFormat;
         private String originalFormatLabel;
@@ Expand Down Expand Up / @@ -61,6 +62,14 @@ public String getContentType() { @@
         public void setContentType(String contentType) {
             this.contentType = contentType;
         }
+        public Long getFileSize() {
+            return filesize;
+        }
+        public void setFileSize(Long fileSize) {
+            this.filesize = fileSize;
+        }
         public String getFilename() {
             return filename;
@@ Expand Down @@

src/main/java/edu/harvard/iq/dataverse/api/dto/DatasetVersionDTO.java

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -12,8 +12,9 @@
  
    public class DatasetVersionDTO {

        String archiveNote;

        String deacessionLink;

        // FIXME: Change to versionNumberMajor and versionNumberMinor? Some partial renaming of "minor" was done.

        Long versionNumber;

        String minorVersionNumber;

        Long versionMinorNumber;

        long id;

        VersionState versionState;

        String releaseDate;

    @@ -36,7 +37,8 @@ public class DatasetVersionDTO {
  
        String availabilityStatus;

        String contactForAccess;

        String sizeOfCollection;

        String studyCompletion; 

        String studyCompletion;

        boolean fileAccessRequest;

        String citation;

        String license;

        boolean inReview;

    @@ -172,7 +174,15 @@ public String getStudyCompletion() {
  
        public void setStudyCompletion(String studyCompletion) {

            this.studyCompletion = studyCompletion;

        }

        public boolean isFileAccessRequest() {

        	return fileAccessRequest;

        }

        public void setFileAccessRequest(boolean fileAccessRequest) {

        	this.fileAccessRequest = fileAccessRequest;

        }

        public String getCitation() {

            return citation;

        }

    @@ -229,12 +239,12 @@ public void setVersionNumber(Long versionNumber) {
  
            this.versionNumber = versionNumber;

        }

        public String getMinorVersionNumber() {

            return minorVersionNumber;

        public Long getMinorVersionNumber() {

            return versionMinorNumber;

        }

        public void setMinorVersionNumber(String minorVersionNumber) {

            this.minorVersionNumber = minorVersionNumber;

        public void setMinorVersionNumber(Long minorVersionNumber) {

            this.versionMinorNumber = minorVersionNumber;

        }

        public long getId() {

    @@ -320,7 +330,7 @@ public List<FieldDTO> getDatasetFields() {
  
        @Override

        public String toString() {

            return "DatasetVersionDTO{" + "archiveNote=" + archiveNote + ", deacessionLink=" + deacessionLink + ", versionNumber=" + versionNumber + ", minorVersionNumber=" + minorVersionNumber + ", id=" + id + ", versionState=" + versionState + ", releaseDate=" + releaseDate + ", lastUpdateTime=" + lastUpdateTime + ", createTime=" + createTime + ", archiveTime=" + archiveTime + ", UNF=" + UNF + ", metadataBlocks=" + metadataBlocks + ", fileMetadatas=" + fileMetadatas + '}';

            return "DatasetVersionDTO{" + "archiveNote=" + archiveNote + ", deacessionLink=" + deacessionLink + ", versionNumber=" + versionNumber + ", minorVersionNumber=" + versionMinorNumber + ", id=" + id + ", versionState=" + versionState + ", releaseDate=" + releaseDate + ", lastUpdateTime=" + lastUpdateTime + ", createTime=" + createTime + ", archiveTime=" + archiveTime + ", UNF=" + UNF + ", metadataBlocks=" + metadataBlocks + ", fileMetadatas=" + fileMetadatas + '}';

        }

src/main/java/edu/harvard/iq/dataverse/api/dto/FileDTO.java

-Original file line number
+Diff line change
@@ -1,9 +1,27 @@
     package edu.harvard.iq.dataverse.api.dto;
     public class FileDTO {
+    	String label;
+    	boolean restricted;
         DataFileDTO dataFile;
+        public String getLabel() {
+        	return label;
+        }
+        public void setLabel(String label) {
+        	this.label = label;
+        }
+        public boolean isRestricted() {
+        	return restricted;
+        }
+        public void setRestricted(boolean restricted) {
+        	this.restricted = restricted;
+        }
         public DataFileDTO getDataFile() {
             return dataFile;
         }
@@ Expand Down @@

src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1154,10 +1154,10 @@ private void parseVersionNumber(DatasetVersionDTO dvDTO, String versionNumber) {
  
            int firstIndex = versionNumber.indexOf('.');

            if (firstIndex == -1) {

                dvDTO.setVersionNumber(Long.parseLong(versionNumber));

                dvDTO.setMinorVersionNumber("0");

                dvDTO.setMinorVersionNumber(0L);

            } else {

                dvDTO.setVersionNumber(Long.parseLong(versionNumber.substring(0, firstIndex - 1)));

                dvDTO.setMinorVersionNumber(versionNumber.substring(firstIndex + 1));

                dvDTO.setMinorVersionNumber(Long.valueOf(versionNumber.substring(firstIndex + 1)));

            }

src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibUtil.java

-Original file line number
+Diff line change
@@ Expand Up @@
          *
          * - "Guido|van Rossum"
          *
-         * - "Philip Seymour|Hoffman"
+         * - "Philip Seymour|Hoffman" (see FirstNameTest.java)
          *
          * Also, we currently compel all Shibboleth IdPs to send us firstName and
          * lastName so the logic to handle null/empty values for firstName and
@@ Expand Down @@

src/main/java/edu/harvard/iq/dataverse/export/ExportService.java

-Original file line number
+Diff line change
@@ Expand Up @@
             try {
                 dataAccess = DataAccess.getStorageIO(dataset);
             } catch (IOException ioex) {
-                throw new IOException("IO Exception thrown exporting as " + "export_" + formatName + ".cached");
+                throw new IOException("IO Exception thrown exporting as " + "export_" + formatName + ".cached", ioex);
             }
             InputStream cachedExportInputStream = null;
@@ Expand All @@
                 cachedExportInputStream = dataAccess.getAuxFileAsInputStream("export_" + formatName + ".cached");
                 return cachedExportInputStream;
             } catch (IOException ioex) {
-                throw new IOException("IO Exception thrown exporting as " + "export_" + formatName + ".cached");
+                throw new IOException("IO Exception thrown exporting as " + "export_" + formatName + ".cached", ioex);
             }
         }
@@ Expand Down @@

src/main/java/edu/harvard/iq/dataverse/export/OpenAireExporter.java

-Original file line number
+Diff line change
@@ -0,0 +1,75 @@
+    package edu.harvard.iq.dataverse.export;
+    import java.io.OutputStream;
+    import javax.json.JsonObject;
+    import javax.xml.stream.XMLStreamException;
+    import com.google.auto.service.AutoService;
+    import edu.harvard.iq.dataverse.DatasetVersion;
+    import edu.harvard.iq.dataverse.export.openaire.OpenAireExportUtil;
+    import edu.harvard.iq.dataverse.export.spi.Exporter;
+    import edu.harvard.iq.dataverse.util.BundleUtil;
+    @AutoService(Exporter.class)
+    public class OpenAireExporter implements Exporter {
+        public OpenAireExporter() {
+        }
+        @Override
+        public String getProviderName() {
+            return "oai_datacite";
+        }
+        @Override
+        public String getDisplayName() {
+            return BundleUtil.getStringFromBundle("dataset.exportBtn.itemLabel.dataciteOpenAIRE");
+        }
+        @Override
+        public void exportDataset(DatasetVersion version, JsonObject json, OutputStream outputStream)
+                throws ExportException {
+            try {
+                OpenAireExportUtil.datasetJson2openaire(json, outputStream);
+            } catch (XMLStreamException xse) {
+                throw new ExportException("Caught XMLStreamException performing DataCite OpenAIRE export", xse);
+            }
+        }
+        @Override
+        public Boolean isXMLFormat() {
+            return true;
+        }
+        @Override
+        public Boolean isHarvestable() {
+            return true;
+        }
+        @Override
+        public Boolean isAvailableToUsers() {
+            return true;
+        }
+        @Override
+        public String getXMLNameSpace() throws ExportException {
+            return OpenAireExportUtil.RESOURCE_NAMESPACE;
+        }
+        @Override
+        public String getXMLSchemaLocation() throws ExportException {
+            return OpenAireExportUtil.RESOURCE_SCHEMA_LOCATION;
+        }
+        @Override
+        public String getXMLSchemaVersion() throws ExportException {
+            return OpenAireExportUtil.SCHEMA_VERSION;
+        }
+        @Override
+        public void setParam(String name, Object value) {
+            // not used
+        }
+    }

src/main/java/edu/harvard/iq/dataverse/export/openaire/Cleanup.java

-Original file line number
+Diff line change
@@ -0,0 +1,28 @@
+    package edu.harvard.iq.dataverse.export.openaire;
+    import org.apache.commons.lang3.StringUtils;
+    /**
+     *
+     * @author francesco.cadili@4science.it
+     */
+    public class Cleanup {
+        /**
+         * Normalize sentence
+         *
+         * @param sentence full name or organization name
+         * @return normalize string value
+         */
+        static public String normalize(String sentence) {
+            if (StringUtils.isBlank(sentence)) {
+                return "";
+            }
+            sentence = sentence.trim()
+                    .replaceAll(", *", ", ")
+                    .replaceAll(" +", " ");
+            return sentence;
+        }
+    }

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenAIRE guidelines for Data Archives #4664

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!