Merge branch 'develop' into 8305-payara6-ee10-v3 #8305

Conflicts:
doc/sphinx-guides/source/admin/make-data-count.rst
src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java

conf/docker-aio/readme.md

@@ -1,7 +1,7 @@
 # Docker All-In-One
 
 > :information_source:  **NOTE: Sunsetting of this module is imminent.** There is no schedule yet, but expect it to go away.
-> Please let the [Dataverse Containerization Working Group](https://dc.wgs.gdcc.io) know if you are a user and
+> Please let the [Dataverse Containerization Working Group](https://ct.gdcc.io) know if you are a user and
 > what should be preserved.
 
 First pass docker all-in-one image, intended for running integration tests against.

doc/release-notes/9704-GeotifShp.md

@@ -0,0 +1 @@
+Two file previewers for GeoTIFF and Shapefiles are now available for visualizing geotiff image files and zipped Shapefiles on a map.

doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv

@@ -2,5 +2,5 @@ Tool	Type	Scope	Description
 Data Explorer	explore	file	A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/dataverse-data-explorer-v2 for the instructions on adding Data Explorer to your Dataverse. 
 Whole Tale	explore	dataset	A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide <https://wholetale.readthedocs.io/en/stable/users_guide/integration.html>`_.
 Binder	explore	dataset	Binder allows you to spin up custom computing environments in the cloud (including Jupyter notebooks) with the files from your dataset. `Installation instructions <https://github.com/data-exp-lab/girder_ythub/issues/10>`_ are in the Data Exploration Lab girder_ythub project. See also :ref:`binder`.
-File Previewers	explore	file	A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, zip, HDF5, NetCDF, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers
+File Previewers	explore	file	A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, GeoTIFF, Shapefile, zip, HDF5, NetCDF, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers
 Data Curation Tool	configure	file	A GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Curation-Tool for the installation instructions.

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -73,6 +73,7 @@ Enable or Disable Display of Make Data Count Metrics
 By default, when MDC logging is enabled (when ``:MDCLogPath`` is set), your Dataverse installation will display MDC metrics instead of it's internal (legacy) metrics. You can avoid this (e.g. to collect MDC metrics for some period of time before starting to display them) by setting ``:DisplayMDCMetrics`` to false.
 
 The following discussion assumes ``:MDCLogPath`` has been set to ``/usr/local/payara6/glassfish/domains/domain1/logs/mdc``
+You can also decide to display MDC metrics along with Dataverse's traditional download counts from the time before MDC was enabled. To do this, set the :ref:`:MDCStartDate` to when you started MDC logging.
 
 Configure Counter Processor
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/api/native-api.rst

@@ -298,6 +298,8 @@ The fully expanded example above (without environment variables) looks like this
 
   curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/metadatablockfacets/isRoot -d 'true'
 
+.. _create-role-in-collection:
+
 Create a New Role in a Dataverse Collection
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -317,16 +319,7 @@ The fully expanded example above (without environment variables) looks like this
 
   curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/roles --upload-file roles.json
 
-Where ``roles.json`` looks like this::
-
-  {
-    "alias": "sys1",
-    "name": “Restricted System Role”,
-    "description": “A person who may only add datasets.”,
-    "permissions": [
-      "AddDataset"
-    ]
-  } 
+For ``roles.json`` see :ref:`json-representation-of-a-role`
 
 .. note:: Only a Dataverse installation account with superuser permissions is allowed to create roles in a Dataverse Collection.
 
@@ -3094,26 +3087,14 @@ Optionally, you may use a third query parameter "sendEmailNotification=false" to
 Roles
 -----
 
-Create a New Role in a Dataverse Collection
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Creates a new role under Dataverse collection ``id``. Needs a json file with the role description:
-
-.. code-block:: bash
-
-  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-  export SERVER_URL=https://demo.dataverse.org
-  export ID=root
-
-  curl -H X-Dataverse-key:$API_TOKEN -X POST -H "Content-type:application/json" $SERVER_URL/api/dataverses/$ID/roles --upload-file roles.json
+A role is a set of permissions.
 
-The fully expanded example above (without environment variables) looks like this:
-
-.. code-block:: bash
+.. _json-representation-of-a-role:
 
-  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -H "Content-type:application/json" https://demo.dataverse.org/api/dataverses/root/roles --upload-file roles.json
+JSON Representation of a Role
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Where ``roles.json`` looks like this::
+The JSON representation of a role (``roles.json``) looks like this::
 
   {
     "alias": "sys1",
@@ -3124,8 +3105,12 @@ Where ``roles.json`` looks like this::
     ]
   } 
 
-.. note:: Only a Dataverse installation account with superuser permissions is allowed to create roles in a Dataverse Collection.
+.. note:: alias is constrained to a length of 16 characters
+
+Create Role
+~~~~~~~~~~~
 
+Roles can be created globally (:ref:`create-global-role`) or for individual Dataverse collections (:ref:`create-role-in-collection`).
 
 Show Role
 ~~~~~~~~~
@@ -3986,12 +3971,24 @@ List all global roles in the system. ::
 
     GET http://$SERVER/api/admin/roles
 
+.. _create-global-role:
+
 Create Global Role
 ~~~~~~~~~~~~~~~~~~
 
 Creates a global role in the Dataverse installation. The data POSTed are assumed to be a role JSON. ::
 
     POST http://$SERVER/api/admin/roles
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=root
+
+  curl -H X-Dataverse-key:$API_TOKEN -X POST $SERVER_URL/api/admin/roles --upload-file roles.json
+
+``roles.json`` see :ref:`json-representation-of-a-role`
 
 Delete Global Role
 ~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/developers/testing.rst

@@ -175,7 +175,7 @@ Running the full API test suite using Docker
 
 .. note::
     Sunsetting of this module is imminent.** There is no schedule yet, but expect it to go away.
-    Please let the `Dataverse Containerization Working Group <https://dc.wgs.gdcc.io>`_ know if you are a user and
+    Please let the `Dataverse Containerization Working Group <https://ct.gdcc.io>`_ know if you are a user and
     what should be preserved.
 
 To run the full suite of integration tests on your laptop, we recommend using the "all in one" Docker configuration described in ``conf/docker-aio/readme.md`` in the root of the repo.

doc/sphinx-guides/source/installation/config.rst

@@ -2460,7 +2460,7 @@ please find all known feature flags below. Any of these flags can be activated u
       - Description
       - Default status
     * - api-session-auth
-      - Enables API authentication via session cookie (JSESSIONID). **Caution: Enabling this feature flag exposes the installation to CSRF risks!** We expect this feature flag to be temporary (only used by frontend developers, see `#9063 <https://github.com/IQSS/dataverse/issues/9063>`_) and removed once support for bearer tokens has been implemented (see `#9229 <https://github.com/IQSS/dataverse/issues/9229>`_).
+      - Enables API authentication via session cookie (JSESSIONID). **Caution: Enabling this feature flag exposes the installation to CSRF risks!** We expect this feature flag to be temporary (only used by frontend developers, see `#9063 <https://github.com/IQSS/dataverse/issues/9063>`_) and for the feature to be removed in the future.
       - ``Off``
 
 **Note:** Feature flags can be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
@@ -3469,6 +3469,20 @@ Sets the path where the raw Make Data Count logs are stored before being process
 
 ``curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:DisplayMDCMetrics``
 
+.. _:MDCStartDate:
+
+:MDCStartDate
++++++++++++++
+
+It is possible to display MDC metrics (as of the start date of MDC logging) along with legacy download counts, generated before MDC was enabled.
+This is enabled via the new setting `:MDCStartDate` that specifies the cut-over date. If a dataset has any legacy access counts collected prior to that date, those numbers will be displayed in addition to the MDC views and downloads recorded since then.
+(Nominally, this date should be when your installation started logging MDC metrics but it can be any date after that if desired.)
+
+
+``curl -X PUT -d '2019-10-01' http://localhost:8080/api/admin/settings/:MDCStartDate``
+
+
+
 .. _:Languages:
 
 :Languages

doc/sphinx-guides/source/user/dataset-management.rst

@@ -208,6 +208,8 @@ Previewers are available for the following file types:
 - Zip (preview and extract/download)
 - HTML
 - GeoJSON
+- GeoTIFF
+- Shapefile
 - NetCDF/HDF5
 - Hypothes.is
 
@@ -343,6 +345,20 @@ GeoJSON
 
 A map will be shown as a preview of GeoJSON files when the previewer has been enabled (see :ref:`file-previews`). See also a `video demo <https://www.youtube.com/watch?v=EACJJaV3O1c&t=588s>`_ of the GeoJSON previewer by its author, Kaitlin Newson.
 
+.. _geotiff:
+
+GeoTIFF
+-------
+
+A map is also displayed as a preview of GeoTiFF image files, whose previewer must be enabled (see :ref:`file-previews`). Since GeoTIFFs do not have their own mimetype, it is advisable to use this previewer only when GeoTIFFs are used (and not "normal" TIFs). For performance reasons, this previewer has a file size limit of 15 MB and a row/column limit of 50,000 so that larger files are not loaded.
+
+.. _shapefile:
+
+Shapefile
+---------
+
+Another previewer can be enabled for shapefiles (see :ref:`file-previews`). This previewer only works with zipped shapefiles (see :doc:`/developers/geospatial`). A file size limit of 20 MB is set for this previewer (also because of performance reasons).
+
 .. _netcdf-and-hdf5:
 
 NetCDF and HDF5

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -2167,14 +2167,19 @@ private void displayPublishMessage(){
         }               
     }
 
+    Boolean valid = null;
+
     public boolean isValid() {
-        DatasetVersion version = dataset.getLatestVersion();
-        if (!version.isDraft()) {
-            return true;
+        if (valid == null) {
+            DatasetVersion version = dataset.getLatestVersion();
+            if (!version.isDraft()) {
+                valid = true;
+            }
+            DatasetVersion newVersion = version.cloneDatasetVersion();
+            newVersion.setDatasetFields(newVersion.initDatasetFields());
+            valid = newVersion.isValid();
         }
-        DatasetVersion newVersion = version.cloneDatasetVersion();
-        newVersion.setDatasetFields(newVersion.initDatasetFields());
-        return newVersion.isValid();
+        return valid;
     }
 
     public boolean isValidOrCanReviewIncomplete() {
@@ -6099,16 +6104,23 @@ public String getWebloaderUrlForDataset(Dataset d) {
 
     /**
      * Add Signposting
+     * 
      * @return String
      */
+
+    String signpostingLinkHeader = null;
+
     public String getSignpostingLinkHeader() {
         if (!workingVersion.isReleased()) {
             return null;
         }
-        SignpostingResources sr = new SignpostingResources(systemConfig, workingVersion,
-                JvmSettings.SIGNPOSTING_LEVEL1_AUTHOR_LIMIT.lookupOptional().orElse(""),
-                JvmSettings.SIGNPOSTING_LEVEL1_ITEM_LIMIT.lookupOptional().orElse(""));
-        return sr.getLinks();
+        if (signpostingLinkHeader == null) {
+            SignpostingResources sr = new SignpostingResources(systemConfig, workingVersion,
+                    JvmSettings.SIGNPOSTING_LEVEL1_AUTHOR_LIMIT.lookupOptional().orElse(""),
+                    JvmSettings.SIGNPOSTING_LEVEL1_ITEM_LIMIT.lookupOptional().orElse(""));
+            signpostingLinkHeader = sr.getLinks();
+        }
+        return signpostingLinkHeader;
     }
 
 }

src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java

@@ -7,6 +7,7 @@
 import edu.harvard.iq.dataverse.FileMetadata;
 import edu.harvard.iq.dataverse.TermsOfUseAndAccess;
 import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress;
+import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
 import static edu.harvard.iq.dataverse.dataaccess.DataAccess.getStorageIO;
 import edu.harvard.iq.dataverse.dataaccess.StorageIO;
@@ -497,18 +498,34 @@ public static Long getDownloadSizeNumericBySelectedFiles(List<FileMetadata> file
 
     public static boolean validateDatasetMetadataExternally(Dataset ds, String executable, DataverseRequest request) {
         String sourceAddressLabel = "0.0.0.0"; 
+        String userIdentifier = "guest";
 
         if (request != null) {
             IpAddress sourceAddress = request.getSourceAddress();
             if (sourceAddress != null) {
                 sourceAddressLabel = sourceAddress.toString();
             }
+
+            AuthenticatedUser user = request.getAuthenticatedUser();
+
+            if (user != null) {
+                userIdentifier = user.getUserIdentifier();
+            }
         }
 
         String jsonMetadata; 
 
+        // We are sending the dataset metadata encoded in our standard json 
+        // format, with a couple of extra elements added, such as the ids of 
+        // the home collection and the user, in order to make it easier 
+        // for the filter to whitelist by these attributes. 
+
         try {
-            jsonMetadata = json(ds).add("datasetVersion", json(ds.getLatestVersion())).add("sourceAddress", sourceAddressLabel).build().toString();
+            jsonMetadata = json(ds).add("datasetVersion", json(ds.getLatestVersion()))
+                    .add("sourceAddress", sourceAddressLabel)
+                    .add("userIdentifier", userIdentifier)
+                    .add("parentAlias", ds.getOwner().getAlias())
+                    .build().toString();
         } catch (Exception ex) {
             logger.warning("Failed to export dataset metadata as json; "+ex.getMessage() == null ? "" : ex.getMessage());
             return false; 
@@ -524,7 +541,7 @@ public static boolean validateDatasetMetadataExternally(Dataset ds, String execu
         try {
             File tempFile = File.createTempFile("datasetMetadataCheck", ".tmp");
             FileUtils.writeStringToFile(tempFile, jsonMetadata);
-                                    
+
             // run the external executable: 
             String[] params = { executable, tempFile.getAbsolutePath() };
             Process p = Runtime.getRuntime().exec(params);

src/main/java/edu/harvard/iq/dataverse/dataverse/DataverseUtil.java

@@ -4,6 +4,7 @@
 import edu.harvard.iq.dataverse.Dataverse;
 import edu.harvard.iq.dataverse.DvObjectContainer;
 import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress;
+import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.authorization.users.User;
 import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
 import edu.harvard.iq.dataverse.util.BundleUtil;
@@ -37,16 +38,32 @@ public static boolean validateDataverseMetadataExternally(Dataverse dv, String e
         String jsonMetadata;
 
         String sourceAddressLabel = "0.0.0.0";
+        String userIdentifier = "guest";
+        String parentAlias = dv.getOwner() == null ? "" : dv.getOwner().getAlias();
 
         if (request != null) {
             IpAddress sourceAddress = request.getSourceAddress();
             if (sourceAddress != null) {
                 sourceAddressLabel = sourceAddress.toString();
             }
+            AuthenticatedUser user = request.getAuthenticatedUser();
+
+            if (user != null) {
+                userIdentifier = user.getUserIdentifier();
+            }
         }
-
+
+        // We are sending the collection metadata encoded in our standard json 
+        // format, with a couple of extra elements added, such as the id of 
+        // the user sending the request and the alias of the parent collection, 
+        // in order to make it easier for the filter to manage whitelisting. 
+
         try {
-            jsonMetadata = json(dv).add("sourceAddress", sourceAddressLabel).build().toString();
+            jsonMetadata = json(dv)
+                    .add("sourceAddress", sourceAddressLabel)
+                    .add("userIdentifier", userIdentifier)
+                    .add("parentAlias", parentAlias)
+                    .build().toString();
         } catch (Exception ex) {
             logger.warning(
                     "Failed to export dataverse metadata as json; " + ex.getMessage() == null ? "" : ex.getMessage());

src/main/java/edu/harvard/iq/dataverse/export/ddi/DdiExportUtil.java

@@ -39,6 +39,7 @@
 import jakarta.json.JsonArray;
 import jakarta.json.JsonArrayBuilder;
 import jakarta.json.JsonObject;
+import jakarta.json.JsonString;
 import jakarta.json.JsonValue;
 import javax.xml.stream.XMLOutputFactory;
 import javax.xml.stream.XMLStreamException;
@@ -1895,7 +1896,7 @@ private static void createVarDDI(XMLStreamWriter xmlw, JsonObject dvar, String f
             for (Entry<String, JsonValue> sumStat : dvar.getJsonObject("summaryStatistics").entrySet()) {
                 xmlw.writeStartElement("sumStat");
                 writeAttribute(xmlw, "type", sumStat.getKey());
-                xmlw.writeCharacters(sumStat.getValue().toString());
+                xmlw.writeCharacters(((JsonString)sumStat.getValue()).getString());
                 xmlw.writeEndElement(); // sumStat
             }
         }

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

@@ -696,7 +696,12 @@ public static JsonObjectBuilder json(DataFile df, FileMetadata fileMetadata, boo
          * InternalExportDataProvider fileDetails.
          */
         if (forExportDataProvider) {
-            builder.add("restricted", df.isRestricted());
+            builder.add("restricted", df.isRestricted())
+            .add("fileMetadataId", fileMetadata.getId())
+            .add("dataTables", df.getDataTables().isEmpty() ? null : JsonPrinter.jsonDT(df.getDataTables()))
+            .add("varGroups", fileMetadata.getVarGroups().isEmpty()
+                    ? JsonPrinter.jsonVarGroup(fileMetadata.getVarGroups())
+                    : null);
         }
         return builder;
     }