Skip to content

v6.12

Latest
Latest

Choose a tag to compare

View all tags
@pdurbin pdurbin released this 17 Sep 19:08
v6.12
9769107
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Dataverse 6.12

Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.12 rather than the list of releases, which will cut them off.

This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project!

Release Highlights

Highlights for Dataverse 6.12 include:

  • HTML is now supported in the description of files. See #11397 and #12318.
  • "Translator" is now available as a "Type" under the "Contributor" field. See #12346.
  • External vocabulary scripts now support the use of multiple PID services per field. For example, the Author field can allow both an ORCID value and a ROR value. See #12331.
  • New and improved APIs
  • Bug fixes
  • Security fixes
  • Performance improvements

Features Added

These are features that weren't already mentioned under "highlights" above.

  • The performance when saving counts/guestbook responses has been improved (particularly for datasets with large numbers of files). See #12479.
  • A public auxiliary file can now be previewed even its associated data file is restricted. An example use case is a previewer for redacted REFI-QDAS files, as presented by QDR at the 2026 Dataverse Community Meeting. See #12190.
  • When users edit a dataset that has missing required metadata, the page scrolls to the first case of missing metadata, making it easier for users to see what fields they need to fill in order to save and publish their datasets. See #2190 and #12634.

Bug Fixes

  • A bug that allowed required custom questions in guestbooks to be optional has been fixed. See #12483 and #12501.
  • Preview URL users could not download files from the dataset being previewed if a guestbook was assigned to that dataset. This is now fixed. A similar issue with Locally FAIR content is also fixed. See #12535, #12579, and #12479.
  • Fixed an issue where, in some cases, the license acceptance popup for the archival format option did not display correctly when downloading a complete dataset. See #12411 and #12412.
  • Previewing a draft file with a retention period was failing but has been fixed. See #12433 and #12434.
  • When uploading a provenance JSON file, it was not clear in the user interface or the guides that only PROV-JSON is accepted. Error handling has been improved to explain this, the preview button has been disabled for invalid formats, and the guides have been updated. See the guides, #8613, #9342, and #12468.
  • Users without permission to edit a dataset or collection were not able to access the permission and group management pages via the dataset or collection page menus even if they had the relevant "manage permissions" permission, set up via a custom role. Previously, the equivalent "edit" dataset or collection permission was required. See #3981, #12218, and #12503.
  • The guides have explained how public NcML auxiliary files are automatically generated from NetCDF and HDF5 files since the feature was introduced in #9239 but now they explicitly state that this happens for restricted files as well. See #12190.
  • The creator of a featured item can now view and download images when the collection has not yet been published. (The ViewUnpublishedDataverse permission is not required for the creator. Being the creator is enough.) See #11537 and #12381.
  • PUT /api/datasets/{datsetId}/{guestbookBody} was failing for guestbook IDs over 127. This has been fixed. See #12556 and #12592.
  • In the context of creating or importing datasets via API, it was not possible to serialize a dataset as JSON and then recreate it due to a mismatch between releaseTime and releaseDate fields. For backward compatibility, on create or import, the original releaseDate field will continue to be allowed and will continue to be truncated to date only. On create and import, releaseTime is now supported and preferred and no truncation occurs. See #11594 and #12320.
  • /api/guestbooks/{id}/responses was returning incorrect stats in the pagination block. The counts now include the full hierarchy of parent collections. See #12523 and #12557.
  • The /api/admin/requestSignedUrl API works again for URLs containing special characters such as the : in persistentId query parameters. This restores functionality accidentally broken in Dataverse 6.10. See #12435.

Other Changes

  • Payara 7.2026.8 is now the supported and recommended version. See upgrade instructions below and #12498. We are aware that Payara 7.2026.9 is out but it didn't "just work" so we don't recommend it yet. See #12715.
  • Database indexes were added to the guestbookresponse table to speed up queries. See #12616.
  • The QA Guide was added as an experiment in #10103 but has been removed because it has proved less useful than imagined. It can still be viewed as it existed as of Dataverse 6.11. See #12582.

API Updates

  • The quality of the OpenAPI document has been greatly improved. See the guides and #12439.
  • Added the ability to set and view the guestbookRoot setting of a collection. See the guides, #12459, and #12611.
  • POST /dataverses/{identifier}/datasets payload JSON now allows for "templateId":$templateId to be included to set the template on the new dataset. See #12132 and #12405.

Security Updates

This release contains important security updates. If you are not receiving security advisories, please sign up by following the steps in the guides (email support@dataverse.org to ask for an invite). These advisories are sent through the dataverse-security Google Group, as announced on the mailing list.

We would like to thank Huynh Kien Minh and Santosh Kumar Puppala for notifying us about vulnerabilities that were fixed in this release.

Backward Incompatible Changes

Generally speaking, see the API Changelog for a list of backward-incompatible API changes.

  • The message field in API responses from certain endpoints (see the API Changelog for a list) was incorrectly returned as a nested object ({"message": {"message": "..."}}) instead of a plain string ({"message": "..."}). This has been fixed. To preserve backward compatibility, you can set dataverse.legacy.api-response-message-style to true for now, but we plan to remove this flag in a future version. You can also preview a potential new format where the message is nested in the "data" object by enabling dataverse.feature.unify-api-response-message-style. See #12096 and #12097.
  • The default license will automatically be applied when datasets are created via API and do not contain a license. You can opt-out of this new behavior by setting dataverse.feature.do-not-assume-default-license to true. See the guides, #11161, and #12551.
  • The /api/datafile/{fileId}/metadata/ddi API endpoint now returns 403/Forbidden rather than 400/Bad Request when the caller can't access the file due to permissions, embargo, or retention period (consistent with other file access APIs). See #12642.
  • Whether file extensions are included in the "Content-disposition" header returned when downloading auxiliary files depends on whether the relevant format is one of the content types supported in Tika. A recent update to the version of Tika has added new content types, including "text/markdown", and auxiliary files with these types now have a a file extension included (e.g. ".md" in this case) in the header. See #12498.
  • Permissions reindexing was undocumented, as described below, but now requires POST and superuser access. See #12200.

Notes for Dataverse Installation Administrators

Export Improvements

An optional query parameter formats has been added to the reExportAll and reExportDataset, allowing an administrator to force re-export of only the formats specified (e.g. formats=Datacite,croissant). See the guides and #11799.

In addition, DDI export should be more reliable and memory-efficient. See #11405, #11767, #11799 and #12642.

The DataCite XML metadata format was updated to:

  • indicate compliance with the version 4.7 schema,
  • add support for specifying a "Translator" contributor,
  • add a valueURI attribute to a subject element when a value exists in the keywordTermURI field,
  • add a language element when a dataset has one language defined in its Citation block metadata,
  • accept dates of the form YYYY or YYYY-MM in the timePeriodCovered and dateOfCollection fields, and
  • avoids sending the word "null" as part of a date range when the start or end date is unspecified.

See #12346.

Permissions Reindexing

The previously undocumented Solr permissions reindexing API endpoints have been improved.

  • The endpoints are /api/admin/index/perms (asynchronous, all objects) and /api/admin/index/perms/{id} (synchronous, single object) now use POST instead of GET.
  • Both endpoints require superuser access.
  • For the asynchronous reindex all endpoint, if an indexing process is already in progress, the API will return a 409 Conflict status.
  • The asynchronous reindex all endpoint no longer runs as a single transaction. This avoids potential timeouts in larger installations.
  • These endpoints are now documented.

See the guides and #12200.

Improved JSON Performance

A Jakarta JSON-P provider is now reused for every builder or value creation. This improves the performance of large JSON operations, especially large exports and API responses, without changing the existing behavior or the JSON output. See #12362 and #12541.

Ceph Object Gateway Support

A new configuration setting has been introduced for S3 compatible storage drivers that addresses an incompatibility between the AWS S3 library used in Dataverse and certain S3 implementations such as the Ceph Object Gateway:
dataverse.files.<id>.disable-multipart-download-for-indirect-download (default: false).

When set to true, multipart download is disabled for the specified S3 driver, forcing the server to handle part reassembly and avoiding the incompatible headers. This is recommended for Ceph-backed S3 storage if 412 errors are encountered during download when dataverse.files.<id>.download-redirect is set to false.

See the guides, #12413, and #12660.

Signed URLs Across Multiple App Servers

A new, optional dataverse.api.signing-secret setting has been added for installations that run multiple app servers. When enabled, the signing secret is persisted to disk rather than only existing in memory on a particular app server. This solves the problem of a user getting an error when trying to make use of a signed URL that was generated on the "wrong" app server. See dataverse.api.signing-secret in the guides and #12435.

New Settings

New JVM Options (MicroProfile Config Settings)

  • dataverse.api.signing-secret
  • dataverse.feature.do-not-assume-default-license
  • dataverse.feature.unify-api-response-message-style
  • dataverse.files.<id>.disable-multipart-download-for-indirect-download
  • dataverse.legacy.api-response-message-style

New Database Settings

Updates for Translators

  • It is now possible to translate password reset email messages. See #12536 and #12544.

Updates for Developers

  • For external tools, an allowedApiCalls urlTemplate should not use any of Dataverse's reserved authentication-related query parameters until, user, method, token, key or signed: launching a tool whose manifest includes them will be rejected with an error naming the parameter. See the guides and #12435.
  • If you are adding or updating Dataverse APIs you are now required to add OpenAPI annotations. See the guides and #12439.
  • We have refactored the metadata export code to use a new and improved Data Export framework. The relevant interface and the accompanying classes have been moved out of the main Dataverse repository and into a dedicated dataverse-spi project. As mentioned above, this will result in a measurable improvement when exporting the DDI format, including unnecessary database queries. It should be possible to take advantage of this refactoring to improve the exports of other data-rich formats in future releases. See #11405, #11767, #11799 and #12642.
  • Preliminary support for Podman, an alternative to Docker, has been added. See #12513.

Complete List of Changes

For the complete list of code changes in this release, see the 6.12 milestone in GitHub.

Getting Help

For help with upgrading, installing, or general questions please see getting help in the Installation Guide.

Installation

If this is a new installation, please follow our Installation Guide. Please don't be shy about asking for help if you need it!

Once you are in production, we would be delighted to update our map of Dataverse installations around the world to include yours! Please create an issue or email us at support@dataverse.org to join the club!

You are also very welcome to join the Global Dataverse Community Consortium (GDCC).

Upgrade Instructions

Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc.

Note: These instructions assume that you are upgrading from the immediate previous version. That is to say, you've already upgraded through all the 6.x releases and are now running Dataverse 6.11. See tags on GitHub for a list of versions. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version.

If you are running Payara as a non-root user (and you should be!), remember not to execute the commands below as root. By default, Payara runs as the dataverse user. In the commands below, we use sudo to run the commands as a non-root user.

Also, we assume that Payara is installed in /usr/local/payara7. If not, adjust as needed.

The instructions below describe the upgrade procedure based on moving your existing Payara 7.2026.2 domain directory into the new Payara 7.2026.8 distribution. We recommend this method because it is the easiest way to recreate your current configuration and preserve your data.

  1. Undeploy Dataverse, if deployed, using the unprivileged service account ("dataverse", by default).

    The new version of Payara is not compatible with previous versions of Dataverse so you must undeploy the running Dataverse 6.11 war file.

    sudo -u dataverse /usr/local/payara7/bin/asadmin list-applications

sudo -u dataverse /usr/local/payara7/bin/asadmin undeploy dataverse-6.11

  2. Stop Payara.

    sudo systemctl stop payara

  3. Move the current Payara 7.2026.2 directory out of the way.

    sudo mv /usr/local/payara7 /usr/local/payara7-2026.2

  4. Download the new Payara version 7.2026.8, and unzip it.

    curl -L -O https://nexus.payara.fish/repository/payara-community/fish/payara/distributions/payara/7.2026.8/payara-7.2026.8.zip

sudo unzip payara-7.2026.8.zip -d /usr/local/

  5. Set permission for the service account ("dataverse" by default).

    sudo chown -R root:root /usr/local/payara7

sudo chown dataverse /usr/local/payara7/glassfish/lib

sudo chown -R dataverse:dataverse /usr/local/payara7/glassfish/domains/domain1

  6. Replace the brand new payara7/glassfish/domains/domain1 with your old, preserved domain1.

    sudo mv /usr/local/payara7/glassfish/domains/domain1 /usr/local/payara7/glassfish/domains/domain1_DIST

sudo mv /usr/local/payara7-2026.2/glassfish/domains/domain1 /usr/local/payara7/glassfish/domains/

    You may want

  7. Remove the cache directories.

    sudo rm -rf /usr/local/payara7/glassfish/domains/domain1/generated/

sudo rm -rf /usr/local/payara7/glassfish/domains/domain1/osgi-cache/

  8. Update configuration files in the moved domain.

    The file glassfish-acc.xml from Payara 7.2026.2 has been renamed to payara-acc.xml in Payara 7.2026.8. Also default-logging.properties has been updated in the Payara 7.2026.8. We recommend copying these new files from the Payara 7.2026.8 distribution domain into the domain you copied over from your Payara 7.2026.2 installation.

    sudo rm /usr/local/payara7/glassfish/domains/domain1/config/glassfish-acc.xml

sudo cp /usr/local/payara7/glassfish/domains/domain1_DIST/config/payara-acc.xml /usr/local/payara7/glassfish/domains/domain1/config/

sudo cp /usr/local/payara7/glassfish/domains/domain1_DIST/config/default-logging.properties /usr/local/payara7/glassfish/domains/domain1/config/

  9. Double check request-timeout-seconds setting.

    The Dataverse 6.10 release notes explained how to upgrade from Payara 6 to 7 and mentioned that any other settings that were customized also need to be migrated. Several examples were given but they did not originally include the commonly-edited "request-timeout-seconds" setting for the number of seconds before a request times out. The 6.10 and 6.11 release notes have been updated to now include this example.

    It may be a good idea to back up the domain.xml file before proceeding to make any changes. For example:

    cd /usr/local/payara7/glassfish/domains/domain1/config
cp domain.xml domain.xml.BACKUP

    As described in the guides you might want to shorten or lengthen this value. The default is 900 seconds (15 minutes). If you set it to an hour using the command from the guides (asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds=3600) you should see a line like this in your domain.xml file:

    <http request-timeout-seconds="3600" max-connections="250" default-virtual-server="server">

    See also #12543 and #12606.

  10. Start Payara.

    sudo systemctl start payara

  11. Deploy the Dataverse 6.12 war file.

    wget https://github.com/IQSS/dataverse/releases/download/v6.12/dataverse-6.12.war

sudo -u dataverse /usr/local/payara7/bin/asadmin deploy dataverse-6.12.war

    Please note: when watching logs while waiting for the deployment to finish, you may encounter SEVERE level warnings like "Leaking injection manager found in the current thread. Very likely because it wasn't removed in a previous invocation and leaked into this invocation." We have not found severe issues during testing this release and advise to ignore but monitor for unusual resource usage. We plan to address the underlying issue with the next release. See #12715 for more details.

  12. Update metadata blocks.

    In #12346 we added the value "Translator" under "contributorType" field and we need to reload the citation.tsv file because of this.

    Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages).

    wget https://raw.githubusercontent.com/IQSS/dataverse/v6.12/scripts/api/data/metadatablocks/citation.tsv

curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv

  13. Ensure API endpoints are blocked.

    It is very important for the "admin" API endpoint to be blocked. Additionally, we recommend blocking the "builtin-users" endpoint. See the guides for details.

    In the command below, replace "demo.dataverse.org" with the name of your server. Do the check remotely, not from the server itself.

    curl https://demo.dataverse.org/api/admin/settings

    If you can see your settings, follow the instructions in the guides to block API endpoints.

    Please note that as reported in #12232, versions of the guides from 6.7 through 6.11 incorrectly described how to configure the setting dataverse.api.blocked.endpoints. This was fixed in pull request #12636 for the 6.12 guides. The following is the correct command to use:

    asadmin create-jvm-options '-Ddataverse.api.blocked.endpoints=admin,builtin-users'

    That is, the comma-separated list should be "admin,builtin-users" and not "api/admin,api/builtin-users" as described in previous versions of the guides.

Assets 4
Loading