v4.6

@kcondon kcondon released this Dec 9, 2016

Overview:

This release introduces a new File Landing Page and several important enhancements:

  • Introduce a new File Landing Page
  • Improve Deaccession Behavior
  • API clean up
  • OAI-PMH compliance improvements
  • Optionally support SHA1 in place of MD5 checksum
  • Fix an important issue with Request Access workflow
  • Improve File Upload behavior, particularly drag and drop
  • Document how to run Dataverse with SELinux enabled

For the complete list of issues, see the 4.6 milestone in Github.

For help with upgrading, installing, or general questions please email support@dataverse.org.

Installation:

If this is a new installation, please see our Installation Guide.

Upgrade:

If you are upgrading from v4.x, you must upgrade to each intermediate version before installing this version. When upgrading from the previous version, you will need to do the following:

  1. Undeploy the previous version.
    • /glassfish4/bin/asadmin list-applications
    • /glassfish4/bin/asadmin undeploy dataverse
  2. Stop glassfish and remove the generated directory
    • service glassfish stop
    • remove the generated directory: rm -rf /usr/local/glassfish4/glassfish/domains/domain1/generated
  3. If not yet present, add a jvm option to /usr/local/glassfish4/glassfish/domains/domain1/config/domain.xml to support timers, start glassfish
  4. Deploy this version.
    • /glassfish4/bin/asadmin deploy dataverse-4.6.war
  5. Run the database update script.
    psql -U -d -f upgrade_v4.5.1_to_v4.6.sql
  6. Update tool tip in metadata block.
    • Download the attached, latest social science metadata block file (social_science.tsv) to your glassfish server:
    • Update metadata block with latest .tsv file. From glassfish server in directory where .tsv was downloaded:
  7. Update schema.xml.
    • Stop running solr process (kill -9 )
    • Replace current schema.xml file with latest attached version.
    • Restart solr (java -jar start.jar &)
  8. Run incremental index.

If you are upgrading from v3.x, you will need to perform a migration to v4.x since our application was redesigned and the database schema are completely different. This is a significant undertaking. Please contact us (support at dataverse.org) before beginning. Also refer to our migration google group for additional support and information: https://groups.google.com/d/msgid/dataverse-migration-wg

Please note: v4.x does not currently support creating new handles though it will support existing ones. We intend to add this feature but have not yet scheduled this work.

IMPORTANT: If you are running TwoRavens with your dataverse:
Make sure the two applications are using the same version of the "pre-processed statistics" R code. Compare the 2 files:
On the TwoRavens side:
.../dataexplore/rook/preprocess/preprocess.R
On the Dataverse side:
.../applications/dataverse-4.6/WEB-INF/classes/edu/harvard/iq/dataverse/rserve/scripts/preprocess.R

If they are different, replace the Dataverse copy with the TwoRavens copy (i.e., the TwoRavens version wins!).
And, also, remove all the already-generated pre-processed fragments in your Dataverse file directory, for example:

cd [files directory]
rm -f find . -name '*.prep'

If the two copies are the same, you don't need to do any of this.
Please note that this is a temporary measure, we are working on a fix that will make the two applications resolve code version conflicts like this automatically.

Downloads

v4.5.1

@kcondon kcondon released this Sep 30, 2016 · 234 commits to master since this release

Overview:

The main feature of this release is the restoration of the IP Groups functionality. Additional features include email verification, dynamic copyright in the page footer, and support for Piwik.

  • #700 IP groups in Dataverse are a way to specify a certain IP address range so that permissions can be applied based on the user's location or the device used to access the application. This is useful for installations that want to allow traffic from certain networks.
  • #2170 Email verification encourages new users to confirm their account email address.
  • #2905 Dynamic copyright automatically keeps the copyright date current and allows installations to add information about their organization.
  • #3018 Piwik support provides an alternative to Google Analytics, that supports local collection and hosting of usage metrics.

Special thanks to Jeremy Richard from Sciences Po who contributed the Piwik support feature.

For the complete list of issues, see the 4.5.1 milestone in Github.

For help with upgrading, installing, or general questions please email support@dataverse.org.

Installation:

If this is a new installation, please see our Installation Guide.

Upgrade:

If you are upgrading from v4.x, you must upgrade to each intermediate version before installing this version. When upgrading from the previous version, you will need to do the following:

  1. Undeploy the previous version.
    • /glassfish4/bin/asadmin list-applications
    • /glassfish4/bin/asadmin undeploy dataverse
  2. Stop glassfish and remove the generated directory
    • service glassfish stop
    • remove the generated directory: rm -rf /usr/local/glassfish4/glassfish/domains/domain1/generated
  3. If not yet present, add a jvm option to /usr/local/glasssfish4/glassfish/domains/domain1/config/domain.xml to support timers, start glassfish
  4. Deploy this version.
    • /glassfish4/bin/asadmin deploy dataverse-4.5.1.war
  5. Run the database update script.
    "psql -U -d -f upgrade_v4.5_to_4.5.1.sql"

If you are upgrading from v3.x, you will need to perform a migration to v4.x since our application was redesigned and the database schema are completely different. This is a significant undertaking. Please contact us (support at dataverse.org) before beginning. Also refer to our migration google group for additional support and information: https://groups.google.com/d/msgid/dataverse-migration-wg

Please note: v4.x does not currently support creating new handles though it will support existing ones. We intend to add this feature but have not yet scheduled this work.

IMPORTANT: If you are running TwoRavens with your dataverse:
Make sure the two applications are using the same version of the "pre-processed statistics" R code. Compare the 2 files:
On the TwoRavens side:
.../dataexplore/rook/preprocess/preprocess.R
On the Dataverse side:
.../applications/dataverse-4.5.1/WEB-INF/classes/edu/harvard/iq/dataverse/rserve/scripts/preprocess.R

If they are different, replace the Dataverse copy with the TwoRavens copy (i.e., the TwoRavens version wins!).
And, also, remove all the already-generated pre-processed fragments in your Dataverse file directory, for example:

cd [files directory]
rm -f find . -name '*.prep'

If the two copies are the same, you don't need to do any of this.
Please note that this is a temporary measure, we are working on a fix that will make the two applications resolve code version conflicts like this automatically.

Downloads

v4.5

@kcondon kcondon released this Aug 24, 2016

Overview:

The main features of this release are Harvesting, Metadata Export, Private URL, and Federated Login Mode.

  • Harvesting allows the dataset metadata from another site to be imported so that these files appear to be local, though data files remain on the remote site. This allows Dataverse installations and other repositories to share metadata with each other to create a data sharing community and to provide more access to the datasets stored in each repository. Harvesting is implemented using the standard OAI-PMH protocol - any Dataverse installation can be configured as an OAI client and server.
  • Metadata Export makes it easier to share and discover from Dataverse. Another key function of the export is to store the metadata in the file system in a format for preservation. In this release, the metadata is exported in Dublin Core, Data Documentation Initiative (DDI), and native JSON. Metadata export will support additional standards in the future.
  • Private URL provides a simple way to generate and send a special URL to an unpublished dataset for a group of anonymous peer reviewers to review the data.
  • Federated Login Mode allows users from 200+ different organizations to sign in using his or her organizational credentials.

Big thanks to all the community members who participated in usability testing, provided feedback in Github issues, and worked with us in other ways to make sure that this release was successful.

For the complete list of issues, see the 4.5 milestone in Github.

For help with upgrading, installing, or general questions please email support@dataverse.org.

Installation:

If this is a new installation, please see our Installation Guide.

Upgrade:

If you are upgrading from v4.x, you must upgrade to each intermediate version before installing this version. When upgrading from the previous version, you will need to do the following:

  1. Undeploy the previous version.
    • /glassfish4/bin/asadmin list-applications
    • /glassfish4/bin/asadmin undeploy dataverse
  2. Stop glassfish and remove the generated directory
    • service glassfish stop
    • remove the generated directory: rm -rf /usr/local/glassfish4/glassfish/domains/domain1/generated
  3. Add a new jvm option to /usr/local/glasssfish4/glassfish/domains/domain1/config/domain.xml to support timers, start glassfish
  4. Deploy this version.
    • /glassfish4/bin/asadmin deploy dataverse-4.5.war
  5. Run the database update script.
    "psql -U -d -f upgrade_v4.4_to_4.5.sql"
  6. Replace the Solr schema.xml file
    • stop solr (kill -9 pid)
    • replace schema.xml
    • start solr (java -jar start.jar &)
  7. Run the incremental index to prepare for harvesting facet.

If you are upgrading from v3.x, you will need to perform a migration to v4.x since our application was redesigned and the database schema are completely different. This is a significant undertaking. Please contact us (support at dataverse.org) before beginning. Also refer to our migration google group for additional support and information: https://groups.google.com/d/msgid/dataverse-migration-wg

Please note: v4.x does not currently support creating new handles though it will support existing ones. We intend to add this feature but have not yet scheduled this work.

IMPORTANT: If you are running TwoRavens with your dataverse:
Make sure the two applications are using the same version of the "pre-processed statistics" R code. Compare the 2 files:
On the TwoRavens side:
.../dataexplore/rook/preprocess/preprocess.R
On the Dataverse side:
.../applications/dataverse-4.5/WEB-INF/classes/edu/harvard/iq/dataverse/rserve/scripts/preprocess.R

If they are different, replace the Dataverse copy with the TwoRavens copy (i.e., the TwoRavens version wins!).
And, also, remove all the already-generated pre-processed fragments in your Dataverse file directory, for example:

cd [files directory]
rm -f find . -name '*.prep'

If the two copies are the same, you don't need to do any of this.
Please note that this is a temporary measure, we are working on a fix that will make the two applications resolve code version conflicts like this automatically.

Downloads

v4.4

@kcondon kcondon released this Jun 29, 2016

Overview:

This release contains feature enhancements for embedding content using widgets, downloading guestbook data for a dataverse in a consolidated fashion, and introduces a new feature, support for remote authentication using Shibboleth.

  • Two new widgets: dataset and citation in addition to the existing widgets, dataverse listing and search box.
  • Improved widget navigation: child objects open within widgets rather than redirecting to Dataverse.
  • A new dataset metadata field, alternativeURL, can be used to link a dataset to an author's website.
  • Shibboleth for remote authentication is now officially supported.
  • Downloading guestbook data is now easier: all guestbook data for all datasets within a dataverse can be downloaded by one click.

Please see the <a href=https://github.com/IQSS/dataverse/issues?q=is%3Aissue+milestone%3A4.4+is%3Aclosed>4.4 milestone issues list for more information.

Installation:

If this is a new installation, please see our Installation Guide.

Upgrade:

If you are upgrading from v4.x, you must upgrade to each intermediate version before installing this version. When upgrading from the previous version, you will need to do the following:

  1. Undeploy the previous version.
    • /glassfish4/bin/asadmin list-applications
    • /glassfish4/bin/asadmin undeploy dataverse
  2. Deploy this version.
    • /glassfish4/bin/asadmin deploy dataverse-4.4.war
  3. Restart glassfish.
    • service glassfish stop
    • service glassfish start
  4. Update the citation.tsv file
  5. Replace the Solr schema.xml file
    • stop solr
    • replace schema.xml
    • start solr
  6. Run the incremental index to correct the targeting for citation links.

If you are upgrading from v3.x, you will need to perform a migration to v4.x since our application was redesigned and the database schema are completely different. This is a significant undertaking. Please contact us (support at dataverse.org) before beginning. Also refer to our migration google group for additional support and information: https://groups.google.com/d/msgid/dataverse-migration-wg

Please note: v4.x does not currently support creating new handles though it will support existing ones. We intend to add this feature but have not yet scheduled this work.

If you install Two Ravens or have an older version of Two Ravens configured and upgrade Two Ravens and Dataverse, you will need to update the preprocess.R file from the Two Ravens project:

  1. Update preprocess.R
    replace the current preprocess.R file in:
    /usr/local/glassfish4/glassfish/domains/domain1/applications/dataverse-4.4/WEB-INF/classes/edu/harvard/iq/dataverse/rserve/scripts/preprocess.R

with the one from:

https://raw.githubusercontent.com/IQSS/TwoRavens/master/rook/preprocess/preprocess.R

  1. Remove any existing preprocess files

cd /usr/local/glassfish4/glassfish/domains/domain1/files
rm -f find . -name '*.prep'

Note the special apostrophe in the rm statement, it is not a single quote. Copy and pasting is the easiest way.

  1. Restart glassfish

Downloads

v4.3.1

@kcondon kcondon released this May 11, 2016

Overview:

This release is a small patch release to address a potential security issue discovered by Andy Boughton. It also includes some unrelated minor changes.

  • Escape script elements in HTML-enabled fields.
  • Added support for downloading the citation in BibTeX format.
  • Add Laboratory and Research Group as categories of dataverses.
  • Correct a number of file tag issues, improving their function.

Please see the <a href=https://github.com/IQSS/dataverse/issues?q=is%3Aissue+milestone%3A4.4+is%3Aclosed>4.4 milestone issues list for more information.

Installation:

If this is a new installation, please see our Installation Guide.

Upgrade:

If you are upgrading from v4.x, you must upgrade to each intermediate version before installing this version. When upgrading from the previous version, you will need to do the following:

  1. Undeploy the previous version.
    • /glassfish4/bin/asadmin list-applications
    • /glassfish4/bin/asadmin undeploy dataverse.war
  2. Deploy this version.
    • /glassfish4/bin/asadmin deploy /dataverse-4.3.1.war
  3. Restart glassfish.
    • service glassfish stop
    • service glassfish start
  4. Replace the Solr schema.xml file
    • stop solr
    • replace schema.xml
    • start solr
  5. Run the incremental index to correct the targeting for citation links.

If you are upgrading from v3.x, you will need to perform a migration to v4.x since our application was redesigned and the database schema are completely different. This is a significant undertaking. Please contact us (support at dataverse.org) before beginning. Also refer to our migration google group for additional support and information: https://groups.google.com/d/msgid/dataverse-migration-wg

Please note: v4.x does not currently support creating new handles though it will support existing ones. We intend to add this feature but have not yet scheduled this work.

Downloads

v4.3

@kcondon kcondon released this Mar 17, 2016 · 1535 commits to develop since this release

Overview:

This release added one major feature, DataCite API support and several bug fixes and feature enhancements:

  • Added support for creating DOIs using the DataCite API.
  • Improved the installer to include more configuration steps.
  • Addressed a potential security issue involving the default configuration.
  • #2115 Allow log in using email address in addition to username.
  • Check for email address uniqueness regardless of case.
  • Improved performance for the zip download of a large number of files.
  • Allow customizable text for publish dataset popup.
  • Improved performance of user name look up on permissions pages.
  • Allow hiding explore button when two ravens is not configured.
  • Allow disabling google analytics if not needed.
  • Support using DOI as identifier in dataset APIs.
  • Fixed a URL targeting issue for citation links inside a dataverse widget and iframe on remote sites.

Please see the <a href=https://github.com/IQSS/dataverse/issues?q=milestone%3A4.3+is%3Aclosed>4.3 milestone issues list for more information.

Installation:

If this is a new installation, please see our Installation Guide.

Upgrade:

If you are upgrading from v4.x, you must upgrade to each intermediate version before installing this version. When upgrading from the previous version, v4.2.4 you will need to do the following:

  1. First, check for and correct duplicate email addresses, including duplicates due only to different case. This is a new requirement for this version.
    • See remove_duplicate_emails.txt for more information.
  2. Undeploy the previous version.
    • /glassfish4/bin/asadmin list-applications
    • /glassfish4/bin/asadmin undeploy dataverse.war
  3. Deploy this version.
    • /glassfish4/bin/asadmin deploy /dataverse-4.3.war
  4. Run the database update script.
    • psql -U <db user> -d <db name> -f upgrade_v4.2.4_to_4.3.sql
  5. Restart glassfish.
    • service glassfish stop
    • service glassfish start
  6. If you were running Two Ravens, enable it for this version.
  7. Run the incremental index to correct the targeting for citation links.

If you are upgrading from v3.x, you will need to perform a migration to v4.x since our application was redesigned and the database schema are completely different. This is a significant undertaking. Please contact us (support at dataverse.org) before beginning. Also refer to our migration google group for additional support and information: https://groups.google.com/d/msgid/dataverse-migration-wg

Please note: v4.x does not currently support creating new handles though it will support existing ones. We intend to add this feature but have not yet scheduled this work.

Downloads

v4.2.4

@kcondon kcondon released this Feb 9, 2016

Overview:

This is a patch release to address issues with harvested dataset links and correct the behavior of zip downloads when selecting restricted files to which you do not have permission. There are a few other corrections such as fixing the :ZipDownloadLimit setting, allowing a Dataverse installation administrator to adjust the maximum amount of data that can be downloaded at one time. We have also reorganized our Installation Guide and added significantly to the content.

For the complete list of issues, see the <a href=https://github.com/IQSS/dataverse/issues?q=milestone%3A4.2.4+is%3Aclosed>individual issues.

Installation:
Please see our Installation Guide

Upgrades:
If upgrading from v3.6.2 or earlier, please contact us at support@dataverse.org since this requires considerable planning and effort.

If upgrading from v4.x, you will need to follow the upgrade instructions for each intermediate version.
This version requires you to first undeploy the current application, then deploy this war file.

  1. undeploy the current application
    /glassfish4/bin/asadmin list-applications
    /glassfish4/bin/asamin undeploy dataverse-4.2.3
  2. deploy the new war file
    /glassfish4/bin/asadmin deploy dataverse-4.2.4.war

Downloads

v4.2.3

@kcondon kcondon released this Jan 19, 2016

This release is small in scope; it's mostly about a technology upgrade that includes moving to Java 8 and upgrading to Prime Faces 5.3. We have fixed a number of issues, including making the file ingest page refresh more reliable.

For more information see the <a href=https://github.com/IQSS/dataverse/milestones?state=closed>project milestone notes and <a href=https://github.com/IQSS/dataverse/issues?q=is%3Aissue+milestone%3A4.2.3+is%3Aclosed>individual issues.

IMPORTANT, for both new installations and upgrades:

For the file uploads to work properly, please follow the instructions in the README.txt file in https://github.com/IQSS/dataverse/tree/4.2.3/conf/jhove (In future version of the Dataverse the installer script will be taking care of this).

Installation:
Please see our Installation Guide

Upgrades:
If upgrading from v3.6.2 or earlier, please contact us at support@dataverse.org since this requires considerable planning and effort.

If upgrading from v4.x, you will need to follow the upgrade instructions for each intermediate version.
This version requires you to install Java 8, undeploy the current application, then deploy this war file.

  1. undeploy the current application
    /glassfish4/bin/asadmin list-applications
    /glassfish4/bin/asamin undeploy dataverse-4.2.2
  2. deploy the new war file
    /glassfish4/bin/asadmin deploy dataverse-4.2.3.war

Please note that this version of Dataverse requires Java 8 to be installed, otherwise the application will not deploy. See the java section of the Installation Guide.

If you are using shape files and GeoConnect functionality, please run the following commands from the web server console since it has been made disabled by default:

Set GeoconnectCreateEditMaps to true to allow the user to create GeoConnect Maps. This boolean effects whether the user sees the map button on the dataset page and if the ingest will create a shape file.

curl -X PUT -d true http://localhost:8080/api/admin/settings/:GeoconnectCreateEditMaps

Set GeoconnectViewMaps to true to allow a user to view existing maps. This boolean effects whether a user will see the "Explore" button.

curl -X PUT -d true http://localhost:8080/api/admin/settings/:GeoconnectViewMaps

Downloads

v4.2.2

@kcondon kcondon released this Dec 11, 2015

Overview:

v4.2.2 is focused on improving performance of the Dataverse page and Files facet. Additionally, several important support issues were addressed.

For more information see the <a href=https://github.com/IQSS/dataverse/milestones?state=closed>project milestone notes and <a href=https://github.com/IQSS/dataverse/issues?q=milestone%3A%224.2.2+Performance+patch%22+is%3Aclosedd>individual issues

Installation:

Please see our Installation Guide

Upgrades:

If upgrading from v3.6.2 or earlier, please contact us at support@dataverse.org since this requires considerable planning and effort.

If upgrading from v4.x, you will need to follow the upgrade instructions for each intermediate version.

Please note that tabular data tags are indexed in v4.2.2 and that requires a new Solr schema and a complete reindex (see below, under the "post-deployment tasks").

For upgrading from the previous version, v4.2.1 to v4.2.2 do the folllowing:

undeploy the current application:

/glassfish4/bin/asadmin undeploy dataverse-4.2.1

update the solr schema.xml file:
stop solr
remove current schema.xml
replace with the 4.2.2 schema.xml file provided
start solr

deploy the new war file:

/glassfish4/bin/asadmin deploy dataverse-4.2.2.war

run the upgrade psql script provided: upgrade_v4.2.1_to_v4.2.2.sql
restart glassfish

Post-deployment tasks:

1. Reindex Solr

This is a non destructive reindex and can be done while the application is serving users. First clear index time from the dvobject table, by executing the following SQL query:

 update dvobject set indextime=null;

Then from the web server console run:

 curl -X GET "http://localhost:8080/api/admin/index/continue?numPartitions=1&partitionIdToProcess=0"

Alternatively, if you don’t mind doing a destructive reindex, clear the current index:

 curl -X GET “http://localhost:8080/api/admin/index/clear"

Then do an index all:

 curl -X GET "http://localhost:8080/api/admin/index"

Where 8080 is the port where Glassfish is running.

2. The following shell script needs to be run in your files directory:

(the script simply renames any legacy "saved original files", with the filenames like _filename, as filename.orig)

  #!/bin/sh

  find . -type f | while read file
  do
    origfile=`echo $file | sed 's:[^/]*$:_&:'`
    if [ -f "$origfile" ]
    then
       /bin/mv -f $origfile $file".orig" 
    fi
  done

("files directory" is where the Dataverse application is storing physical data files. It is specified as

<jvm-options>-Ddataverse.files.directory=...</jvm-options>

in your domain.xml configuration file)

Downloads

v4.2.1

@kcondon kcondon released this Nov 2, 2015 · 1463 commits to master since this release

Dataverse v4.2.1 is focused on improving performance of the dataset page and system stability by greatly reducing the number database calls made and limiting to 25 the number of files displayed on initialization. In addition, the SWORD API performance was improved to allow retrieval of file listings with large numbers of files.

For more information see the <a href=https://github.com/IQSS/dataverse/milestones?state=closed>project milestone notes and <a href=https://github.com/IQSS/dataverse/issues?q=milestone%3A%224.2.1+Performance+patch%22+is%3Aclosedd>individual issues .

Installation:
Please see our Installation Guide

Upgrades:
If upgrading from v3.6.2 or earlier, please contact us at support@dataverse.org since this requires considerable planning and effort.

If upgrading from v4.x, you will need to follow the upgrade instructions for each intermediate version.

For upgrading from the previous version, v4.2 to v4.2.1 do the folllowing:

  1. undeploy the current application
    /glassfish4/bin/asadmin list-applications
    /glassfish4/bin/asdamin undeploy dataverse-4.2
  2. deploy the new war file
    /glassfish4/bin/asadmin deploy dataverse-4.2.1.war
    restart glassfish

Downloads