v3.13: Release Notes

OpenWIS 3.13 - Release Notes

This document highlights the changes made as part of the release of OpenWIS 3.13. This document also contains the upgrade procedure from OpenWIS 3.12 to OpenWIS 3.13.

Changes

Headline Changes

Open Source License: Version 3.13 is the first version of OpenWIS released under an Open Source license from the OpenWIS Association. Previous versions of OpenWIS were built and released under a commercial agreement with the five founding members of the OpenWIS Association (previously known as the OpenWIS consortium) for use only by these members. This commercial agreement has since expired, and a new association has adopted the rights to the OpenWIS code base with the intent of releasing it to the Open Source community.
OAI-PMH Dates: Version 3.13 include a fix to the date stamp used in OAI-PMH as reported by the metadata itself (the element identified using the XPath expression /gmd:MD_Metadata/gmd:dateStamp), which could be any date specified by the metadata author. This was causing problems with the daily metadata synchronisation process whereby GISCs synchronising from the OpenWIS catalogue were encountering dates which were falling outside the from and until range given in the OAI-PMH queries.

This version of OpenWIS now uses an internally tracked date within the response to these OAI-PMH queries. This date is now used in the response of the OAI-PMH queries and is updated whenever the metadata changes in any way, including changes that do not affect the contents of the metadata record (e.g. moving the metadata amongst different categories). This change was made in order to comply with the recommendations raised as part of the TT-GISC 2014 meeting in Toulouse.
Case Insensitive URNS: Version 3.13 includes a change in how it treats URNs within the catalogue (OWT-482). Previously OpenWIS stored and compared URNs in a case sensitive manor which could allow metadata records with URNs that differ only in case to co-exist within the catalogue. This led to potential problems with the catalogue synchronisation amongst GISCs which treated URNs in a case insensitive manor, as indicated in the WMO Core Metadata Profile ver. 1.3 (WCMP 1.3).

URNs are now stored and compared in a case insensitive manor in order to comply with WCMP 1.3 and the recommendations raised as part of the TT-GISC 2014 meeting held in Toulouse. The case of URNs of new metadata records will be preserved but it is no longer possible for two metadata records with URNs that only differ in case to co-exist within the catalogue.

Other Features and Improvements

Upgraded OpenLayers JavaScript mapping library from version 2.8 to version 2.13.1 (OWT-458).
Added a dedicated category search field to the Browse My Metadata page and the Monitor Metadata administration page (OWT-485).
Added detailed reporting to metadata harvesting tasks which lists all the URNs added, updated and removed (OWT-484).
Added references to DateTypeCode and dataCentre to KeywordTypeCode in metadata editor to comply with WCMP 1.3.
Improved the harvesting task edit dialogue to include fields for setting the start date and time of recurrent harvesting tasks (OWT-468).
Added permalinks to metadata records (OWT-467).
Added the ability go to the service associated with series metadata, if one exists, from within the search results (OWT-465).
Added service for performing user authorization checks from external systems as part of the Inspire facade (OWT-464).
Improved metadata search ranking to favour records that contain search terms within the title, abstract or keywords (OWT-364).
Modified the bulk metadata removal logic to skip records with subscriptions made against them instead of aborting the entire operation (OWT-481).
Improved the performance of removing multiple records of metadata in bulk (OWT-481).
Added "User Alarms" which will display error messages from the dissemination harnesses for requests or subscriptions instantiated by the user.

Bug Fixes

Fixed issue with dissemination from local data sources which was producing empty archives when WMO_FTP compression option is used (OWT-461).
Fixed metadata validation report to include the metadata URN in validation error messages (OWT-495).
Fixed issue with search that was returning no results when a query with common English (stop) words is entered (OWT-457).
Fixed the logic used to calculate the total volume of data extracted from the local data source to include all extract files instead of the first one encountered (OWT-488).
Fixed potential SQL injection bug which was not escaping apostrophes of the alias field within the data policy form (OWT-471).
Fixed the request and subscription button which was not displaying the sub-selection box for metadata associated with a local data source.
Fixed issue with cache purge routine task that was failing due to transaction roll-backs, causing the cache to grow without bounds (OWT-487).
Removed the social networks link bar (OWT-466).
Fixed presentation issue in the metadata viewer that was not displaying the indeterminate position of the temporal extent of ISO 19139 metadata records (OWT-474).
Fixed issue which could result in duplicate metadata when white-space character appear within the URN of records uploaded or imported into the system (OWT-434).
Fixed issue with search that was combining parameters entered in the Advanced Search section when using the Normal Search (OWT-476).

Changes to the Operating Environment

The following list identifies the significant changes to the operating environment of OpenWIS. The instructions to change the operating environment to conform to these changes are listed in the upgrade procedures.

OpenWIS now requires the citext PostgreSQL extension which provides a case insensitive data type and is traditionally included as part of the PostgreSQL Contrib package. The procedure below give instructions on how to install postgresql-contrib package from an RPM, but it may need to be modified depending on how PostgreSQL was installed on your system.
OpenWIS now requires version 8.4-702 of the PostgreSQL JDBC drivers, which will replace the previous version of the JDBC drivers used by the data service. The new version of the drivers is included as part of the built artefacts of the user and admin portal.
OpenWIS no longer performs housekeeping of the cache directory on the file-system. This is now to be performed by scheduling a cron job.
Version numbers have been removed from the SOAP URLs.

Upgrade Procedure from OpenWIS 3.12

This section contains the instructions for upgrading from version 3.12 to version 3.13. These instructions are adapted from the instructions within the v3.13 Deployment Instructions WIKI Page.

The instructions are broken up into the following sections:

The list of artefacts that need to be deployed.
The instructions that are to be performed before deploying the new artefacts.
The instructions that are to be performed after deploying the new artefacts but before starting up the system.
The instructions that are to be performed after OpenWIS has been started.

Scope of Upgrade

This is a significant upgrade which may impact the delivery of subscription data and the availability of the public user portal. The upgrade itself may take up to an hour to complete and will require a rebuild of the search index which may take several more hours, depending on the size of the catalogue. Please notify all stakeholders that the search capabilities may not be available for a little while after the conclusion of the upgrade.

The following artefacts have been changed and will require re-deployment when upgrading to 3.13:

openwis-user-portal.war
openwis-admin-portal.war
openwis-portal-solr.war
openwis-dataservices.ear
openwis-management-service.ear
openwis-securityservice.ear

Before Deploying the New Artefacts

Shut-down OpenWIS completely using the standard procedure.
Backup the database and Solr index.
Install the PostgreSQL contrib package. The actual package name may depend on your system but it will have a name like postgresql-contrib. You will need to be the root user, or have sudo access to do this:

To install the contrib package on Red Hat:
```
 $ sudo yum install postgresql-contrib
```
Install CIText extensions from PostgreSQL contrib package. You will need to be the postgres user to do this. The actual path of the citext.sql script may vary depending on the OS distribution.
```
 $ psql OpenWIS -f '/usr/share/pgsql/contrib/citext.sql'
```

Download PostgreSQL JDBC drivers v8.4-702:

 $ wget http://jdbc.postgresql.org/download/postgresql-8.4-702.jdbc4.jar

Log into the data services host and replace "EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/lib/postgresql-8.4-701.jdbc4.jar" with the newly downloaded drivers:

 $ rm EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/lib/postgresql-8.4-701.jdbc4.jar
 $ cp postgresql-8.4-702.jdbc4.jar EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/lib/.

Open "EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/deploy/openwis-ds.xml" in an editor and add ?stringtype=unspecified to the end of the "connection-url" value. The XML should look like the following:
```
 <connection-url>jdbc:postgresql://dbserver:5432/OpenWIS?stringtype=unspecified</connection-url>
```

Log into the OpenWIS database as the "openwis" user and execute the following script:

 -- Data migration for URN case insensitive changes
 --
 alter table metadata alter column uuid type citext;
 alter table deletedmetadata alter column uuid type citext;
 alter table openwis_product_metadata alter column urn type citext;


 -- New table for user alarms
 --
 create table OPENWIS_USER_ALARM (
   ALARM_ID int8 not null,
   DATE timestamp,
   USER_ID varchar(255),
   REQ_TYPE varchar(255),
   REQ_ID int8,
   PROCESSED_REQ_ID int8,
   MESSAGE varchar(255),

   primary key (ALARM_ID)
 );

 create sequence USER_ALARM_SEQ;

 create index USER_ID_AND_REQ_TYPE_IDX on OPENWIS_USER_ALARM 
 (user_id, req_type);


 -- Data migration for adding a start date to harvesting tasks.
 --
 ALTER TABLE harvestingtask ADD startingdate character varying(24);

On the data service host, add the following cron jobs:

 # Remove all cache files older than 1 day
 */15 * * * * /usr/bin/find /cache -mmin +1500 -exec /bin/rm -rf {} \;
 # Remove empty cache folders
 */20 * * * * /usr/bin/find /cache -mindepth 1 -depth -type d -empty -exec rmdir {} \;

Deploy the new artefacts. Do not start the application yet.

After Deploying the New Artefacts

These instructions are to be performed once the new artefacts have been deployed but BEFORE the application is started.

Go to each of the following configuration files and remove -1.0-SNAPSHOT or -develop-SNAPSHOT from each URL that ends with ?wsdl. Keep in mind URLs to third-party services (e.g. dissemination harnesses, local data sources, etc.) as they may need to stay as they are.

The list of configuration files to change are:

Service	File
Sercurity Service	EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/deploy/openwis-securityservice-jndi-service.xml
Data Services	EnterprisePlatform-5.1.0/jboss-eap-5.1/jboss-as/server/production/deploy/openwis-dataservice-jndi-service.xml
User Portal	apache-tomcat-6.0.29/webapps/openwis-user-portal/WEB-INF/classes/openwis-metadataportal.properties
Admin Portal	apache-tomcat-6.0.29/webapps/openwis-admin-portal/WEB-INF/classes/openwis-metadataportal.properties

Example: Subset of URLS from within openwis-metadataportal.properties before the change:

After removing -1.0-SNAPSHOT:

Perform the following changes on both the user and admin portal:

Open "WEB-INF/config.xml" in a text editor.

Locate the <resource> element that is enabled and has a <name> element with the value "main-db". Add the snip-it in the example below:

 <resource enabled="true">
   <name>main-db</name>
   ...
   <config>
     <user>openwis</user>
     <password>whatever</password>
     ...
     <!-- Add the following snip-it -->
     <properties>
       <property key="stringtype">unspecified</property>
     </properties>
     <!-- End snip-it -->
   </config>
 </resource>

Open "WEB-INF/classes/openwis-metadataportal.properties" within a text editor and add the following snippit, replacing {dataService} with the hostname of the data services:
```
 openwis.management.useralarmmanager.wsdl=http://{dataService}/openwis-dataservice-openwis-dataservice-cache-ejb/UserAlarmManager?wsdl
```

Perform the following changes on the admin portal server only:
1. Create a data directory which will be used to store the harvesting reports. This will be referred to as REPORTS_DIR.
2. Add the following properties to "openwis-metadataportal.properties":
```
 openwis.metadataportal.report.file.path=<REPORTS_DIR>
 openwis.metadataportal.report.file.prefix=harvest
 openwis.metadataportal.report.file.ext=txt
```
3. Add a cron to purge any harvest reports which are older than 7 days:
```
 # Remove harvesting logs older than 7 days at 00:30
 30 00 * * * /usr/bin/find <REPORTS_DIR>/harvest-*.txt -mtime +7 -exec rm {} \;
```
For centres that provide Inspire services, the following configuration changes need to be performed for the user and admin portal:
1. Create the WEB-INF/config-inspire.xml file containing the Jeeves service declaration:
```
 <?xml version="1.0" encoding="ISO-8859-1"?>
 <geonet>
   <services package="org.openwis.metadataportal">
     <service name="xml.CheckAuthorization" type="xml">
       <class name=".services.inspire.CheckAuthorization"/>
     </service>
   </services>
 </geonet>
```
2. Add the following line to WEB-INF/userPortal-profiles.xml (for the user portal) and WEB-INF/adminPortal-profiles.xml to authorize access to this service for the Guest profile:
```
 <profile name="Guest">
   ...
   <allow service="xml.CheckAuthorization"/>  
 </profile>
```
3. Add the following line to WEB-INF/config.xml file:
```
 <include>config-inspire.xml</include>
```
Start OpenWIS using the standard procedure.

After Starting OpenWIS v3.13

These instructions are to be performed after starting OpenWIS 3.13 for the first time, but before announcing the system as available.

Log into the admin portal and rebuild the index.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly