The First Time Deployment Guide

seanlinxs edited this page Aug 30, 2013 · 24 revisions

1. Components Diagram

Components Diagram

Note that this document use <VMx> to refer to the server's hostname, for example, if your VM1 is www.example.com, then all <VM1> should be replaced by www.example.com

2. Operating System Setup (On all VMs)

  • Install CentOS 6.4 x86_64 with “Basic Server” packages
  • Disable SELinux, open /etc/selinux/config, set SELINUX=disabled then reboot the server
  • Configure iptables according to “Firewall Ports need to be opened” in the diagram. For example if you want to allow access to 8080 port, open /etc/sysconfig/iptables, add this line:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT

just before line (The order/position is important!):

-A INPUT -j REJECT --reject-with icmp-host-prohibited

Then run sudo service iptables restart to apply the new rules.

3. Create an account

Create shed for all VMs to install and deploy components, all the following steps use this account by default if not specified:

$ sudo useradd shed
$ sudo passwd shed
Changing password for user shed.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Then add this line to /etc/sudoers:

shed    ALL=(ALL)       ALL

4. Install Oracle Java 6 (On all VMs)

This project use Oracle Java 6, here is the steps to make Oracle Java 6 the default java environment:

Note: Select rpm.bin package, e.g. jdk-6u43-linux-x64-rpm.bin

  • Change to root user
$ sudo -i
  • Run Sun/Oracle Java binary
# sh /path/to/jdk-6u43-linux-*-rpm.bin
  • Install Sun/Oracle JDK java/javac/jar with alternatives –-install command
# alternatives --install /usr/bin/java java /usr/java/jdk1.6.0_43/jre/bin/java 20000
# alternatives --install /usr/bin/javac javac /usr/java/jdk1.6.0_43/bin/javac 20000
# alternatives --install /usr/bin/jar jar /usr/java/jdk1.6.0_43/bin/jar 20000
  • Check current java, javac versions
# java -version
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b04)
Java HotSpot(TM) Server VM (build 20.6-b01, mixed mode)
# javac -version
javac 1.6.0_43
  • Swap between OpenJDK and Sun/Oracle JDK versions
# alternatives --config java   # or javac, jar
There are 4 programs which provide 'java'.
  Selection    Command
-----------------------------------------------
   1           /usr/lib/jvm/jre-1.6.0-openjdk/bin/java
   2           /usr/lib/jvm/jre-1.5.0-gcj/bin/java
*  3           /usr/java/jdk1.6.0_18/jre/bin/java
 + 4           /usr/java/jdk1.6.0_43/jre/bin/java

Enter to keep the current selection[+], or type selection number:
  • Post-Installation Setup

Add JAVA_HOME environment variable to /etc/profile file or $HOME/.bash_profile file

## export JAVA_HOME ##
export JAVA_HOME="/usr/java/jdk1.6.0_43"

5. Install Apache Tomcat 7.0.37 (VM1, VM2, VM3)

$ unzip apache-tomcat-7.0.37.zip
$ echo '$TOMCAT_HOME=”/home/shed/apache-tomcat-7.0.37”' >> ~/.bashrc
$ source ~/.bashrc
$ chmod +x $TOMCAT_HOME/bin/*.sh
  • Add JAVA_OPTS in $TOMCAT_HOME/bin/catalina.sh:
JAVA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8
-server -Xms1536m -Xmx1536m -XX:SoftRefLRUPolicyMSPerMB=36000
-XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=512m
-XX:MaxPermSize=512m -XX:+DisableExplicitGC -XX:+UseParallelGC"

Notes: For safety, before deploy any new applications, remove all applications from tomcat installer:

$ ls -l $TOMCAT_HOME/webapps
total 20
drwxr-xr-x. 13 shed shed 4096 Feb 12 22:45 docs
drwxr-xr-x.  6 shed shed 4096 Feb 12 22:45 examples
drwxr-xr-x.  5 shed shed 4096 Feb 12 22:45 host-manager
drwxr-xr-x.  5 shed shed 4096 Feb 12 22:45 manager
drwxr-xr-x.  3 shed shed 4096 Feb 12 22:45 ROOT
$ rm -fr $TOMCAT_HOME/webapps/*

6. Install PostgreSQL/PostGIS (VM3)

  • Install PostgreSQL Database Server/Client/Development headers
$ sudo yum groupinstall “PostgreSQL Database server”
$ sudo yum groupinstall “PostgreSQL Database client”
$ sudo yum groupinstall “Development tools”
$ sudo yum install postgresql-devel
$ sudo yum install libxml2-devel
  • Initialize postgresql database:
$ sudo service postgresql initdb
  • Configure connections

Open /var/lib/pgsql/data/pg_hba.conf, change last 6 lines as (Put your network/mask):

# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 connections:
host    all         all         172.20.2.0/24         md5
# IPv6 local connections:
host    all         all         ::1/128               md5

Then open /var/lib/pgsql/data/postgresql.conf. Find configuration line that read as follows:

listen_addresses='localhost' 

Change it to:

listen_addresses='*'

Then start postgresql:

sudo chkconfig postgresql on
sudo service postgresql start
  • Install PostGIS

Follow http://postgis.net/docs/manual-2.0/postgis_installation.html. Build requirements in shed home directory, install dependencies first (2.2. Requirements, only proj, geos, json-c, gdal needed):

PROJ, download from http://download.osgeo.org/proj/proj-4.8.0.tar.gz, then:

$ tar xzvf proj-4.8.0.tar.gz
$ cd proj-4.8.0
$ ./configure
$ make
$ sudo make install

GEOS, download from http://download.osgeo.org/geos/geos-3.3.8.tar.bz2, then:

$ tar xjvf geos-3.3.8.tar.bz2
$ cd geos-3.3.8
$ ./configure
$ make
$ sudo make install

JSON-C, download from https://s3.amazonaws.com/json-c_releases/releases/json-c-0.10.tar.gz, then:

$ tar xzvf json-c-0.10.tar.gz
$ cd json-c-0.10
$ ./configure
$ make
$ sudo make install
$ sudo cp json_object_iterator.h /usr/local/include/json/

GDAL, download from http://download.osgeo.org/gdal/gdal-1.9.2.tar.gz, then:

$ tar xzvf gdal-1.9.2.tar.gz
$ cd gdal-1.9.2
$ ./configure
$ make
$ sudo make install

PostGIS, download from http://download.osgeo.org/postgis/source/postgis-2.0.3.tar.gz, then:

$ tar postgis-2.0.3.tar.gz
$ tar xzvf postgis-2.0.3.tar.gz
$ cd postgis-2.0.3
$ ./configure --with-raster --with-topology
$ make
$ sudo make install
  • Create links for 64 bit libraries
$ sudo ln -s /usr/local/lib/libgeos_c.so.1 /usr/lib64/
$ sudo ln -s /usr/local/lib/libproj.so.0 /usr/lib64/
$ sudo ln -s /usr/local/lib/libjson.so.0 /usr/lib64/
$ sudo ln -s /usr/local/lib/libgdal.so.1 /usr/lib64/
  • Create database user
$ createuser -s shed -U postgres
$ psql postgres postgres
postgres=# ALTER USER shed ENCRYPTED PASSWORD 'shed';
ALTER ROLE
  • Create database for wmsscanner
$ createdb -O shed wms_scanner -U shed
$ psql wms_scanner -U shed
psql (8.4.13)
Type "help" for help.

wms_scanner=# \l
                                    List of databases
    Name     |   Owner    | Encoding |  Collation  |    Ctype    |   Access privileges
-------------+------------+----------+-------------+-------------+-----------------------
 geonetwork  | geonetwork | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 geos        | shed       | LATIN1   | en_AU       | en_AU       |
 postgres    | postgres   | LATIN1   | en_AU       | en_AU       |
 sosdb       | sos        | LATIN1   | en_AU       | en_AU       |
 template0   | postgres   | LATIN1   | en_AU       | en_AU       | =c/postgres
                                                                 : postgres=CTc/postgres
 template1   | postgres   | LATIN1   | en_AU       | en_AU       | =c/postgres
                                                                 : postgres=CTc/postgres
 wms_scanner | shed       | LATIN1   | en_AU       | en_AU       |
(7 rows)
  • Create database for aodn-portal
$ createdb -O shed -U shed -l en_US.UTF-8 -E UTF-8 -T template0 aodn_portal
$ psql aodn_portal -U shed
psql (8.4.13)
Type "help" for help.

aodn_portal=# \l
                                    List of databases
    Name     |   Owner    | Encoding |  Collation  |    Ctype    |   Access privileges
-------------+------------+----------+-------------+-------------+-----------------------
 aodn_portal | shed       | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 geonetwork  | geonetwork | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 geos        | shed       | LATIN1   | en_AU       | en_AU       |
 postgres    | postgres   | LATIN1   | en_AU       | en_AU       |
 sosdb       | sos        | LATIN1   | en_AU       | en_AU       |
 template0   | postgres   | LATIN1   | en_AU       | en_AU       | =c/postgres
                                                                 : postgres=CTc/postgres
 template1   | postgres   | LATIN1   | en_AU       | en_AU       | =c/postgres
                                                                 : postgres=CTc/postgres
 wms_scanner | shed       | LATIN1   | en_AU       | en_AU       |
(8 rows)

7. Install THREDDS (VM2)

Download the latest thredds.war file, and put it into $TOMCAT_HOME/webapps. Start/restart Tomcat so that it has a chance to create initial files in $TOMCAT_HOME/content/thredds.

$ wget ftp://ftp.unidata.ucar.edu/pub/thredds/4.3/current/thredds.war
$ cp thredds.war $TOMCAT_HOME/webapps/
$ $TOMCAT_HOME/bin/shutdown.sh
$ $TOMCAT_HOME/bin/startup.sh

Modify $TOMCAT_HOME/content/thredds/catalog.xml for your site, as in this example:

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="Sydney Harbour Environment Data Catalog"
        xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
        xmlns:xlink="http://www.w3.org/1999/xlink">

  <service name="all" base="" serviceType="compound">
    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
    <service name="wms" serviceType="WMS" base="/thredds/wms/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
  </service>

  <datasetScan name="Sydney Harbour Environment Datasets" ID="shed" path="shed" location="content/shed/">

    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>Grid</dataType>
    </metadata>

  </datasetScan>
</catalog>

Note The location attribute location="content/shed/" in <datasetScan> element means $TOMCAT_HOME/content/thredds/public/shed in host file system.

Modify $TOMCAT_HOME/content/thredds/threddsConfig.xml for your site in the following manner:

  • Add the needed information to the serverInformation
  • Enable any other optional services like WMS or WCS.

If needed, limit access to the TDS and create a robots.txt file in $TOMCAT_HOME/webapps/ROOT/ to restrict crawler activity.

Other information please refer to http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/tutorial/Checklist.html

Deploy netCDF monitor script

From build machine

$ scp ~/aodn-portal/download/efdc_update.sh shed@<VM2>:~

Then on VM2

$ sudo -i
# crontab -e

Add this line to the end:

0 1 * * * /bin/bash /path/to/efdc_update.sh /path/to/netcdf http://<portal_root_url> >> /var/log/efdc_update.log 2>&1

This will run efdc_update.sh on 1:00am every day to check status of model output, if new netcdf generated, netcdf metadata will update automatically.

8. Install ncWMS (VM2)

The official site is http://www.resc.rdg.ac.uk/trac/ncWMS/. You can download to your desktop then upload to server, and deploy to tomcat:

$ cp ncWMS-1.0.war $TOMCAT_HOME/webapps/ncWMS.war

Then follow the guide http://www.resc.rdg.ac.uk/trac/ncWMS/wiki/DownloadPage to configure ncWMS, here is a sample:

ncWMS Admin page

9. Install GeoNetwork (VM3)

  • Prepare geonetwork database:
$ createuser -s geonetwork -U postgres
$ createdb -O geonetwork -U geonetwork -l en_US.UTF-8 -E UTF-8 -T template0 geonetwork
$ psql geonetwork geonetwork
psql (8.4.13)
Type "help" for help.

geonetwork=# ALTER USER geonetwork ENCRYPTED PASSWORD 'geonetwork';
ALTER ROLE
geonetwork=# \q
$ cp geonetwork.war $TOMCAT_HOME/webapps/
$ $TOMCAT_HOME/bin/shutdown.sh
$ $TOMCAT_HOME/bin/startup.sh
  • Configure datasource:

Open $TOMCAT_HOME/webapps/geonetwork/WEB-INF/config.xml, Find “postgresql” section and change to this, also remember to disable the default resource (Usually the first <resource> block):

                <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
                <!-- postgresql -->
                <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->

                <resource enabled="true">
                        <name>main-db</name>
                        <provider>jeeves.resources.dbms.ApacheDBCPool</provider>
                        <config>
                                <user>geonetwork</user>
                                <password>geonetwork</password>
                                <!-- we use org.postgis.DriverWrapper in place of
                                org.postgresql.Driver to support both postgresql and postgis -->
                                <driver>org.postgresql.Driver</driver>

                                <url>jdbc:postgresql://localhost:5432/geonetwork</url>
                                <poolSize>10</poolSize>
                                <validationQuery>SELECT 1</validationQuery>
                        </config>
                </resource>
  • Configure <layer> block in $TOMCAT_HOME/webapps/geonetwork/WEB-INF/config-gui.xml:

Make sure layer names match geoserver layer names (“Data->Layer Preview”), geonetwork uses following layers by default, note that layer name might be different, see “10. Install GeoServer WAR”:

  <mapSearch options="{projection: 'EPSG:4326', maxExtent: new OpenLayers.Bounds(-180,-90,180,90), units: 'degrees', restrictedExtent: new OpenLayers.Bounds(-180,-90,180,90)}">
  <layers>
    <layer server="http://localhost:8080/geoserver/wms" tocName="Borders"  params="{layers: 'geonetwork:ne_50m_admin_0_boundary_breakaway_disputed_areas,geonetwork:ne_50m_admin_0_boundary_lines_land,geonetwork:ne_50m_coastline', transparent: 'true', format: 'image/png'}"  options="{}" />
    <layer server="http://localhost:8080/geoserver/wms" tocName="Ortophoto" params="{layers: 'geonetwork:bluemarble_jpeg_small', format: 'image/jpeg'}" options="{isBaseLayer: true}" />
  </layers>
 </mapSearch>

  <mapViewer options="{projection: 'EPSG:4326', maxExtent: new OpenLayers.Bounds(-180,-90,180,90), units: 'degrees', restrictedExtent: new OpenLayers.Bounds(-180,-90,180,90)}">
  <layers>
    <layer server="http://localhost:8080/geoserver/wms" tocName="Borders"  params="{layers: 'geonetwork:ne_50m_admin_0_boundary_breakaway_disputed_areas,geonetwork:ne_50m_admin_0_boundary_lines_land,geonetwork:ne_50m_coastline', transparent: 'true', format: 'image/png'}"  options="{}" />
    <layer server="http://localhost:8080/geoserver/wms" tocName="Ortophoto" params="{layers: 'geonetwork:bluemarble_jpeg_small', format: 'image/jpeg'}" options="{isBaseLayer: true}" />
  </layers>

Then restart tomcat:

$ $TOMCAT_HOME/bin/shutdown.sh
$ $TOMCAT_HOME/bin/startup.sh

10. Install GeoServer WAR (VM3)

Official manual is http://docs.geoserver.org/stable/en/user/

$ createuser -s shed -U postgres
$ createdb -O shed geos -U postgres
$ psql geos -U shed
psql (8.4.13)
Type "help" for help.

geos=# \l
                              List of databases
   Name    |  Owner   | Encoding | Collation | Ctype |   Access privileges
-----------+----------+----------+-----------+-------+-----------------------
 geos      | shed     | LATIN1   | en_AU     | en_AU |
 postgres  | postgres | LATIN1   | en_AU     | en_AU |
 template0 | postgres | LATIN1   | en_AU     | en_AU | =c/postgres
                                                     : postgres=CTc/postgres
 template1 | postgres | LATIN1   | en_AU     | en_AU | =c/postgres
                                                     : postgres=CTc/postgres
(4 rows)

geos=# ALTER ROLE shed ENCRYPTED PASSWORD 'shed'
ALTER ROLE

$ createlang plpgsql geos -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/postgis.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/postgis_comments.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/spatial_ref_sys.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/rtpostgis.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/raster_comments.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/topology.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/topology_comments.sql -U shed
$ psql -d geos -f /usr/share/pgsql/contrib/postgis-2.0/legacy.sql -U shed

Import default base maps of geonetwork (Use /usr/bin/shp2pgsql to convert any shape file format map layers to postgis layers if you need other maps):

From build machine

$ scp ~/aodn-portal/download/ne_50m_admin_0_boundary_breakaway_disputed_areas.sql shed@<VM3>:~
$ scp ~/aodn-portal/download/ne_50m_admin_0_boundary_lines_land.sql shed@<VM3>:~
$ scp ~/aodn-portal/download/ne_50m_coastline.sql shed@<VM3>:~

Then on VM3

$ psql -d geos -f ne_50m_admin_0_boundary_breakaway_disputed_areas.sql -U shed
$ psql -d geos -f ne_50m_admin_0_boundary_lines_land.sql -U shed
$ psql -d geos -f ne_50m_coastline.sql -U shed

Check if the map data is created:

$ psql geos -U shed
psql (8.4.13)
Type "help" for help.
	
geos=# \d
                                   List of relations
  Schema  |                           Name                           |   Type   | Owner
----------+----------------------------------------------------------+----------+-------
 public   | geography_columns                                        | view     | shed
 public   | geometry_columns                                         | view     | shed
 public   | ne_50m_admin_0_boundary_breakaway_disputed_areas         | table    | shed
 public   | ne_50m_admin_0_boundary_breakaway_disputed_areas_gid_seq | sequence | shed
 public   | ne_50m_admin_0_boundary_lines_land                       | table    | shed
 public   | ne_50m_admin_0_boundary_lines_land_gid_seq               | sequence | shed
 public   | ne_50m_coastline                                         | table    | shed
 public   | ne_50m_coastline_gid_seq                                 | sequence | shed
 public   | raster_columns                                           | view     | shed
 public   | raster_overviews                                         | view     | shed
 public   | spatial_ref_sys                                          | table    | shed
 topology | layer                                                    | table    | shed
 topology | topology                                                 | table    | shed
 topology | topology_id_seq                                          | sequence | shed
(14 rows)
  • Add BlueMarble_world map to geoserver:

From build machine

$ scp ~/aodn-portal/download/BlueMarble_world.tar.gz shed@<VM3>:~

Then on VM3

$ cd $TOMCAT_HOME/webapps/geoserver/data/coverages/
$ tar xzvf /path/to/BlueMarble_world.tar.gz
$ $TOMCAT_HOME/bin/shutdown.sh
$ $TOMCAT_HOME/bin/startup.sh
  • Create a new “geonetwork” workspace, from left menu pane, click “Data->Workspaces->Add new workspace”:

GeoNetwork New Workspace

  • Enable “geonetwork” workspace and all 3 services:

GeoNetwork Enable Workspace and Services

  • Create a new “BlueMarble_world” store, click “Data->Stores->Add new Store”, then select “Raster Data Sources->GeoTIFF”, make sure select “geonetwork” for Workspace and “file:coverages/BlueMarble_world/bluemarble_jpeg_small.tiff” for URL of “Connection Parameters”

GeoNetwork Add BlueMarble_world

GeoNetwork Select BlueMarble URL

After click save, select “Publish”, then “Save”, you do not touch anything here.

GeoNetwork Publish BlueMarble_world

GeoNetwork Publish BlueMarble_world Save

  • Create a new “naturalearth_boundaries” store, click “Data->Stores->Add new Store”, then select “Vector Data Sources->PostGIS”, make sure select “geonetwork” for Workspace, then fill in only host/port/database/user/password according to your geos connection string. Then click “Save” to create this store.

GeoNetwork Add Naturalearth Bundaries Store

  • Create and publish new layers from “geonetwork:naturalearth_boundaries”, click “Data->Layers->Add a new resource”, select “geonetwork:naturalearth_boundaries”, then click “Publish”, and click “Compute from native bounds” to fill out the Lat/Lon Bounding Box, then click “Save” to publish this layer. Repeat for other 2 layers in this store.

GeoNetwork Add Layers from Naturalearth_Boundaries

GeoNetwork Compute Bounding Box

11. Install 52north SOS (VM3)

The official manual is https://svn.52north.org/svn/swe/main/SOS/Service/trunk/SOS/52n-sos/doc/howto/how2install_SOS.pdf if you need more details

From build machine

$ scp ~/aodn-portal/download/datamodel_postgres83.sql shed@<VM3>:~
$ scp ~/aodn-portal/download/phenomena.sql shed@<VM3>:~
$ scp ~/aodn-portal/download/52nSOS.tar.gz shed@<VM3>:~

Then on VM3

$ createuser -s sos -U postgres
$ createdb -O sos sosdb -U postgres
$ psql sosdb -U sos
psql (8.4.13)
Type "help" for help.

sosdb=# ALTER USER sos ENCRYPTED PASSWORD 'sos';
ALTER ROLE
sosdb=# \l
                              List of databases
   Name    |  Owner   | Encoding | Collation | Ctype |   Access privileges
-----------+----------+----------+-----------+-------+-----------------------
 geos      | shed     | LATIN1   | en_AU     | en_AU |
 postgres  | postgres | LATIN1   | en_AU     | en_AU |
 sosdb     | sos      | LATIN1   | en_AU     | en_AU |
 template0 | postgres | LATIN1   | en_AU     | en_AU | =c/postgres
                                                     : postgres=CTc/postgres
 template1 | postgres | LATIN1   | en_AU     | en_AU | =c/postgres
                                                     : postgres=CTc/postgres
(5 rows)

$ createlang plpgsql sosdb -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/postgis.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/postgis_comments.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/spatial_ref_sys.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/rtpostgis.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/raster_comments.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/topology.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/topology_comments.sql -U sos
$ psql -d sosdb -f /usr/share/pgsql/contrib/postgis-2.0/legacy.sql -U sos
$ psql -d sosdb -f datamodel_postgres83.sql -U sos
$ psql -d sosdb -f phenomena.sql -U sos
$ $TOMCAT_HOME/bin/shutdown.sh
$ cd $TOMCAT_HOME/webapps/
$ tar xzvf ~/52nSOS.tar.gz
$ $TOMCAT_HOME/bin/startup.sh

Configure datasource in $TOMCAT_HOME/webapps/52nSOS/WEB-INF/conf/dssos.config, change CONNECTIONSTRING/user/password to your settings.

12. Install AODN-Portal (VM1)

THIS DEPLOYMENT GUIDE ONLY APPLIES TO PRODUCTION ENVIRONMENT

Install and sync time (Important!!!)

$ sudo yum install ntp
$ sudo ntpdate server 0.centos.pool.ntp.org

Install aodn-portal

From build machine

$ git clone https://github.com/IntersectAustralia/aodn-portal.git
$ cd aodn-portal
$ git pull
$ git checkout <release_tag>
$ grails -Dgrails.env=production clean
$ grails -Dgrails.env=production cmopile
$ grails -Dgrails.env=production war
$ scp target/Portal2-<release_tag>-production.war shed@<VM1>:~
$ scp download/wmsscanner.war shed@<VM1>:~

Then on VM1

$ cp Portal2-<release_tag>-production.war $TOMCAT_HOME/webapps/ROOT.war

Configure aodn-portal

Add this line to $TOMCAT_HOME/conf/context.xml:

<Resource name="jdbc/aodn_portal" auth="Container" type="javax.sql.DataSource" driverClassName="org.postgresql.Driver" url="jdbc:postgresql://<VM3_HOST_URL>:5432/aodn_portal" username="shed" password="shed" maxActive="20" maxIdle="10" maxWait="-1" />

Deploy a wmsscanner

$ cp ~/wmsscanner.war $TOMCAT_HOME/webapps/

Configure wmsscanner

Add this line to $TOMCAT_HOME/conf/context.xml:

<Resource name="jdbc/wms_scanner" auth="Container" type="javax.sql.DataSource" driverClassName="org.postgresql.Driver" url="jdbc:postgresql://<VM3_HOST_URL>:5432/wms_scanner" username="shed" password="shed" maxActive="20" maxIdle="10" maxWait="-1" />

Create dataset storage and populate metadata scripts:

$ sudo mkdir /aodn-portal
$ sudo chown -R shed:shed /aodn-portal
$ mkdir /aodn-portal/data
$ mkdir /aodn-portal/data/colRecords
$ mkdir /aodn-portal/data/ptyRecords
$ mkdir /aodn-portal/scripts

From build machine

$ scp aodn-portal/scripts/metadata/carbondioxide.groovy shed@<VM1>:/aodn-portal/scripts
$ scp aodn-portal/scripts/metadata/carbondioxide-delete.groovy shed@<VM1>:/aodn-portal/scripts
$ scp aodn-portal/scripts/metadata/nutrient.groovy shed@<VM1>:/aodn-portal/scripts
$ scp aodn-portal/scripts/metadata/nutrient-delete.groovy shed@<VM1>:/aodn-portal/scripts
$ scp aodn-portal/scripts/metadata/sonde.groovy shed@<VM1>:/aodn-portal/scripts
$ scp aodn-portal/scripts/metadata/sonde-delete.groovy shed@<VM1>:/aodn-portal/scripts

Install groovy and jdbc driver on VM1

$ curl -s get.gvmtool.net | bash
$ gvm install groovy 1.7.8
$ sudo yum install postgresql-jdbc
$ mkdir -p ~/.groovy/lib
$ cp /usr/share/java/postgresql-jdbc-8.4.701.jar ~/.groovy/lib/

Configure Time Series Database connection

$ cd /aodn-portal/scripts/
$ sed -i.bak -e 's/localhost/<VM3 Hostname>/g' *.groovy
$ sed -i.bak -e 's/aodn/sos/g' *.groovy

The database connection string now should read like this:

db = [url:"jdbc:postgresql://<VM3 Hostname>:5432/${args[1]}", user:'sos', password:'sos', driver:'org.postgresql.Driver']

13. Install & Configure jOAI (VM1)

13.1 Overview

jOAI is developed by Digital Learning Sciences (DLS) (http://www.dlsciences.org/) at the University Corporation for Atmospheric Research (http://www.ucar.edu/). jOAI can be installed on Windows, Linux, Mac OS or UNIX systems. It is a Java-based open source Open Archives Initiative data provider and harvester tool that runs in a servlet container such as Apache Tomcat.

The jOAI data provider allows XML files from a file system to be exposed as items in an OAI data repository and made available for harvesting by others using the OAI-PMH. After pointing the software to one or more file directories, the software monitors the XML files inside, adding, updating or deleting them from the OAI repository as files are added, updated or deleted from the directories.

The jOAI harvester is used to retrieve metadata records from remote OAI data providers and save them to the local file system, one record per file. In addition, records that have been harvested are packaged into zip archives that can be downloaded and opened through the harvester's web-based interface.

13.2 Installation

Three steps are involved:

13.2.1 jOAI system requirements

The jOAI software runs on Windows, Mac, Linux and Unix platforms. The following components are needed to operate the software:

oai.war - The jOAI software Apache Tomcat servlet container, version 5.5.x or 6.x Java Standard Edition (SE) version 5 or 6

13.2.2 Download jOAI from: http://www.dlese.org/oai/docs/provider.jsp

13.2.3 Install jOAI

Four steps are involved:

  1. Place the file 'oai.war' into the 'webapps' directory of your tomcat.
  2. Start or restart Tomcat. Upon startup the first time, Tomcat will automatically unpack the oai.war archive, creating a directory and application context named 'oai'.
  3. The jOAI software should now be running. Launch a web browser and type in the URL to the oai servlet context. For example: http://localhost:8080/oai/provider. If there are errors accessing the software try repeating or verifying steps 1 and 2. Tip: Check the Tomcat logs for error messages located in the 'logs' directory of your Tomcat installation if there are problems.
  4. At the URL mentioned above, you will be greeted by a 'Getting started' page that will walk through the remaining few steps necessary to use the OAI harvester, data provider or repository search portions of the software.

13.3 Configure jOAI to work as a RIF-CS OAI-PMH Data Provider.

STEP 1.

Complete the 'Repository Information' by clicking 'Edit repository info' in the Repository Information and Administration page and then:

  • Enter a repository name (required)
  • Include an administrators e-mail address (required)
  • Provide a namespace identifier (optional but strongly recommended)
  • Provide a description (optional)

The namespace-identifier is similar to an Internet domain name, for example "dlese.org" or "project.dlese.org." If specified, the namespace identifier is used to compose the OAI Identifier for items in the repository. See the OAI Identifier Format guidelines for more information. Leave the description blank if not using.

STEP 2.

Complete the 'Metadata Files Configuration' in the Metadata Files Configuration page by adding one or more metadata directories to the repository. For each directory:

  • Enter an appropriate nickname for the directory of files (required)
  • Provide the metadata format of the files (required)
  • Enter the complete directory path to the metadata files (required)
  • Enter the metadata namespace and schema for the format (recommended)
1. Collection Record.
2. Party Record.

STEP 3.

After completing STEP 2, the software automatically indexes the metadata files, which may take several minutes to complete. Once the files are indexed, the metadata is available for harvesting:

Identify:

http://localhost:8080/oai/provider?verb=Identify

RIF-CS:

http://localhost:8080/oai/provider?verb=ListRecords&metadataPrefix=rif

You can also browse using the OAI protocol via the Explorer page and for textual searching using the Search or Admin search pages. The 'Metadata Files Configuration' shows information about the status of the files and indexing process. If metadata files are added, modified or deleted at a later time, the software automatically detects these changes and adds, deletes or re-indexes them every 8 hours . The index can also be updated manually at any time from the 'Files index administration' area.

13.4 Enable access control in Tomcat

Restrict access to the software administrative pages so that the public does not have access to sensitive information or can change administrative settings. To enable password protection for the administrative pages, do the following (this assumes use of at least Tomcat v 5.x):

  1. Uncomment the 'security-constraint' and 'login-config' elements found in the oai "WEB-INF/web.xml" file. Assuming a default installation, this would be located at $CATALINA_HOME/webapps/oai/WEB-INF/web.xml($CATALINA_HOME refers to the location of the Tomcat installation).

  2. Copy the following Context and Realm elements into the <Host> element of the 'server.xml' configuration file found in the Tomcat 5.x 'conf' directory (the configuration below assumes the OAI context and directory name is 'oai'):

<Realm className="org.apache.catalina.realm.MemoryRealm" pathname="webapps/oai/WEB-INF/users.xml"/>
<Context path="/oai" docBase="oai" debug="0" reloadable="true"/>
  1. Edit the file 'WEB-INF/users.xml' to define the user names and passwords for those you wish to grant access to the OAI administration pages. An example user and instructions are provided in that file.

  2. Start or restart Tomcat.

Note that this type of authentication does not provide encryption for user names and passwords sent over the Internet.

14. Install AAF shibboleth (VM4)

Please refer to http://wiki.aaf.edu.au/tech-info/sp-install-guide for a full document.

Download and installation (as root)

# yum install httpd mod_ssl
# wget http://download.opensuse.org/repositories/security://shibboleth/CentOS_CentOS-6/security:shibboleth.repo -P /etc/yum.repos.d
# yum install shibboleth

Generate certificate with the correct hostname

Run the following, substituting the externally visible hostname for sp.example.org:

# cd /etc/shibboleth
# ./keygen.sh -f -h sp.example.org -e https://sp.example.org/shibboleth

Configuration

Download AAF metadata signing certificate

AAF Test Federation

# wget https://ds.test.aaf.edu.au/distribution/metadata/aaf-metadata-cert.pem -O /etc/shibboleth/aaf-metadata-cert.pem

AAF Production Federation

# wget https://ds.aaf.edu.au/distribution/metadata/aaf-metadata-cert.pem -O /etc/shibboleth/aaf-metadata-cert.pem

Edit /etc/shibboleth/shibboleth2.xml

Replace all instances of sp.example.org with your hostname.

In the <Sessions> element, make session handler use SSL: set handlerSSL="true"

Recommended: go even further and in the Sessions element, change the handlerURL from a relative one ("/Shibboleth.sso" to an absolute one - handlerURL="https://sp.example.org/Shibboleth.sso". In the URL, use the hostname used in the endpoint URLs registered in the Federation Registry. This makes sure the server is always issuing correct endpoint URLs in outgoing requests, even when users refer to the server with alternative names. This is in particular important when there are multiple hostnames resolving to your server (such as one prefixed with "www." and one without).

Optionally, customize in the <Errors> element the pages and settings. Your users will come in contact with these if an error occurs. Change the SupportContact attribute to something more meaningful than root@localhost.

Load the federation metadata: add the following (or equivalent) section into /etc/shibboleth/shibboleth2.xml just above the sample (commented-out) MetadataProvider element

AAF Test Federation

<MetadataProvider type="XML" uri="https://ds.test.aaf.edu.au/distribution/metadata/metadata.aaf.signed.complete.xml"
     backingFilePath="metadata.aaf.xml" reloadInterval="7200">
   <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
   <MetadataFilter type="Signature" certificate="aaf-metadata-cert.pem"/>
</MetadataProvider>

AAF Production Federation

<MetadataProvider type="XML" uri="https://ds.aaf.edu.au/distribution/metadata/metadata.aaf.signed.complete.xml"
     backingFilePath="metadata.aaf.xml" reloadInterval="7200">
   <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
   <MetadataFilter type="Signature" certificate="aaf-metadata-cert.pem"/>
</MetadataProvider>

Configure Session Initiator

  1. Locate the <SSO> element (new in SP 2.4) and:
  • Remove reference to default idp.example.org - delete the entityID attribute
  • Configure the Discovery Service URL in the discoveryURL attribute:

AAF Test Federation

discoveryURL="https://ds.test.aaf.edu.au/discovery/DS"

AAF Production Federation

discoveryURL="https://ds.aaf.edu.au/discovery/DS"
  1. Attributes received from the IdP have to be mapped. This is configured in the attribute-map. Change the attribute mapping definition by either editing attribute-map.xml to accept attributes. Block those attributes irrelevant for your SP as necessary.

  2. Attributes can be filtered out by Shibboleth. These filter rules are defined in attribute-policy.xml. However, the default setting is to just keep it as-is.

  3. Leave <CredentialResolver> to point to the default keypair.

64-bit platforms

On x86_64, if you have installed also the i386 version of shibboleth and its configuration Apache configuration file is taking over, edit /etc/httpd/conf.d/shib.conf and change the path to the Shibboleth Apache module to the 64-bit version:

 LoadModule mod_shib /usr/lib64/shibboleth/mod_shib_22.so

Attribute-policy.xml

The attribute-policy.xml file describes rules to filter out attributes or let them pass. More information on how to set up these rules is described on https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPAttributeFilter

Logging

There are 2 different Shibboleth-related log files you can access for troubleshooting.

  • native.log: is located in /var/log/httpd and can be configured in /etc/shibboleth/native.logger
  • shibd.log: is located in /var/log/shibboleth and can be configured in /etc/shibboleth/shibd.logger

Make sure that the right processes have write permissions to the log files!

Protect a resource

You can protect a resource with Shibboleth by configuring your Apache webserver. Edit the file /etc/httpd/conf.d/shib.conf, replace /secure with /auth/logon:

<Location /secure>
    ShibRequestSetting authType shibboleth
    ShibRequestSetting requireSession false
    require valid-user
</Location>

More information on how to protect your resource can be found on https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPhtaccess.

Finishing up

Start up Apache and shibd:

# service httpd start
# service shibd start
# chkconfig httpd on
# chkconfig shibd on

Testing

Place a script inside the protected directory. PHP example script such as the following is good enough: <?php print_r($_SERVER) ?>

Access the protected directory/script (http://your.server/secure) from your browser, this should trigger a complete SSO cycle where you can authenticate on your IdP.

Upon successful authentication, the page should display all received attributes. Make sure you have non empty Shib-Application-ID amongst other attributes (if your IdP release them).

Check your shibd.log to see if there are attributes received or errors encountered.

Startup

We recommend shibd start before apache start so that in case of shibd errors, apache isn't trying to make erroneous connections while something is being fixed.

15. Install and configure Apache and Tomcat integration (VM1, VM2, VM4)

In the setup described here, requests from browsers are intercepted first by Apache httpd. The Shibboleth SP then checks these requests to enforce authentication requirements. After an assertion is received and a Shibboleth session is established, the SP or Apache httpd can enforce access control rules, or it can just pass attributes to the application. The request is then forwarded to the servlet through the use of the AJP13 protocol. Subsequent requests can leverage the Shibboleth session or a session maintained by the application or servlet container to persist the login.

Setup AJP13 support in your servlet container (VM1)

Tomcat has an AJP 1.3 connector enabled by default. Setting the tomcatAuthentication="false" attribute on the AJP <Connector> element allows for passing REMOTE_USER from Apache httpd.

Be careful that there is no direct HTTP listener opened by the servlet container. If, for example, there's an HTTP connector listening on port 8080 and no interceding firewall, users would be able to directly access the servlet on port 8080, which bypasses Apache httpd. This also means they would bypass Shibboleth authentication and authorization.

AJP packet size

Service Providers that request many attributes or receive many attribute values can expect to exceed the default maximum AJP packet size (8kb). In order to prevent this, raise the maximum AJP packet size to 65kb (maximum allowed by the AJP protocol). This value should be specified both in Apache httpd and your servlet container configuration.

Tomcat (VM1): Add a packetSize="65536" to the AJP <Connector> element.

Configure Apache httpd to route requests to your servlet

Add a tomcat.conf file to /etc/httpd/conf.d with these lines, to map requests on the proper virtual hosts to your application through AJP 1.3.

Apache httpd with mod_proxy_ajp (VM4): Add a ProxyIOBufferSize directive.

ProxyIOBufferSize 65536
ProxyPass /Shibboleth.sso !
ProxyPass /geoserver ajp://<VM3>:8009/geoserver
ProxyPass /geonetwork ajp://<VM3>:8009/geonetwork
ProxyPass /thredds ajp://<VM2>:8009/thredds
ProxyPass /ncWMS ajp://<VM2>:8009/ncWMS
ProxyPass / ajp://<VM1>:8009/

Note: Sometimes mod_proxy_ajp does not work well with firewall if firewall enable dropping idle connection. When this happens you could receive a 503 Service Temporarily Unavailable error message. If you check apache's /var/log/httpd/error_log, you will also see much of this errors:

[Mon Jun 17 09:18:41 2013] [error] (70007)The timeout specified has expired: proxy: read response failed from <backend_tomcat>:8009 (<backend-tomcat server name>)
[Mon Jun 17 09:18:41 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

You can add keepalive=On connectiontimeout=240 to reverse proxy configuration to work around this issue, e.g.:

ProxyIOBufferSize 65536
ProxyPass /Shibboleth.sso !
ProxyPass /geoserver ajp://<VM3>:8009/geoserver keepalive=On connectiontimeout=240
ProxyPass /geonetwork ajp://<VM3>:8009/geonetwork keepalive=On connectiontimeout=240
ProxyPass /thredds ajp://<VM2>:8009/thredds keepalive=On connectiontimeout=240
ProxyPass /ncWMS ajp://<VM2>:8009/ncWMS keepalive=On connectiontimeout=240
ProxyPass / ajp://<VM1>:8009/ keepalive=On connectiontimeout=240

Here connectiontimeout=240 is 240s. See https://httpd.apache.org/docs/2.2/mod/mod_proxy.html for details about parameters.

And you also need to change AJP connector configuration ($TOMCAT_HOME/conf/server.xml) on tomcat server, read like this:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false" packetSize="65536" connectionTimeout="240000" />

Here connectionTimeout="240000" is 240000ms. See http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html for details about parameters.

The value of connectionTimeout must be less than the idle connection timeout value set in the firewall. (In the example the firewall has a 5 minutes to drop idle connections)

For security reason, after successfully setting up and testing AJP connections, the default 8080 connector in tomcat should be closed. So that nobody can work around authentication system to access 8080 directly. In $TOMCAT_HOME/conf/server.xml, comment out the 8080 connector as this:

<!--
<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           redirectPort="8443" />
-->

Since environment variables are not passed by mod_proxy_ajp unless they have AJP_ prefixes, you'll also need to add attributePrefix="AJP_" to the <ApplicationDefaults> (or appropriate <ApplicationOverride>) element in your shibboleth2.xml:

<ApplicationDefaults id="default" policyId="default"
    entityID="https://sp.example.org/shibboleth"
    REMOTE_USER="eppn persistent-id targeted-id"
    signing="false" encryption="false"
    attributePrefix="AJP_">

16. Configure Tomcat to Run as a Service (VM1, VM2, VM3)

We will now see how to run Tomcat as a service and create a simple Start/Stop/Restart script, as well as to start Tomcat at boot.

Change to the /etc/init.d directory and create a script called 'tomcat' as shown below:

$ sudo -i
# cd /etc/init.d
# vi tomcat

And here is the script we will use:

#!/bin/bash
# description: Tomcat Start Stop Restart
# processname: tomcat
# chkconfig: 234 20 80
JAVA_HOME=/usr/lib/jvm/jre-1.6.0-sun.x86_64
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH
CATALINA_HOME=/home/shed/apache-tomcat-7.0.37

case $1 in
start)
/bin/su - shed $CATALINA_HOME/bin/startup.sh
;; 
stop)
/bin/su - shed $CATALINA_HOME/bin/shutdown.sh
;; 
restart)
/bin/su - shed $CATALINA_HOME/bin/shutdown.sh
/bin/su - shed $CATALINA_HOME/bin/startup.sh
;; 
esac    
exit 0

Now, set the permissions for your script to make it executable:

# chmod 755 tomcat

We now use the chkconfig utility to have Tomcat start at boot time.

# chkconfig --add tomcat
# chkconfig --level 234 tomcat on
# chkconfig --list tomcat
tomcat          0:off   1:off   2:on    3:on    4:on    5:off   6:off

Now, do quick tests:

# service tomcat start
# service tomcat stop
# service tomcat restart

17. Essential configurations

Now we have all the components deployed and running. It's time to do some essential configurations in order to do Smoke Tests.

Site Configuration

Login aodn-portal as administrator. Then go to "Administration"->"Site Configuration", and fill in fields as following:

Field Value Note
Site Name Sydney Harbour Observatory This is displayed as the head line in home page
Initial Map Bounding Box 151.00691,-33.98598,151.39590,-33.68935 This will zoom in map just for sydney harbour area
Default Dateline Zoom Bounding Box 151.00691,-33.98598,151.39590,-33.68935
Catalog Url http://uat.sho.sydney.edu.au/geonetwork This is the geonetwork instance we are using
WMS Scanner Callback Password 123456 Must be set, but value is not important
WFS Scanner Callback Password 123456 Must be set, but value is not important

Site Configuration First Part Site Configuration Second Part

Set up WMS server

Go to "Administration"->"Server"->"Create", and fill in fields as following:

Field Value Note
Uri http://uat.sho.sydney.edu.au/ncWMS/wms WMS URL of ncWMS server
Short Acron ncWMS
Type WMS-1.3.0
Name Sydney Harbour Environment Data Displayed in map layers tree
Allow Discoveries Yes

Set Up WMS Server

Then go to "Administration"->"WMS Scanner Controls". Click "Create Scan Job" for "Sydney Harbour Environment Data"

Create Scan Job

If everything is ok, you will see the "Server List", and "WMS Scanner Status" will display last scanned time stamp

WMS Scanner Status

Set up layers menu

Go to "Administration"->"Menu"->"Default Layers Menu", drag "Sydney Harbour Environment Data" grey bar to the menu tree, click "Save Edited Menu" to save the configuration.

Set Up Layers Menu

18. Smoke Tests

Smoke Tests is used to verify all components and aodn-portal are deployed correctly. Also after updating aodn-portal or any other components, including OS/Java/Firewall, please run Smoke Tests again to make sure everything still work.

19. Troubleshooting

wmsscanner callback url connection refused

If click Administration -> WMS Scanner Controls -> Test and get any network issues, like connection refused, no route to host, etc.

To resolve this issue, make sure VM1 can connect to callback host on specified port, which is 80 for http and 443 for https. For example if the call backurl is http://example.com, this can be tested by:

$ telnet example.com 80

Otherwise if the call back url is https://example.com

$ telnet example.com 443

If succeed you should get something like this:

Trying 1.1.1.1...
Connected to example.com.
Escape character is '^]'.

wmsscanner call back url SSLHandshakeException

If callback url test gives this error message:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This means the certificate of callback host is not in VM1's jvm keystore, you can download the certificate from callback host and import it:

$ sudo keytool -import -noprompt -trustcacerts -file <certificate> -keystore <KeystoreFile> -storepass <Password>

The default keystorefile is "$JAVA_HOME/lib/security/cacerts" The default password is "changeit"

After that you need to restart tomcat to use the certificate.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.