More Details on Installing the THREDDS Data Server
####Special thanks to Roland (at NOAA PMEL) for providing this detailed THREDDS installation guide.
###THREDDS Installation
We use the THREDDS data server to provide some of the services LAS needs. To make this all work you'll have to install TDS and integrate it with LAS.
<div class="plain">
N.B. If you ever need to upgrade your TDS in an existing LAS installation, see the upgrading section below.
N.B. We have noticed that our Tomcat environment is much more stable if we tell robots to not crawl our THREDDS catalog HTML interface. We actually exclude robots from our entire Tomcat site with a robots.txt file in the ROOT webapp directory. The robots.txt file we use contains:
User-agent: *
Disallow: /
N.B. We know that the integration of LAS and F-TDS can sometimes be a little confusing. This section has some tips for debugging the F-TDS and LAS integration.
LAS allows users to define new variables which are analytic transformations of existing variables. For example, a user can define a new variable that is the average over all depths from an existing LAS variable that is defined on latitude, longitude and depth. The implementation of the transformation is done via the integration of the analytical capabilities of Ferret and the OPeNDAP data services of the THREDDS Data Server. Installing and integrating TDS with your LAS installation is what allows user defined variables and enables automatic interpolation between grids when making comparison plots (like differences between variables on different grids) possible. A separate write-up has the full details on the integration between TDS and Ferret (as used by LAS or as an independent OPeNDAP server). So let's get started getting TDS ready to be used by LAS.
Important Note: The LAS configuration process assumes you have the same proxy setup for your TDS server as you do for your LAS server. If you have added proxy directives to an Apache server for LAS, you'll need to add similar proxy statements to the configure for your THREDDS server using. For example your LAS set up is as follows:
<Location /las>you'll need a similar setup for your THREDDS server:
ProxyPass http://myhost.gov:8080/las
ProxyPassReverse http://myhost.gov:8080/las
</Location>
<Proxy http://myhost.gov:8080/las>
AllowOverride None
Order allow,deny
Allow from All
</Proxy>
<Location /thredds>
ProxyPass http://myhost.gov:8080/thredds
ProxyPassReverse http://myhost.gov:8080/thredds
</Location>
<Proxy http://myhost.gov:8080/thredds>
AllowOverride None
Order allow,deny
Allow from All
</Proxy>
Install TDS.
This is pretty straight forward and there are excellent instructions at the Unidata site.
Temporary disk areas for data and temporary files.
The installation procedure will automatically create the data and temporary directories F-TDS uses. Then when LAS starts up it will automatically build the necessary Ferret journal files that will allow TDS to serve the data from your LAS via OPeNDAP. In order to do this, LAS needs to write a small journal file in a directory that is accessible by the tomcat host and allows read/write privileges by the tomcat user. The installation procedure will create this directory $TOMCAT_HOME/content/las/conf/server/data to hold those files.
The process of building and serving the LAS data and new variables based on these data requires some temporary disk space. An ideal directory would be large scratch area configured on your server which is not part of the regular back ups, but is accessible to the tomcat host and grants read/write privileges to the tomcat user. Files written to this area are also reused by F-TDS in the event that the same data are requested more that once so it's nice if they can stay around for a while, but if a file is not found it will be recreated as needed. The default directory selected by the configuration process is $TOMCAT_HOME/content/las/conf/server/temp.
If you feel you would be better served by changing the locations of these data and temporary directories you can do so by editing the $LAS_HOME/JavaSource/resources/ferret/FerretBackendConfig.xml and the $TOMCAT_HOME/JavaSource/resources/iosp/FerretConfig.xml files. The directories you choose must be the same for each of these files and you must redeploy the LAS server (cd $LAS_HOME; ant deploy) and restart tomcat server to see any changes.
Create a <datascan> element where the LAS data hierarchy will appear in TDS.
Now that TDS is installed and you've figured out where you're going to store the data and temporary files you can finish the integration of TDS with your LAS installation. Find the THREDDS catalog that is being used to configure your TDS. It's $TOMCAT_HOME/content/thredds/catalog.xml. Edit this file and add an entry so that it will server your LAS data.
<datasetScan name="Data From LAS" path="las" location="/usr/local/tomcat/content/las/conf/server/data" serviceName="gridded">
<filter>
<include wildcard="*.nc"/>
<include wildcard="*.fds"/>
<include wildcard="*.jnl"/>
</filter>
</datasetScan>
The location is the directory you identified or created as your data directory in the previous step. The serviceName should point to a service of serviceType="OpenDAP". Or a compound service that includes an OpenDAP service. For example, I use a compound service that looks like this.
<service name="gridded" serviceType="Compound" base="" >
<service name="opendap" serviceType="OpenDAP" base="/thredds/dodsC/"/>
<!-- Other services are defined here... -->
</service>
Configure in the FerretDataSource and FerretIOServiceProvider.
In order to use TDS with you LAS you have configure a couple of "plug-ins" into your TDS. These plug-ins allow TDS to serve data opened in a Ferret journal file via OPeNDAP and allow new data variables to be defined by passing F-TDS a special URL syntax. To add these capabilities to LAS you'll need to add the following to your $TOMCAT_HOME/content/thredds/threddsConfig.xml file.
<datasetSource>gov.noaa.pmel.tmap.iosp.FerretDataSource</datasetSource>
<nj22Config>
<ioServiceProvider class="gov.noaa.pmel.tmap.iosp.FerretIOServiceProvider"/>
</nj22Config>
Just add the blocks at the top of the file, between the <threddsConfig></threddsConfig> tags:
<?xml version="1.0" encoding="UTF-8"?>
<threddsConfig>
<datasetSource>gov.noaa.pmel.tmap.iosp.FerretDataSource</datasetSource>
<nj22Config>
<ioServiceProvider class="gov.noaa.pmel.tmap.iosp.FerretIOServiceProvider"/>
</nj22Config>
</threddsConfig>
Once you have made all of these changes you're ready to go on with your LAS installation. The configuration script will also compile and copy the FerretDataSource and FerretIOServiceProvider classes to the correct place in the TDS webapps directory.
When LAS starts it will write at least one .jnl file for each LAS data set into the data directory you created in this step. When you're done installing LAS and have restarted you tomcat server you should see your "Data From LAS" (or whatever you called it in your datasetScan element) in your THREDDS catalog. Go ahead and poke around in there and verify that you can manipulate those data sets via the THREDDS HTML interface.
Configure the logging for the FerretIOServiceProvider and the FerretDataSource (optional).
The TDS comes configured to write logging information into a special directory. The log messages are separated into different files based on the component of TDS that is logging the message. You add configuration information into the log4j.xml file so that message from the LAS components are written to their own file in the TDS logs directory.
Find the log4j.xml file in $TOMCAT_HOME/webapps/thredds/WEB-INF/ and copy it to $TOMCAT_HOME/content/thredds. This is necessary since the log configuration files is not one of the files that the THREDDS distribution will preserve when you up-grade your TDS.
Edit your copy in $TOMCAT_HOME/content/thredds to add an appender:
<appender name="ferretIOSP" class="org.apache.log4j.RollingFileAppender">
<param name="File" value="${tds.log.dir}/ferretIOSP.log"/>
<param name="MaxFileSize" value="10MB"/>
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd'T'HH:mm:ss.SSS Z} [%10r][%8X{ID}] %-5p - %c - %m%n"/>
</layout>
</appender>
and to add a logger:
<logger name="gov.noaa.pmel.tmap.iosp" additivity="false">
<level value="DEBUG"/>
<appender-ref ref="ferretIOSP"/>
</logger>
Save the file and copy it back to $TOMCAT_HOME/webapps/thredds/WEB-INF/. You have to restart you tomcat server for these changes to take effect. After the restart any logging messages written by the LAS components installed in TDS will appear in the $TOMCAT_HOME/content/thredds/logs/ferretIOSP.log file.
The first step is to upgrade your TDS as instructed. Typically this means replacing the $TOMCAT_HOME/webapps/thredds.war file, removing the $TOMCAT_HOME/webapps/thredds directory and restarting the tomcat server.
All of the TDS configuration changes you made to install your Ferret I/O Service Provider and associate the installation with LAS will be preserved in the $TOMCAT_HOME/content/thredds directory after the upgrade. Now all that remains to do is reinstall the actual software from LAS and the F-TDS specific resource files. This is a one line command from your $LAS_HOME directory.
Change to the $LAS_HOME (the place where you expanded the LAS tar file and ran the configure script) and type command:
% ant deploy-iosp
Then stop and start your Tomcat server and you're done.