Sea Ice Information System (SIIS) for the Sir David Attenborough (SDA).
This application is a mature alpha being prepared for its first test deployment.
The Sea Ice Information Service (SIIS) provides up-to-date sea ice information products for decision making and route planning when the RRS SDA is in, near or close to, sea ice in Antarctica and the Arctic.
SIIS is comprised of a number of components, represented by top-level directories:
- API
api/: Provides information about available layers/granules and key-value storage for the app component, in future will manage shore -> ship data syncing and other tasks as needed - Key-Value API (Temp)
api-kv/: Provides key-value storage for the app component temporarily until available from the main API - App
app/: Frontend application used by users to visualise sea ice information - GeoServer
geoserver/: Used to provide OGC services for sea-ice, and more general, data products - PostGIS: Persists settings and information in the App/API and provides storage for vector data products
This application supports exporting waypoints/routes to a file compliant with the IEC 61774:2015 standard (RTZ).
RTZ versions supported by this application: 1.0
See the BAS Metadata Library documentation for RTZ parameters/options supported by this application.
No RTZ extensions are supported by this application.
A common data directory, the SIIS data directory, is used for storing configuration and data for:
This data is stored separately from the Project Git repository (hosted in GitLab) because it contains artefacts rather than source code.
The authoritative version of this data directory is a volume in the BAS (Cambridge) SAN bucket: /data/siis/.
This section applies if you have changed files in a local copy of the SIIS data directory (i.e. in a development
environment). rsync can then be used to preview, and then upload, updates to the remote, authoritative, SIIS data
directory.
In general terms:
$ docker compose run rsync
# to preview changes
$ rsync -e "ssh -o StrictHostKeyChecking=no" -avzh --dry-run . [user]@bslcenb.nerc-bas.ac.uk:/data/siis/
# to perform changes
$ rsync -e "ssh -o StrictHostKeyChecking=no" -avzh --progress . [user]@bslcenb.nerc-bas.ac.uk:/data/siis/After making updates, ensure to communicate with other developers, currently via the MAGIC Slack workspace (internal).
This section applies if someone else has Updated The SIIS Data Directory and you have a local copy of the SIIS data directory (i.e. in a development environment).
It is important to pull outstanding changes down to avoid conflicts between files and other unintended side effects of an out of sync directory.
rsync can be used to first preview and then perform updates to a local ./data/ directory, from the remote,
authoritative, SIIS data directory.
$ docker compose run rsync
# to preview change
$ rsync -e "ssh -o StrictHostKeyChecking=no" -avzh --dry-run --exclude /gwc --exclude /datastagein --exclude /datastageout --exclude /data/xfer [user]@bslcenb.nerc-bas.ac.uk:/data/siis/ .
# to perform changes
$ rsync -e "ssh -o StrictHostKeyChecking=no" -avzh --progress --exclude /gwc --exclude /datastagein --exclude /datastageout --exclude /data/xfer [user]@bslcenb.nerc-bas.ac.uk:/data/siis/ .The GeoServer data directory is part of the SIIS Data Directory:
- location (locally):
./data/geoserver/ - location (BAS SAN):
/data/siis/geoserver/
This directly contains all layer definitions, styles and other configuration settings. It does not include datasets
(the inner data/ directory), see the Product Samples section for more information.
Local development instances:
- stdout
/usr/local/tomcat/logs/geoserver.log
The GeoWebCache configuration is tracked as files within the Project Data Directory:
./data/geoserver/gwc-gs.xml//data/siis/geoserver/gwc-gs.xml./data/geoserver/gwc/geowebcache.xml//data/siis/geoserver/gwc/geowebcache.xml./data/geoserver/gwc-layers/*]/data/siis/geoserver/gwc-gwc-layers/*./data/geoserver/global.xml(for the updatedupdateSequenceproperty) //data/siis/geoserver/global.xml
Changes to these configuration files need to captured by Updating The SIIS Data Directory:
Note: You do not need to capture the dated backup of these files that GeoServer generates automatically.
A FileBlobStore is used for storing generated tiles. This blob store should be used by default.
Within local development instances, the base directory for the blob store is: /data/siis/gwc/blobstore/, which is
mapped to runtime/gwc/ within the container.
Note: This blob store is not part of the SIIS Data Directory.
The PostGIS instance requires a set of SQL scripts to initialise the database structure used by the API and for storing Product Samples.
These scripts are stored as files within the SIIS Data Directory:
- location (locally):
./data/siis/psql/ - location (remote):
/data/siis/sql/
Changes to these scripts need to captured by Updating The SIIS Data Directory.
In local development environments, these scripts will be mapped to /docker-entrypoint-initdb.d/ within the PostgreSQL
container to be automatically executed on start up.
These scripts will need to be executed manually in other environments
A sample of each product this project should support is available to simulate the data that will exist in a real deployment. This sample is:
- representative of the the breadth of products that need to be supported
- not representative of the depth (volume) of product series that have a large, or variable, spatial/temporal extent
- a static and stable set of data, to enable reproducible results if (automated) tests are added in the future
Product samples are distributed as files within the SIIS Data Directory:
- location (locally):
./data/raw/ - location (S3):
/data/siis/data/raw/
Changes to extensions need to captured by Updating The SIIS Data Directory.
...
See the Usage section for how to use the application.
...
A local development environment is available using Docker containers, coordinated using Docker Compose:
# required local tools
$ brew install git
$ brew cask install docker
# clone project
$ git clone https://gitlab.data.bas.ac.uk/MAGIC/SIIS.git
$ cd ./SIIS
# pull container images
$ docker-compose pullPopulate your local data directory.
# start stack
# (note: the *app* service may display startup errors [1] which can be safely ignored)
$ docker-compose up
# (visit http://localhost:9000 when up)
# when finished
# (press [ctrl] + c to stop containers)
$ docker-compose downYou will need access to the BAS GitLab instance to clone the project Git repository.
You will need access to the BAS Private Docker Registry (provided by the BAS GitLab instance) to pull containers for this project, see the BAS GitLab README for more information.
[1]
app_1 | Server running at http://0.0.0.0:9000 - configured port 9000 could not be used.
app_1 | /bin/sh: lscpu: not found
JavaScript dependencies for the frontend application are managed using Yarn.
To check for outdated dependencies:
$ docker-compose run app ash
$ yarn outdatedTo upgrade dependencies:
# update `package.json` and `yarn.lock` files in a container instance
$ docker-compose run app ash
$ yarn upgrade-interactive
# copy updated files outside the container to persist and track changes in project repository
$ cp package.json yarn.lock assets/
$ exit
$ mv app/assets/package.json app/assets/yarn.lock app/
# rebuild application container to use updated package versions
$ docker-compose build app
$ docker-compose push appTo add new dependencies:
# update `package.json` and `yarn.lock` files in a container instance
$ docker compose run app ash
$ yarn add [package]
# copy updated files outside the container to persist and track changes in project repository
$ cp package.json yarn.lock assets/
$ exit
$ mv app/assets/package.json app/assets/yarn.lock app/
# rebuild application container to use updated package versions
$ docker compose build app
$ docker compose push appPEP-8 style and formatting guidelines must be used for this project, with the exception of the 80 character line limit.
Black is used to ensure compliance.
Black can be integrated with a range of editors, such as PyCharm, to perform formatting automatically.
To apply formatting manually:
$ docker-compose run api black api/To check compliance manually:
$ docker-compose run api black --check api/Checks are ran automatically in Continuous Integration.
Application packages are automatically created by Continuous Deployment for the frontend application when new releases (tags) are created. Packages are held in the per-project GitLab Package Registry:
- frontend application: compressed distribution bundle for deployment into a web root:
app-{version}-{environment}.zip: per environment bundle including source mapsapp-{version}-{environment}-no-src-maps.zip: per environment bundle without source maps, to reduce file sizeapp-{version}.zip: default bundle including source mapsapp-{version}-no-src-maps.zip: default bundle without source maps, to reduce file size
Self-contained deployment containers are built by Continuous Deployment for the frontend
application and backend API components. Images for these containers are tagged under the /deploy namespace in the
BAS Docker Registry.
All commits will trigger a Continuous Deployment process using GitLab's CI/CD platform, configured in .gitlab-ci.yml.
For all releases:
- create a release branch
- close release in
CHANGELOG.md - push changes, merge the release branch into
mainand tag with version
The maintainer of this project is the BAS Mapping and Geographic Information Centre (MAGIC). We welcome feedback by email addressed to magic@bas.ac.uk.
This project uses issue tracking, see the Issue tracker for more information.
Note: Read & write access to this issue tracker is restricted. Contact the project maintainer to request access.
Copyright (c) 2019-2022 UK Research and Innovation (UKRI), British Antarctic Survey (BAS).
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.