Skip to content

Releasing NationalMap and InvestorMap

Jump to bottom Edit New page
Mike Wu edited this page Oct 1, 2021 · 32 revisions

Here is the process that we use at Data61 to release a new version of NationalMap or Investor Map to Geoscience Australia.

  • Steps marked "NM only" apply for NationalMap releases.
  • Steps marked "IM only" apply for Investor Map releases.
  • Steps marked "NM and IM" apply to both NationalMap and Investor Map releases.

Configure your Amazon Web Services credentials

You need:

  • Access to the nationalmap account on AWS.
  • Appropriate AWS credentials configured in a nationalmap profile so that you can upload files to S3 and deploy a CloudFormation stack.

If you are using macOS:

  • Install gnu-tar with brew install gnu-tar

Do SSO with nationalmap profile with command aws sso login --profile nationalmap then follow on screen instructions.

Release TerriaJS (NM only)

App releases should always use an official version of TerriaJS released on npm. The exception to this is droughtmap, which uses custom GitHub tagged versions to support droughtmap's pre-rendering.

master & next using automated GitHub Actions (preferred method):

  • Checkout and pull master/next branch of terriajs, run npm install if necessary.
  • Bump the version number in package.json.
  • Review and edit CHANGES.md.
    • Ensure that entries for new changes are written in the right section by diffing against the last version. Make sure the section name matches the version you set in package.json above.
    • e.g. git diff HEAD 8.0.0-alpha.46 -- CHANGES.md (where 8.0.0-alpha.46 is the previous published version)
    • Or using GitHub web interface https://github.com/TerriaJS/terriajs/compare/8.0.0-alpha.46...next (change the version and master/next to match the previous release and branch)
  • Commit and push your changes and make a PR to master/next branch.
  • Get someone to review and merge the PR.
  • Wait for slack notification of successful/failed publish.
  • 😄

Publishing to npm without the GitHub Action (not for standard releases):

This method should only be used to publish versions not at the tip of master or next. To publish a new version of TerriaJS to npm with this method, you'll need to be listed as a collaborator here. Existing collaborators can add new ones. Then:

  • Checkout and pull the branch of terriajs to be deployed, run npm install if necessary.
  • Bump the version number in package.json. Follow semver.
  • Review and edit CHANGES.md.
    • Ensure that entries for new changes are written in the right section by diffing against the last version. Make sure the section name matches the version you set in package.json above.
    • e.g. git diff HEAD 8.0.0-alpha.46 -- CHANGES.md (where 8.0.0-alpha.46 is the previous published version)
    • Or using GitHub web interface https://github.com/TerriaJS/terriajs/compare/8.0.0-alpha.46...next (change the version and master/next to match the previous release and branch)
  • Commit and push your changes and make a PR to the branch to be deployed.
  • Get someone to review and merge the PR.
  • Fetch and checkout the merge commit made by merging the PR.
  • rm -rf wwwroot/build
  • Make sure you don't have any changes in your working directory.
  • gulp lint release
  • npm publish --tag off-branch (for a release not at the tip of master or next a tag must be set, and it must not be latest or next. For releases on old versions you could use e.g. 6-5-x or 6-x-x as a tag)
  • Ensure that CHANGES.md on master or next is also updated to list the release and clearly note the nature and reason for release.

The above will publish a new version to npm and also tag that version on GitHub.

Update docs (currently only needed for master, docs for next will start being published soon):

  • Make sure you don't have any changes in your working directory.
  • Install python dependencies for docs (pip install -r doc/requirements.txt) or activate your virtual environment with the dependencies installed.
  • gulp docs
  • Publish the updated documentation to terria.io: npm run publish-doc. Verify that the documentation at https://docs.terria.io still works. Note that it may take a few minutes for GitHub Pages to actually publish your update.

Publish the latest catalogue to S3 (NM only)

This step is only necessary if the catalogue changed since last release.

  • Checkout and pull the master branch of NationalMap-Catalog, run npm install if ncessary.
  • Run gulp. If there any changes in your working directory afterward, inspect them (usually it will just be a new LGA being added or something like that) and then commit and push them.
  • Update/verify CHANGES.md.
  • Tag the catalogue release: git tag -a 2016-08-08 -m '2016-08-08 release'
  • Push the tag: git push origin 2016-08-08
  • Copy the catalogue to S3 by running ./publish.sh 2016-08-08 where 2016-08-08 is today's date. If, for some reason, you already have a deployed instance of NationalMap file named after today's date, add b to end of it or something like that.

Prepare NationalMap (NM only)

  • Update package.json to refer to the new version of TerriaJS that you published above. Verify that both NationalMap and TerriaJS use the same version of terriajs-cesium and react. Usually you'll update NationalMap to match TerriaJS. You can automate this by running gulp sync-terriajs-dependencies.
  • Update CHANGES.md. Include relevant changes from both terriajs and NationalMap-Catalog.
  • Copy the existing private client config file from S3: aws s3 --profile nationalmap cp <packagejson.config.awsS3ClientConfigOverridePath> ./wwwroot/privateconfig.json where <packagejson.config.awsS3ClientConfigOverridePath> is the value of the awsS3ClientConfigOverridePath config property in package.json.
  • Edit wwwroot/privateconfig.json with HTTP path to the new catalog file on S3. Don't forget to include the /proxy/_60s/ bit on the front. You can probably just change the date on the end.
  • Copy the new client config to S3: aws s3 --profile nationalmap cp ./wwwroot/privateconfig.json s3://nationalmap-apps/nationalmap/privateclientconfig-2016-08-08.json where 2016-08-08 is today's date.
  • Change the awsS3ClientConfigOverridePath property in package.json to the new path above.
  • rm wwwroot/privateconfig.json
  • If the private server config file needs to be changed, copy the existing file from S3: aws s3 --profile nationalmap cp <packagejson.config.awsS3ServerConfigOverridePath> ./privateserverconfig.json where <packagejson.config.awsS3ServerConfigOverridePath> is the value of the awsS3ServerConfigOverridePath config property in package.json.
  • Edit ./privateserverconfig.json.
  • Copy the new private server config to S3: aws s3 --profile nationalmap cp ./privateserverconfig.json s3://nationalmap-apps/nationalmap/privateserverconfig-2016-08-08.json where 2016-08-08 is today's date.
  • Change the awsS3ServerConfigOverridePath property in package.json to the new path above.
  • rm ./privateserverconfig.json
  • You may want to update awsEc2ImageId in package.json to refer to the latest version, which will make deployments go faster because fewer patches need to be installed. You can obtain the latest AMI ID with: curl -s https://cloud-images.ubuntu.com/query/bionic/server/released.current.txt | grep ap-southeast-2 | grep amd64 | grep hvm | grep ebs-ssd
  • Commit and push your changes (directly to master is fine). Verify that your working directory is clean.
  • Tag the release: git tag -a 2016-08-08 -m '2016-08-08 release'. Do not push the tag yet.

Prepare Investor Map (formerly Northern Australia Map) (IM only)

git clone https://github.com/TerriaJS/InvestorMap
cd InvestorMap

# Manual steps if it has been more than a week or two since the last release:

# Merge the latest changes from NationalMap into Investor Map. Commit and push
# these changes.
# There may be merge conflicts, particularly in package.json.
# Resolve these by hand, generally favouring HEAD (InvestorMap). Additionally,
# ensure any file removals are intended for Investor Map - keeping in mind that
# IM has custom regionMapping definitions
git remote add nm https://github.com/TerriaJS/NationalMap
git fetch nm
git merge nm/master

# Then `git add` the files.
git commit
git push -u origin master

# Make sure we have the latest catalog
rm -rf node_modules/nationalmap-catalog && npm install

Publish the Investor Map catalog (IM)

Automatic method:

npm run release-catalog

Or manually:

# Build the catalog
gulp build-catalog

#Publish the new catalog to S3. Update with today's date, like 2018-12-17
DATE=`date "+%Y-%m-%d"`
echo $DATE
gzip --keep wwwroot/init/investormap.json 
aws s3 --profile nationalmap cp --content-encoding gzip --content-type "application/json" wwwroot/init/investormap.json.gz s3://static.nationalmap.nicta.com.au/investormap/init/${DATE}.json
rm wwwroot/init/investormap.json.gz
  • Edit wwwroot/config.json to refer to the catalog file published above. Generally you will only need to change the date in initializationUrls.
# Commit and push this change.

git add wwwroot/init/investormap*.json
git add wwwroot/config.json
git commit -m 'Update initialization URL'
git push origin master

# Tag the release. (The most recent annotated tag is used as the subdomain name)
# Do not push the tag yet, in case something goes wrong and you need to tag a different commit and re-release.
git tag -a ${DATE} -m "${DATE} release"

Deploy a CloudFormation stack (NM and IM)

# This will do a full rebuild, then build a CloudFormation stack and upload. It will take awhile. You will need your MFA device.
aws sso login --profile nationalmap
export NODE_OPTIONS=--max_old_space_size=4096
npm run deploy

Note

Before running the above command,

  • Ensure that wwwroot/init/nm.json exists. Its content can be {}. This requirement is due to legacy. Should not be needed in the future.

In case of errors, try this

nvm use 10
npm run deploy-without-reinstall
  • Wait until both the CloudFormation is finished being created and the EC2 instance has finished initializing. You can check this by logging into the AWS console (in the NationalMap role).
  • Visit the new site at https://nationalmap-2016-08-08.nm.terria.io or https://investormap-2016-08-08.nm.terria.io where 2016-08-08 is the name of the tag you created above. Wait 10 minutes before trying. Verify everything works and you are happy to commit to this release.
  • If something is wrong:
    • delete the stack
    • delete the local tag that you hadn't pushed yet
    • go back and try again. :)
  • Point staging.nm.terria.io or inv-staging.nm.terria.io at the new stack using the AWS Route53 UI.
  • Remove the previous staging CloudFormation stack.

Test Investor Map at /investormap/ (IM only)

Test Investor Map at https://inv-staging.nm.terria.io/investormap/, especially for changes to help pages or token-enabled layers, to find any relative url issues that otherwise would only show up in GA's preprod environment.

Send the release to Geoscience Australia (NM and IM)

  • Push the tags you previously created to GitHub:
    • git push origin ${DATE} (Alternative: git push origin <tag-name>)
  • Get the SHA256 sum of the release package:
    • Windows: sha256sum deploy/packages/*.tar.gz.
    • Mac: shasum -a 256 deploy/packages/*.tar.gz.
  • Prepare an email to GA. Use previous emails as a template.
    • If there are any changes to the deployment (e.g. the user-data file), don't forget to include details in the email and to attach the file as well. Usually there aren't any changes.
    • The URL of the release package will be:
      • NM: https://staging.nm.terria.io/<SHA256SUM>/nationalmap-<DATE>.tar.gz
      • IM: https://inv-staging.nm.terria.io/<SHA256SUM>/investormap-<DATE>.tar.gz
Add a custom footer

I want to...

Data custodians

Developers:

Clone this wiki locally