Upgrade Ubuntu operating system in production

[rbreslow]

Steps to take prior to release

• Reconcile marriagez-rings/Stroud_ModelMyWatershed/AWS/default.yml
  • Papertrail
  • Bastion instance type
  • DNS records
• Update RELEASES.md and MIGRATIONS.md with any changes (e.g. new Bastion hostname)
• Create a release/1.25.0 branch
• Build release AMIs using release docs
• Test that it works on staging
• Follow AMI Promotion instructions

Questions to answer prior to release

• Were past migrations modified to appease the Django upgrade?
• Do the on_delete alterations to existing tables lead to new database constraints or database constraint modifications?

Steps to take during the release

• Bring up the dark stack using release docs
• Connect to each node via SSH so that we can review logs in the event that things don't work
• Test the dark stack
• Apply migrations
• Clear tile cache and cut over to new stack

$ aws --profile mmw-stg s3 rm --recursive s3://tile-cache.modelmywatershed.org

• Ensure new tiles are getting cached to S3

Steps to take during the PostgreSQL upgrade

• In a tmux pane, run this script to check for connectivity:

while true;
do
    date
    PGPASSWORD= \
        psql -h database.service.mmw.internal -U modelmywatershed -d modelmywatershed -c "SELECT version();"
    sleep 5
done

• Vaccum the database:

modelmywatershed=> SET maintenance_work_mem='1GB';
modelmywatershed=> \timing
modelmywatershed=> VACUUM (FREEZE, ANALYZE, VERBOSE);

• Verify there is no unsupported usage, using the PostgreSQL upgrade docs
• Upgrade to 9.5.18 and apply the mmw-postgres95 parameter group
• Upgrade the Bastion to postgresql-client-9.5:

sudo apt-get update
sudo apt-get install postgresql-client-9.5

• Verify we aren't using any extensions and upgrade PostGIS:

modelmywatershed=> select * from pg_available_extensions where name like 'postgis%';
modelmywatershed=> ALTER EXTENSION postgis UPDATE;
modelmywatershed=> select postgis_full_version();

• Vaccum the database:

modelmywatershed=> SET maintenance_work_mem='1GB';
modelmywatershed=> \timing
modelmywatershed=> VACUUM (FREEZE, ANALYZE, VERBOSE);

• Upgrade to 9.6.14 and apply the mmw-postgres96 parameter group, using the RDS Console
• Upgrade the Bastion to postgresql-client-9.6:

sudo apt-get install postgresql-client-9.6

• Verify we aren't using any extensions and upgrade PostGIS:

modelmywatershed=> select * from pg_available_extensions where name like 'postgis%';
modelmywatershed=> ALTER EXTENSION postgis UPDATE;
modelmywatershed=> select postgis_full_version();

Steps to take to reconcile CloudFormation stack drift

• Apply the following diff:

diff --git a/deployment/cfn/vpc.py b/deployment/cfn/vpc.py
index d25db866..a9c693bb 100644
--- a/deployment/cfn/vpc.py
+++ b/deployment/cfn/vpc.py
@@ -43,9 +43,9 @@ class VPC(StackNode):
         'Region': 'us-east-1',
         'StackType': 'Staging',
         'KeyName': 'mmw-stg',
-        'AvailabilityZones': 'us-east-1b,us-east-1d',
-        'PublicSubnetCIDRRanges': '10.0.2.0/24,10.0.4.0/24',
-        'PrivateSubnetCIDRRanges': '10.0.3.0/24,10.0.5.0/24',
+        'AvailabilityZones': 'us-east-1a,us-east-1b',
+        'PublicSubnetCIDRRanges': '10.0.0.0/24,10.0.2.0/24',
+        'PrivateSubnetCIDRRanges': '10.0.1.0/24,10.0.3.0/24',
         'NATInstanceType': 't2.micro',
     }

• Generate new VPC stack template:

./mmw_stack.py launch-stacks \
    --aws-profile mmw-prd \
    --mmw-config-path default.yml \
    --mmw-profile production \
    --vpc \
    --print-json > ~/vpc-$(date +"%Y_%m_%d_%H_%M").json

• Apply update stack operation
• See that stacks are in-sync
• Generate new DataPlane stack template:

./mmw_stack.py launch-stacks \
    --aws-profile mmw-prd \
    --mmw-config-path default.yml \
    --mmw-profile production \
    --data-plane \
    --print-json > ~/data-plane-$(date +"%Y_%m_%d_%H_%M").json

• Apply update stack operation
• See that stacks are in-sync

Post release checks

• Vaccum the database:

modelmywatershed=> SET maintenance_work_mem='1GB';
modelmywatershed=> \timing
modelmywatershed=> VACUUM (FREEZE, ANALYZE, VERBOSE);

• Reboot the RDS instance to make sure the parameter group is synced
• Ensure new bastion DNS works
• Confirm logging to Papertrail works
• Ensure RWD works
• Ensure geoprocessing web service works

[rbreslow]

See that stacks are in-sync was not checked due to #3162.