[PVT#181020089] Implement better deployment monitoring and try out asset precompilation in GAH

[andyzito]

Pivotal Story

Logging

I replaced our manual log tailing with a call out to the awscliv2's log tailing, which mimics tail -f and is more stable than our homerolled version. This has significantly reduced the need for asking the user to confirm continuation of the deploy task if the logs can't be found/aren't up to date.

Monitoring

I just added basic timing to a bunch of deploy steps. Here are some times from a representative deployment:

STEP	SECONDS
download secrets (.env)	2
database materialize	1
sync_app_assets	2
clobbering assets	63
checksumming assets	1
------------------------------	-------
pulling compiled assets	4
rake maintenance:create	32
rake db:migrate:primary	36
rake db:migrate:warehouse	77
rake db:migrate:health	70
rake db:migrate:reporting	32
rake reports:seed	40
rake db:seed	54
rake gettext:sync_to_po_and_db	269
./bin/cron_installer.rb	14

Asset Precompilation in GitHub Actions

Here's the flow as it exists in this code:

1. Code is pushed
2. GitHub Action "asset_compilation" starts
   a. Pull in secrets file from Secrets Manager (parameter store char limit too low)
   b. Start bin/compile_assets.rb, which iterates through each install and...
   1. Pulls down the theme files from s3://openpath-ecs-assets/#{target_group_name}
   2. Pulls down the client .env secrets into .env.secrets
   3. Generates a checksum based on app/assets (including the downloaded theme files) and .env.secrets
   4. Checks if we have s3://openpath-ecs-assets/#{target_group_name}/#{checksum}
   5. If we don't, assets have changed. Run assets:precompile and push the results (public/assets) to s3://openpath-ecs-assets/#{target_group_name}/#{checksum}.
3. Deployment is triggered.
   a. Checksum is generated again, and if matching compiled assets are not available, the deploy will warn you and wait for 60 seconds to try again
4. Eventually, after the deploy proceeds, an application container is spun up. In the entrypoint...
   a. Pull down theme files and generate checksum again (this doesn't seem to be very computationally expensive)
   b. Pull down compiled assets the same way we do now, except with the checksum instead of the githash

The idea is that we are directly mapping the contents of the uncompiled assets with the compiled assets in S3, instead of mapping via the githash. And so it will always be easy to check if we have a valid compiled version for the current asset code.

[andyzito]

@eanders Per my most recent commit message, File.sanitize isn't builtin and I think w/the other sanitization stuff I put in it's probably fine? But lmk if you're still not feeling good about it, I assume we can add File.sanitize without too much trouble.

[eanders]

@eanders Per my most recent commit message, File.sanitize isn't builtin and I think w/the other sanitization stuff I put in it's probably fine? But lmk if you're still not feeling good about it, I assume we can add File.sanitize without too much trouble.

I think what you have is fine. Thanks!