Baremetal MEGA PR

[cannikin]

Contains all the changes from the following 7 PRs:

• Updates Baremetal deploy to use git clone and branch names #5282
• Adds support for multiple environments to Baremetal deploy #5334
• Adds maintenance page control to Baremetal #5349
• Remove more than X old releases in Baremetal deploy #5359
• Adds ability to rollback to a previous Baremetal deploy #5361
• Allow override of default yarn and pm2 exec calls during Baremetal deploy #5427
• Adds lifecycle events to Baremetal deploy #5449

Closes #5082
Closes #5083
Closes #5084
Closes #5300
Closes #5347
Closes #5298
Closes #5299
Closes #5422
Closes #5297

Release Notes

This release features some major updates to the Baremetal deploy option! Let's break them down along with any changes you'll need to make to your server and deploy.toml file.

New Code Update Strategy

We've got a good news/bad news situation. The good news: you can now deploy a branch other than main and code updates are much more reliable using git clone! The bad news is that you're going to have to do some more config and server setup to get this latest version working. But once you do, everything will be awesome!

PM2

Baremetal now requires that PM2 be installed globally, rather than a development dependency of your app. This has to do with the new directory structure we're going to create and where pm2 is executed from (short version: it's run outside of your actual codebase, so yarn pm2 won't be available).

You'll want to remove your existing pm2 services. This will cause downtime, so be mindful if running in a production environment. Run the following commands on the server:

yarn pm2 delete all

Now install PM2 globally: https://pm2.io/docs/runtime/guide/installation/

PM2 will still be running in memory as the previous yarn pm2 version, so you can run this command to update the process:

pm2 update

If you previously setup PM2 to run at startup, you'll need to update to the new global install instead:

yarn pm2 unstartup
pm2 startup

Be sure to remove pm2 in your app's package.json file.

Directory Structure

Next we'll need to update the directory structure inside your deploy path.

Current directory structure:

└── var
    └── www
        └── myapp
            ├── api
            ├── web
            ├── ...

New directory structure:

└── var
    └── www
        └── myapp
            ├── .env <────────────┐
            ├── current ────────┐ │
            └── 20220420120000 <┘ │
                ├── .env ─────────┘
                ├── api
                ├── web
                ├── ...

The current symlink is updated at the end of a deploy to point to the latest release (assuming a successful clone and build). This means all build packages in web/dist and api/dist remain self-contained in their parent timestamp directory. This makes a rollback trivial: just point the current symlink to the previous directory and restart your web/api processes! yarn rw deploy baremetal --rollback is coming soon! A shared .env file exists in the root of your app path and each copy of the codebase will contain a symlink back out to this file.

The easiest way to update to this structure is to simply remove everything inside of your currently deploy directory (like /var/www/myapp) and let yarn rw deploy baremetal --first-run set everything up for you. You'll want to keep your .env file, however:

cd /var/www/myapp
mv .env ..
rm -rf *
mv ../.env .

Config Updates

You'll need to add a new option to your ecosystem.config.js file for any processes that run within the context of your app:

module.exports = {
  apps: [
    {
      name: 'serve',
+     cwd: 'current',
      script: 'node_modules/.bin/rw',
      args: 'serve',
      instances: 'max',
      exec_mode: 'cluster',
      wait_ready: true,
      listen_timeout: 10000,
    }
}

You'll also need to add the repo config setting and optionally branch to your deploy.toml file (also note the new environment name prefixing servers, more on that in a moment):

[[production.servers]]
host = "myserver.com"
username = "user"
agentForward = true
sides = ["api", "web"]
path = "/var/www/myapp"
processNames = ["api"]
+ repo = "git@github.com:myorg/myrepo.git"
+ branch = "main"

The branch will default to main if not set.

Make sure everything is saved, committed, and pushed up to your repo because we're ready for the first deploy.

First Deploy

Run the deploy command for the first time to setup the directory structure, check out your code, and build and start monitoring your processes:

yarn rw deploy baremetal --first-run

And that's it! If everything started up correctly then you're good to go for future deploys. If not check out https://redwoodjs.com/docs/deployment/baremetal#troubleshooting

If you want to perform a one-time deploy of a branch other than the one listed in deploy.toml you can do so via a flag at deploy time:

yarn rw deploy baremetal --branch=staging

Multiple Environment Support

Baremetal deploy now supports deploying to multiple environments! If you've got production, and staging, and whatever else, this update is for you. If you're setting up a new deploy from scratch with yarn rw setup deploy baremetal you're good to go. If you have an existing deploy.toml file you may want to make an update:

Instead of just [[servers]] prefixing your server config, make it [[servers.environment]] where environment is the actual name of your environment. For example:

[[production.servers]]
host: 'server.com'
# remaining config...

[[staging.servers]]
host: 'staging.server.com'
# remaining config...

You can still have multiple servers in an environment, just repeat the [[servers.production]] heading multiple times, one for each block of server config:

[[production.servers]]
host: 'prod1.server.com'
# remaining config...

[[production.servers]]
host: 'prod2.server.com'
# remaining config...

This new syntax is optional: keeping your default [[servers]] name will act as production (and leaving off the stage name in yarn rw deploy baremetal will default to use these server settings).

Maintenance Page

You can now enable/disable a maintenance page with the Baremetal deploy! Simply create web/src/maintenance.html containing your maintenance message. (For now this page must just be plain HTML, but if the need is there it should be possible to add the page to the webpack/babel pipeline and make it a full React app. If this is something you'd like to work on, get in touch!)

When you're ready to enable your maintenance page on the site:

yarn rw deploy baremetal --maintenance up

And to take it back down:

yarn rw deploy baremetal --maintenance down

This command respects the environment argument so you can put it up on just your staging environment, for example:

yarn rw deploy baremetal staging --maintenance up

That's it! This feature works by turning web/dist/index.html into a symlink pointing to web/src/maintenance.html. As long as your server is configured to check for the existence of 200.html for the web side and serve that for any URL request that isn't the API, users should see the maintenance page the time time your site loads. Note that if a user already has your site loaded, they will most likely be able to keep clicking through your pages since everything is already cached in the browser. We're thinking about adding a "ping" feature that, on each page request, would check with the server to see if the maintenance page should be displayed, and if so interrupt their browsing session to show it. If this is something you'd like to work on, let us know by opening an issue or PR!

Old Deploy Cleanup

Baremetal will now cleanup old deploy codebase directories at the end of a deployment! It defaults to keep the last 5 deployments, but this can be configured in deploy.toml with the keepReleases option:

[[servers.production]]
host = "server.com"
username = "ubuntu"
agentForward = true
sides = ["api", "web"]
path = "/var/www/myapp"
processNames = ["api"]
repo = "git@github.com:myorg/myrepo.git"
branch = "main"
+ keepReleases = 5

Why keep around old deploys? In case something goes horribly wrong and you need to rollback!

This config setting is optional and will default to keeping the last 5 deploys if not set.

Rollback

If you deploy and find something has gone horribly wrong, you can now rollback your deploy to the previous release!

yarn rw deploy baremetal --rollback

You can even rollback multiple deploys, up to the total number you still have denoted with the keepReleases option:

yarn rw deploy baremetal --rollback 3

Note that this will not rollback your database—if you had a release that changed the database, that updated database will still be in effect, but with the previous version of the web and api sides. Trying to undo database migrations is a very difficult proposition and isn't even possible in many cases.

Make sure to thoroughly test releases that change the database before doing it for real!

There's no Rollforward: you probably rolled back because the newer deploy is faulty! Performing a new deploy would be the equivalent action to rolling the codebase forward to a newer release.

Override Default yarn and pm2 commands

You can now override the default yarn and pm2 exec commands with your own custom command. Why would you want to do this? Tools like doppler provide their own CLI command to inject secrets into your shell, and you then chain on the command you would normally run, like so:

doppler run -- yarn rw prisma migrate deploy

To set this, add the following to your deploy.toml file:

[[servers.production]]
host = "server.com"
username = "user"
agentForward = true
sides = ["api","web"]
path = "/var/www/app"
processNames = ["serve"]
repo = "git@github.com:myorg/myapp.git"
branch = "main"
keepReleases = 5
+ packageManagerCommand = "doppler run -- yarn"
+ monitorCommand = "doppler run -- pm2"

These two settings are optional and will default to the existing yarn and pm2 commands if not set.

Lifecycle Events

When in the course of a deploy, if it becomes necessary to run additional commands not provided for in the stock list of Baremetal steps, you can turn to lifecycle events. These events allow you to run custom commands before or after the existing commands. Here are the current steps in a deploy:

1. update - the latest code is cloned from the server
2. symlinkEnv - a symlink is created in the newly cloned codebase pointing to the .env file in the main app directory
3. install - latest dependencies are installed
4. migrate - database and db migrations are run
5. build - the api and/or web sides are built
6. symlinkCurrent - a symlink is created from the main app directory to the latest deploy directory
7. restart - PM2 restarts any services listed
8. cleanup - old deploy directories are removed

There are three ways you can define your lifecycle events: globally (runs for all environments and all servers), environment-specific (runs for all servers in a given environment) or server-specific (runs only for a single server).

Global Lifecycle Events

Add a top level [before] or [after] key to your deploy.toml, listing which step you want to hook into and what command to run:

[before]
install = 'touch install.lock'

[after]
install = 'rm install.lock'

[[production.servers]]
host = 'server.com'
# ...

You can hook into multiple steps, and execute multiple custom commands:

[before]
install = ['touch install.lock', 'rm logs/deploy.log']
cleanup = 'echo "Removing old deploys" > "deploy.log"'

[[production.servers]]
host = 'server.com'
# ...

Environment Lifecycle Events

[production.before]
install = 'touch install.lock'

[staging.after]
build = 'touch debug-enable'

[[production.servers]]
host = 'server.com'
# ...

[[staging.servers]]
host = 'stage.server.com'
# ...

Server Lifecycle Events

[[production.servers]]
host = 'server.com'
before.install = 'touch install.lock'
# ...

Important Considerations

All commands are run inside the newly deployed code directory (like /var/www/myapp/20220520120000) except update and cleanup! These are run in the path defined in deploy.toml since they are independent of any single release.

The build and restart steps may run multiple times based on how many sides you're building or services you're restarting. Any defined before/after lifecycle events will run as many times as the original step is run, so plan accordingly: make your command idempotent so that it can be run multiple times without side effects.

[netlify]

✅ Deploy Preview for redwoodjs-docs ready!

Name	Link
🔨 Latest commit	fc08999
🔍 Latest deploy log	https://app.netlify.com/sites/redwoodjs-docs/deploys/62853a6b8019670008e09dcc
😎 Deploy Preview	https://deploy-preview-5500--redwoodjs-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[BurnedChris]

Cant wait for this!!!

[cannikin]

This doesn't contain doc updates, but I'll be adding them in another PR (mostly built off the release notes)

Apparently I did write the doc updates! Damn I'm good.