Change deploy model

[jasonaowen]

Currently, this repo is deployed via the sister repo PhilanthropyDataCommons/deploy to a DigitalOcean droplet that is running Docker. This has a lot of costs:

• We have semi-automated deployments but not continuous deployment, as there are some manual steps involved. While I believe I have the keys to that, I haven't done it, so while @bickelj has been working on another project, we haven't really deployed. (Thank you for noticing that and deploying last week, @bickelj!)
• There are some personal access tokens involved, which are a (known) bit of a ticking time bomb in that they will someday expire and what automation we currently have will break.
• The current approach prevents us from using the @dependabot merge command.
• Configuring environment variables is nontrivial.
• The underlying instance requires ongoing sysadmin support, in the form of system updates and monitoring and so on.
• Deploying changes from this repo invalidates Keycloak sessions (Graceful user session handling on deployment deploy#86).
• We haven't yet configured database backups, which is our responsibility in this model.

Instead, I would like for us to use a Platform-as-a-Service (PaaS) to deploy this app, and specifically DigitalOcean App Platform.

• This would give us continuous deployments out of the box: clicking the big green button in a GitHub pull request would cause a new deployment.
• We could build per-pull-request deploy previews, making it easier to test and review pull requests.
• It shouldn't expire, or be tied to a particular user's account.
• Dependabot should work as expected.
• Environment variables should be configurable via both a web UI in DigitalOcean, and I think via its doctl command-line interface.
• OTS sysadmins will need to do less maintenance.
• Out-of-the-box database backups, including point-in-time recovery, via a managed PostgreSQL instance.

After discussing this with @jvasile, @reefdog, and @slifty today, it sounds like OTS does not expect @bickelj to have bandwidth for this project until next year.

Note that disentangling Keycloak is a separate but related issue (PhilanthropyDataCommons/deploy#94); this approaches it from the other direction.

There are a few steps involved:

• Migrate the existing production database from a Docker container to a new DO-hosted database
• Set up a new DO App for production
• Update DNS for api.pdco to point to the new instance
• Build out deploy previews (stretch goal)
• Decommission no-longer-needed portions of the deploy repo

[jasonaowen]

Last week I was able to set up a DigitalOcean App of this repo. Among other things, that involved connecting the DigitalOcean GitHub integration to this repository, making a copy of our existing production database (running in Docker on a droplet) into a hosted PostgreSQL database, and figuring out the necessary environment variables.

My next steps are to tear that instance down and rebuild it to make sure my documentation is adequate, request DNS changes to point away from the old version, and tear down the old instance.

[bickelj]

• We have semi-automated deployments but not continuous deployment, as there are some manual steps involved. While I believe I have the keys to that, I haven't done it, so while @bickelj has been working on another project, we haven't really deployed. (Thank you for noticing that and deploying last week, @bickelj!)

The problem with deploying to production was setting up a way to automatically check that the deployment to test worked successfully. That work has been done as part of deploy#106 and deployments to production are now enabled. As of this comment, https://github.com/PhilanthropyDataCommons/deploy/deployments/production shows 36 automated production deployments.

• There are some personal access tokens involved, which are a (known) bit of a ticking time bomb in that they will someday expire and what automation we currently have will break.

While they are labeled "Personal Access Token" by GitHub, they are actually associated with an organization, and the "Personal" part is only for who created them. The process to replace one of these tokens is documented here. See especially this note:

As of May, 2023, the Personal Access Token can be linked with an organization and particular repositories rather than only an individual having access to the organizations. See issue #343. The active organization tokens can be seen in the organization settings tokens.

• The current approach prevents us from using the @dependabot merge command.

This was resolved by adding a similar REPO_ACCESS_TOKEN to dependabot settings.

• Configuring environment variables is nontrivial.

You add them to .env.example in the code, compose.yml to pass them to the appropriate containers, and .env on the host. If there is a less trivial way I would argue it comes at the cost of being less explicit in which software uses what environment variable.

• The underlying instance requires ongoing sysadmin support, in the form of system updates and monitoring and so on.

The underlying system has automated updates.

• Deploying changes from this repo invalidates Keycloak sessions

As of resolution of deploy#94 this is no longer an issue. Separation creates its own set of issues for anybody trying to start up a new instance of the PDC, though, because now they have to run Keycloak on a separate machine outside of the docker compose script. Another solution could have been found if needed in deploy#86.

• We haven't yet configured database backups, which is our responsibility in this model.

We have snapshots of the instances which contain the database data. We also are expected to destroy the current data.

In summary, all the issues mentioned here are either resolved or moot,

[bickelj]

I agree that deploy previews would be a nice enhancement and that higher-level database backups would be nice in future.