Run auth service on a separate machine

[bickelj]

We would benefit from auth running on a separate machine from PDC service. Deployment of PDC backend would be more cleanly separated, e.g. PDC deployment does not impact auth, sessions, etc.

[bickelj]

#86 would be resolved by this.

[bickelj]

My assumption when creating this ticket was the goal of a single, unified Keycloak instance across several projects. That assumption was incorrect because PDC infrastructure needs to be cleanly separate from other infrastructure.

In light of this, I think we could work on #86 first rather than adding another machine to maintain while the first machine is not maxed out in any way. @jasonaowen How does that sound?

[jasonaowen]

@bickelj I'm not sure I follow the distinction you're making between this and #86! I agreed with your earlier comment that working on this would resolve #86. Can you say more?

[bickelj]

@jasonaowen Sure, I don't think this is immediately needed for #86, but I can see that now is a good time to make a dedicated auth machine if that's desired. So doing.

[bickelj]

The plan is this:

• Create a test machine with associated managed DB.
• Restore a backup of the existing test DB to the new test DB.
• Install new Keycloak as if it is the actual domain name (keeping the test domain name pointed to the existing/old instance) but add a local hosts entry to point to the new instance.
• Use a copy of existing certificates until DNS cutover.
• Compile and use updated extension jars (if needed).
• Point the domain name to the new instance. Either accept some downtime until propagation and certbot creation of certs or copy the existing certificates and keys from the old system to the new.
• Generate new certificates with certbot after DNS has been updated.
• Rotate Keycloak's signing key.
• Repeat the above for production (same order but not necessarily waiting for all steps before starting this).
• Remove the auth service from compose.yml.
• Remove the environment variables related to auth service from .env files on service machines.
• Remove the auth service reverse proxy config from service machines.
• Remove the keycloak database from old machines.
• Remove the auth user from old machines.

[bickelj]

I staged most of the changes in the new production machine while the DNS change for test is pending. I didn't yet take a backup of the production machine (or restore it) because we want that step to be as close to the DNS change as practical. But the prerequisites, the database user and (empty) database, Keycloak software and config, nginx config, systemd unit, env vars are at least there and ready for the dump/restore and DNS change.

[bickelj]

Let's Encrypt certificates are now are on auth test machine using certbot.

[bickelj]

The new production auth machine is running now, pending DNS change.

[bickelj]

DNS change is complete, and the prod machine also is using Let's Encrypt certificates now.

[bickelj]

All but the unchecked cleanup steps are complete here. I have not tried a keycloak key rotation so I do not know how long that would take.

[bickelj]

The old machines should have very little or none to do with the old keycloak instance running there now. The database, database user, system user, and leftover files from the migration should be gone. The related env vars are commented out.