feat: establish secrets management with GitHub environment-scoped secrets

[GitAddRemote]

Tech Story

As a platform engineer, I want all production secrets stored in GitHub environment-scoped secrets and written to the VPS only at deploy time, so that credentials are never committed to git, are isolated between staging and production, and can be rotated in under 10 minutes with a documented procedure.

ELI5 Context

Why not just put secrets in a .env file in the repo?
A .env file committed to git is visible to everyone with repo access — and in git history forever, even after deletion. More importantly, GitHub Actions logs can leak secrets if you're not careful. The correct pattern: secrets live in GitHub's encrypted secret store, get written to the VPS as a .env file at deploy time (never logged), and the .env file is never committed to git.

What is a GitHub Environment?
A GitHub Environment is a named deployment target (e.g. "staging", "production") with its own isolated secrets and optional protection rules. Secrets set on the production environment are only available to workflow jobs that declare environment: production. This means a staging deploy cannot accidentally use production database credentials.

What is secret rotation?
Changing a secret's value — for example, rotating the JWT secret or database password. Without a documented procedure, rotation is risky: you might update the secret in GitHub but forget to update it on the VPS, causing a deployment outage. With a documented procedure, rotation takes 10 minutes and never causes downtime.

Why chmod 600 on the .env file?
File permissions 600 means only the owner (the deploy user) can read or write it. Other users on the VPS — even if they SSH in separately — cannot read the file. This prevents secrets from leaking if the VPS is ever shared or another service is compromised.

Technical Elaboration

Step 1: Create GitHub Environments (manual, one-time)

In the GitHub repo → Settings → Environments:

1. Create environment named staging:

   • No protection rules
   • Add all secrets from the table below with staging-appropriate values

2. Create environment named production:

   • Enable Required reviewers — add yourself
   • Enable Prevent self-review: OFF (you're the only reviewer)
   • Add all secrets from the table below with production-appropriate values

Required Secrets Per Environment

Secret	staging	production	How to generate
DATABASE_HOST	✅	✅	localhost (DB runs on same VPS)
DATABASE_PORT	✅	✅	5433 (prod) / 5434 (staging)
DATABASE_USER	✅	✅	Choose a username
DATABASE_PASSWORD	✅	✅	openssl rand -base64 32
DATABASE_NAME	✅	✅	station / station_staging
JWT_SECRET	✅	✅	openssl rand -base64 48 (min 32 chars)
REDIS_PASSWORD	✅	✅	openssl rand -base64 24
VPS_HOST	✅	✅	VPS public IP address
VPS_USER	✅	✅	deploy
VPS_SSH_KEY	✅	✅	Private key matching the deploy user's authorized_keys
ALLOWED_ORIGIN	✅	✅	https://staging.station.drdnt.org / https://station.drdnt.org
FRONTEND_URL	✅	✅	Same as ALLOWED_ORIGIN
B2_ACCOUNT_ID	✅	✅	From Backblaze B2 console
B2_APPLICATION_KEY	✅	✅	From Backblaze B2 console
B2_BUCKET	✅	✅	station-backups
SENTRY_DSN	✅	✅	From Sentry project settings
LOGTAIL_SOURCE_TOKEN	✅	✅	From Logtail source settings
BACKUP_HEALTHCHECK_URL	❌	✅	From healthchecks.io check detail page

Step 2: Deploy-time .env injection

In .github/workflows/release.yml, add a "Write .env" step at the start of both deploy-staging and deploy-production jobs. Use a quoted heredoc to prevent shell expansion of secret values:

- name: Write .env to VPS
  run: |
    ssh ${{ secrets.VPS_USER }}@${{ secrets.VPS_HOST }} bash -s << 'SSHEOF'
      cat > /opt/station/.env.production << 'ENVEOF'
    DATABASE_HOST=${{ secrets.DATABASE_HOST }}
    DATABASE_PORT=${{ secrets.DATABASE_PORT }}
    DATABASE_USER=${{ secrets.DATABASE_USER }}
    DATABASE_PASSWORD=${{ secrets.DATABASE_PASSWORD }}
    DATABASE_NAME=${{ secrets.DATABASE_NAME }}
    JWT_SECRET=${{ secrets.JWT_SECRET }}
    REDIS_PASSWORD=${{ secrets.REDIS_PASSWORD }}
    ALLOWED_ORIGIN=${{ secrets.ALLOWED_ORIGIN }}
    FRONTEND_URL=${{ secrets.FRONTEND_URL }}
    SENTRY_DSN=${{ secrets.SENTRY_DSN }}
    SENTRY_RELEASE=${{ github.sha }}
    NODE_ENV=production
    PORT=3001
    USE_REDIS_CACHE=true
    ENVEOF
      chmod 600 /opt/station/.env.production
    SSHEOF

Security note: GitHub Actions automatically masks secret values in logs — they appear as ***. The heredoc approach ensures secrets are never echoed in the SSH session output.

Step 3: Verify .gitignore and .dockerignore

Confirm these entries exist:

.gitignore (repo root):

.env
.env.*
!.env.example
!.env.production.example

backend/.dockerignore and frontend/.dockerignore:

.env
.env.*

New file: infra/docs/secrets.md

Sections:

1. Secret inventory — the table above, with description and how-to-generate for each

2. Rotation procedure (generic)

1. Generate new secret value (openssl rand command or service console)
2. Update the secret in GitHub → Settings → Environments → [staging|production]
3. Push a release branch (or manually re-run the deploy workflow)
4. The deploy writes the new .env.production and restarts containers
5. Verify: hit GET /health — if 200, rotation succeeded

3. JWT_SECRET rotation (special case)
JWT_SECRET rotation invalidates all existing tokens — every logged-in user will be logged out.

1. Schedule a maintenance window (or accept the forced logout)
2. Generate new secret: openssl rand -base64 48
3. Update GitHub secret
4. Deploy (this writes the new secret and restarts the backend)
5. All existing access tokens are now invalid — users must log in again
6. Refresh tokens stored in Redis are also invalidated (they're signed with the old secret)

4. Database password rotation

1. Generate new password: openssl rand -base64 32
2. Update the PostgreSQL user password directly: docker exec station-postgres-1 psql -U postgres -c "ALTER USER stationuser PASSWORD 'newpassword';"
3. Update DATABASE_PASSWORD GitHub secret
4. Deploy (writes new .env, restarts backend — backend reconnects with new password)
5. Verify: GET /health returns 200 with database check passing

5. SSH key rotation

1. Generate new key pair: ssh-keygen -t ed25519 -f deploy_key_new
2. Add new public key to VPS: ssh-copy-id -i deploy_key_new.pub deploy@VPS_IP
3. Update VPS_SSH_KEY GitHub secret with new private key
4. Test deploy workflow succeeds
5. Remove old public key from VPS: edit ~/.ssh/authorized_keys on VPS

Definition of Done

• staging GitHub Environment created with all required secrets
• production GitHub Environment created with required reviewer and all secrets
• .github/workflows/release.yml writes .env.production at the start of each deploy job
• .env.production on VPS is chmod 600 — verified by SSHing in and running ls -la /opt/station/
• .gitignore excludes all .env.* files except .example templates
• .dockerignore on backend and frontend excludes .env.* files
• infra/docs/secrets.md written with secret inventory, generic rotation procedure, and JWT/DB/SSH special-case procedures
• End-to-end test: delete .env.production on VPS, run deploy workflow, confirm .env.production is recreated correctly

Dependencies

• Depends on: Tech Story: VPS baseline provisioning (Nginx, Certbot, deploy user) #107 (deploy user and VPS directory structure must exist)
• Depends on: Tech Story: GitHub Actions CI/CD — release-tag SSH deploy with graceful restart #90 (release workflow is where the secret injection step lives)
• Blocks: feat: integrate UptimeRobot and Sentry for production observability #127 (Sentry DSN), ops: structured application log aggregation #131 (Logtail token), ops: backup failure alerting — notify when nightly backup cron fails #133 (healthcheck URL) — all stored here