Proxy is a nginx-based reverse proxy with TLS that runs on AWS. It was written to be the backend for a Recurse Center custom subdomain service, which lets RC alumni register a recurse.com subdomain for their webapp or website.
Proxy is a front-end for an unlimited number of webapps all hosted at *.example.com. For each request, Proxy serves a wildcard TLS certificate for that domain.
Proxy can also optionally be the front-end for multiple apex redirects. This will redirect all HTTP and HTTPS requests from example.com to www.example.com, example.net to www.example.net, etc. This is useful if you use Proxy on the same domain as a Heroku app living at www.example.com because Route 53, which Proxy requires, does not support ALIAS records to non-AWS infrastructure.
- Nightly unattended security updates with zero downtime
- Easy deploys with near-zero downtime
- Dynamic updating of host list from an external HTTPS endpoint
- Secure secret storage with easy secret updating
- Logs to a syslog server (e.g. Papertrail)
- Redirect apex domains to www domains
aws: elb_name: proxy-elb region: us-east-1 ami: ami-0a313d6098716f372 # Ubuntu 18.04.2 for us-east-1 instance_type: m3.medium instance_count: 2 key_name: Zach # you must upload your public key and give it a name on the EC2 dashboard security_group: proxy # Used for instances. Should have ports 22 and 443 open. env: PROXY_ENV: production PROXY_DOMAIN: recurse.com PROXY_DOMAINS_ENDPOINT: https://www.example.com/domain.json # Optional # Apex redirects (example.com -> www.example.com) # PROXY_APEX_DOMAINS: example.com,example.net,example.org # Remote logging over TLS. All three variables must be set. # PROXY_SYSLOG_DRAIN: logs.papertrailapp.com:12345 # PROXY_SYSLOG_ROOT_CERTS: https://papertrailapp.com/tools/papertrail-bundle.pem # PROXY_SYSLOG_PERMITTED_PEER: "*.papertrailapp.com"
How Proxy works
The infrastructure that Proxy runs on consists of a Classic Load Balancer and a configurable number of EC2 instances (we use 2).
Proxy itself is a few pieces of software:
- A command line tool (bin/proxy) that knows how to boot and configure new EC2 instances, register them with the load balancer, and terminate old ones.
- Nginx listening on port port 443. Requests with X-Forwarded-Proto set to http are redirected to HTTPS, and HTTPS requests are reverse-proxied to the configured hosts.
- A backend (backend/bin/proxy-backend) that is responsible for loading subdomain -> redirect URL mappings from $PROXY_DOMAINS_ENDPOINT, and reloading nginx. By default, this happens every 15 seconds.
- Provisioning scripts (backend/bin/setup, backend/bin/proxy-install) that are responsible for configuring new EC2 instances.
The ELB sits in front of the two instances, which are deployed in separate availability zones for redundency. The ELB loads /healthcheck on each instance to make sure that the instances are running.
The ELB has two listeners: HTTP and HTTPS. Both listeners forward to HTTPS on the instances, using Backend Authentication, which consists of a self-signed certificate and associated private key, generated during the deploy process. The public key gets installed on the ELB during deploy, and the ELB only passes traffic to instances that present a certificate with the same public key.
The instances use Upstart to make sure the proxy-backend daemon is always running. Proxy-backend logs to syslog. You can set the optional PROXY_SYSLOG_DRAIN config option to the URL for a remote syslog server, which can collect the logs from all running instances.
Loading subdomain mappings
In order to use Proxy, you need a publicly accessible HTTP endpoint that returns a set of mappings from subdomain to URL. The endpoint must return JSON data in the following format:
[ ["subdomain1", "https://www.example.com/foo"], ["subdomain2", "https://www.example.net"], ["subdomain3", "http://www.example.org"] ]
The deploy process
The code for Proxy's deploy process is located in lib/proxy/deploy.rb. This is a summary of the process:
- Generate self-signed certificate (newly generated each deploy) and dhparams (only generated if necessary)
- Clean up any instances from a failed half-finished deploy
- Add public key from self-signed certificate to the ELB's list of trusted public keys
- Boots new instances
- Uploads a tar file file consisting of everything in
git ls-files, as well as the production config and certificate files to each instance, extracts the tar on the server, and runs
- Registers the new instances with the ELB and waits until they are in service.
- Terminates the old instances
- Removes the old trusted public key from the ELB
What gets deployed
git ls-files to decide what files get uploaded to the instances, it uses the contents of your working directory, not the contents of the git repository. This means that any local modifications you have will get deployed. This is useful for testing changes to Proxy, but may bite you if you're not careful.
Each time you run
bin/proxy deploy, a random UUID is generated and written to .deploy. This file is removed once the deploy is complete. Each instance that gets deployed is tagged with this UUID.
bin/proxy sees a .deploy file when it is run, it assumes there was a broken deploy and cleans up by terminating all instances tagged with that UUID.
Instances are also tagged with the name "proxy-web" so you can easily see which instances are a part of Proxy.
How to deploy Proxy
If this is your first time deploying Proxy, you'll have to create a Classic Load Balancer manually. It should be configured to forward both HTTP and HTTPS connections to HTTPS on its instances.
Make sure your AWS credentials are in ~/.aws/credentials. You can configure this with the aws cli:
Next, create config.production.yml file in Proxy's root directory (see above for an example).
To list instances, run
bin/proxy list. This is useful for SSHing into the instances for debugging purposes (see below).
This command lists all instances tagged with proxy-web that are either pending, running, or shutting down. There's no guarantee that these instances are registered with the load balancer.
If you were the person who deployed Proxy, you can SSH into the instances and poke around to see what's wrong. Here are places to look:
systemctl status proxy
systemctl status nginx
- /var/log/syslog (proxy logs to syslog)
If you didn't deploy proxy, you won't be able to SSH in. The best you can do is deploy proxy yourself so that the instances have your SSH keys and then wait for the problem to happen again so you can debug via SSH.
If you're debugging the RC proxy instance, you can access the syslog output by going to Papertrail and logging in using the "Papertrail (Proxy)" credentials in the RC 1Password vault.
While you can't reboot proxy, the problem might go away if you deploy it again. Make sure your SSH key is set up correctly in the EC2 section of the AWS Console, and referenced in your config.production.yml.
After that, run
- TLS session resumption
- WebSocket support (see http://nginx.com/blog/websocket-nginx/)
- Config stored in the cloud to support multiple people deploying
- In event of broken conf, upload to S3 and include link in error msg
Copyright Recurse Center 2019
AGPLv3 or later