Troubleshooting

ROR services are very stable and rarely experience issues and downtime.

Common issues

Issue	Common causes	Action(s)
ROR API is slow/unstable	Elastic search at 80% CPU usage for long periods	Check Datadog logs to identify IP(s) a high volume of requests and/or troublesome requests that use lots of CPU or cause errors (ex: complicated affilation matching requests, query strings with unescaped reserved characters or unclosed parens).
ROR API is down	Elastic search at 100% CPU usage	Typically nothing; app will recover on its own when traffic subsides. If it does not recover, force a restart with deployment to ror-api or (in case of emergency) AWS CLI request like aws ecs update-service --force-new-deployment --cluster CLUSTER_NAME --service SERVICE_NAME . If this happens repeatedly due to traffic from specific IPs, IPs can be blocked by adding them to the blacklist_ips_prod variable in Terraform Cloud and triggering a manual Terraform run.
Ror-site won’t deploy	Dependency issues	Review Actions log; check dependencies pulled in during actions run .Trigger deployment again if needed
No app logs from ECS containers (not really an issue itself, but makes it hard to troubleshoot)	Nginx logs are not being forwarded (bug in Phusion Passenger https://github.com/phusion/passenger-docker/issues/224	SSH to container (see below) and restart nginx-log-forwarder
API user reports failure or performance issue with a specific request	Request contains unescaped reserved characters, unencoded &, ? or / characters, API performance is slow due to other requests occurring the same time	Get user's request, IP address, and time of request. Find request(s) in ALB logs using Datadog log explorer; review raw request that was received and check for unencoded &, ? or / characters, unclosed ( or ). Check response code; if it's 500, check application logs in Datadog for error messages. If neither of these cases apply, check for high volume of requests from other users around the same time or requests from other users that are generating errors/using excessive CPU.

Reverting/overwriting Elastic search index

In case of issues with the data release process, it's possible to delete and recreated the Elasticsearch index from a data dump.

1. SSH to running ECS container - see Bastion host entry in 1Password

2. Run the setup up command and pass the filename of the data dump and schema version you want to index (no file extension). File must exist in ror-data.

    python manage.py setup v1.0-2022-03-17-ror-data -s 2