Troubleshooting

ROR services are very stable and rarely experience issues and downtime.

Common issues

API

Issue	Common causes	Action(s)
ROR API is slow/unstable	Elastic search at 80% CPU usage for long periods	Check Datadog logs to identify IP(s) a high volume of requests and/or troublesome requests that use lots of CPU or cause errors (ex: complicated affilation matching requests, query strings with unescaped reserved characters or unclosed parens).
ROR API is down	Elastic search at 100% CPU usage	Typically nothing; app will recover on its own when traffic subsides. If it does not recover, force a restart with deployment to ror-api or (in case of emergency) AWS CLI request like aws ecs update-service --force-new-deployment --cluster CLUSTER_NAME --service SERVICE_NAME . If this happens repeatedly due to traffic from specific IPs, IPs can be blocked by adding them to the blacklist_ips_prod variable in Terraform Cloud and triggering a manual Terraform run.
API user reports failure or performance issue with a specific request	Request contains unescaped reserved characters, unencoded &, ? or / characters, API performance is slow due to other requests occurring the same time	Get user's request, IP address, and time of request. Find request(s) in ALB logs using Datadog log explorer; review raw request that was received and check for unencoded &, ? or / characters, unclosed ( or ). Check response code; if it's 500, check application logs in Datadog for error messages. If neither of these cases apply, check for high volume of requests from other users around the same time or requests from other users that are generating errors/using excessive CPU.

Curation

For all issues below, Github actions runs will notify #deployment-updates with errors. Check Github Actions output in https://github.com/ror-community/ror-records/actions for error messages first.

Issue	Common causes	Action(s)
Data indexing in staging or prod fails due to connection closed	Indexing request timed out at load balancer	Run indexing again, using Manual deploy to Prod or Manual deploy to Staging Github action. Load balancer timeout is set fairly high, ES request is sent appropriately as a bulk request, and number of records sent for indexing is not huge, but for whatever reason this just happens sometimes.
A Github action fails with an access denied error	ror-bot Github personal access token has expired or Github API requests are being rate limited	ror-bot personal access token expires every 3 months (see ROR Team Calendar or 1Password for next expiration date). If expired, log into Github as ror-bot (creds in 1Password), generate new token and update in Github organization-level secrets. Also needs to be updated in Terraform ror-api variables; run Terraform manually to apply update.
Validation fails	Geonames API is returning different responses to the same request, record file contains weird characters, or relationships file contains something incorrect/weird	Geonames has a known issue with returning different responses for the same request (seems like caching at the CDN). Curation team should make a choice about which values to include, updated record(s) if needed and ignore validation error. For other validation errors, review record files and relationships file. Add print statements to validation-suite code if needed.

Internal dev/deployment

Issue	Common causes	Action(s)
Ror-site won’t deploy	Dependency issues	Review Actions log; check dependencies pulled in during actions run .Trigger deployment again if needed
No app logs from ECS containers (not really an issue itself, but makes it hard to troubleshoot)	Nginx logs are not being forwarded (bug in Phusion Passenger https://github.com/phusion/passenger-docker/issues/224	SSH to container (see below) and restart nginx-log-forwarder

Blocking problem users

1. Obtain problem user IP(s) from Datadog (see Identifying problem users)
2. Log into Terraform.io https://app.terraform.io/app (see 1Password)
3. Go to ror-vpc workspace
4. Click Variables on the left-hand menu
5. Edit blacklist_custom_msg_ips_prod variable value to add problem IPs. Enclose each IP in quotes and add /32 to the end, ex "XX.XXX.XX.XXX/32". Separate IPs with commas. Make sure to save the variable!
6. Click the New Run button in the upper right
7. In the Run name field enter Update blacklist IPs
8. Click Start. Terraform will complete a "plan" phase, then an "apply" phase. If you see green check marks for both phases, you are good to go. If either phase fails, check the text you just added to blacklist_custom_msg_ips_prod - most likely issue is missing commas, missing /32, missing quotes or copy/pasted smart quotes. Update the variable and repeat steps 6-7.
9. It will take 10-15 minutes for the block to take effect. After waiting 15min, check the IP(s) in the Datadog logs tab to verify that requests from those IPs are being blocked.

Reverting/overwriting Elastic search index

In case of serious issues with the data release process, it's possible to revert the Elasticsearch index to a previous data dump.

ES index can be reverted to any previously published data dump by manually triggering PROD index full data dump or STAGING index full data dump actions from the Github UI.

If it's not possible to use the Github actions above, the index can be replaced by running commands inside running ECS containers.

1. SSH to running ECS container for relevant environment (dev, staging or prod) - see Bastion host entry in 1Password

2. Run the setup up command and pass the filename of the data dump and schema version you want to index (no file extension). File must exist in ror-data. To index multiple schema versions, you must run the setup command multiple times with different schema versions specified.

    python manage.py setup v1.0-2022-03-17-ror-data -s 2
    python manage.py setup v1.0-2022-03-17-ror-data -s 1