Logging & monitoring
- Load balancer and container logs for ROR API and Generate ID apps are forwarded from AWS to Datadog log aggregation service via the Datadog Forwarder Lambda Function
- Metrics for all services that have metrics configured are forwarded to Datadog via the Datadog AWS integration (note: despite the docs saying that this integration also forwards logs, it doesn't)
- We use Datadog's EU service, accessible at https://datadoghq.eu
- Each user should have their own Datadog account - contact tech lead to get an account. There is a backup account registered to ror-bot@ror.org. Creds in 1Password.
- A public Datadog dashboard with basic usage stats is located at https://p.datadoghq.eu/sb/db1aec04-0c1a-11ec-860a-da7ad0900005-7d7c572812608235cca3359ee5ec591a
Get basic stats about API usage (total requests, total requests by type, total requests by API version, top 100 request URLs, etc) from the ROR API Usage Insights Dashboard
When CPU alarms are triggered and/or users complain of latency, it's often an indication that a new user has started sending a heavy volume of requests, or an existing user has increased their request volume and/or complexity. It can also be due to a user sending many requests that generate errors.
- Go to the logs tab and adjust the time period to the past 1 day, 2 days or 1 week and look for indications of major changes (steep increase in requests and/or errors)
- Zoom in on the period of time where the increase ocurred
- Use the Client IP filter to check which IPs sent the highest volume of requests during that period - those are your most likely culprits. Filter the view based on Client IP
- Alternately the http.url filter to check which requests occurred frequently during that time period - are there any problematic requests that are generating errors or requests being sent many thousands of times? Filter the view based on http.
- With the view filted on http.url, use the Client IP filter to check which IPs are sending the problematic requests.
- After identifying problem IPs, use https://whatismyipaddress.com to check the source of the IP. In most cases, the source does not provide much information about who the application sending the requests belongs to, but if it does, try contacting the organization.
- If an organization can't be identified from the IPs or we get no response from the organization, block or apply additional rate limiting to the IPs via Terraform (see Blocking Problem IPs).
If a user reports an issue such as an error response or performance issue, try to get the following information:
- Approximate time of requests(s)
- Full request URL
- User IP address (IP addresses can over hours or days, so their current IP may not be the same as the request IP)
- Go to the logs tab and adjust the time period to cover the approximate time the issue took place
- Search for the user's IP in the Client IP filter and apply filter to view. If it doesn't exist in the time period, remove the filter.
- Search for the user's request in the http.url filter and apply filter to view
- Review the log messages for possible error sources. In the case of slow performance, if there are no issues with the user's request, remove the http.url and Client IP filters and look at the requests the happened around the same time as the user's requests. Other API users sending a high volume of requests, highly complicated requests or requests that generate errors can cause performance issues for all API users.
When looking at API traffic, we are primarily interested in ELB (load balancer) logs, as user requests enter and exit the ROR system through the load balancer. When looking for error messages or print statements from specific applications, different log filtering is needed.
- Go to the logs tab
- In the Source filter, choose Cloudwatch
- In the Host filter, choose the specific application you are intersted in (api-community is the production ROR API, api-dev and api-staging are the Dev and Staging APIs, generateid is middleware for adding credentials to requests to private API routes in a secure way)
- For most other services, logs are available in AWS Cloudwatch
- Historically, log configuration for new infrastructure was spotty, so some services don't have logging configured
- API and Reconciler uptime are monitored with Pingdom
- Public status page is (https://aws.amazon.com/pm/cloudwatch
- Pingdom downtime alerts are sent to Slack #status channel
- Elasticsearch CPU is monitored via Datadog. An alert is triggered in Slack #status when CPU usage reaches >80% for >5min