Logging & monitoring

Datadog log/metric aggregation

• Load balancer and container logs for ROR API and Generate ID apps are forwarded from AWS to Datadog log aggregation service via the Datadog Forwarder Lambda Function
• Metrics for all services that have metrics configured are forwarded to Datadog via the Datadog AWS integration (note: despite the docs saying that this integration also forwards logs, it doesn't)
• We use Datadog's EU service, accessible at https://datadoghq.eu
• Each user should have their own Datadog account - contact tech lead to get an account. There is a backup account registered to ror-bot@ror.org. Creds in 1Password.
• A public Datadog dashboard with basic usage stats is located at https://p.datadoghq.eu/sb/db1aec04-0c1a-11ec-860a-da7ad0900005-7d7c572812608235cca3359ee5ec591a

Common Datadog tasks

Find API usage over the past month (or week, day, hour, etc)

Get basic stats about API usage (total requests, total requests by type, total requests by API version, top 100 request URLs, etc) from the ROR API Usage Insights Dashboard

<iframe src="https://p.datadoghq.eu/sb/embed/db1aec04-0c1a-11ec-860a-da7ad0900005-7d7c572812608235cca3359ee5ec591a" title="ROR API Usage Insights" width="800" height="800"></iframe>

Cloudwatch logs

• For most other services, logs are available in AWS Cloudwatch
• Historically, log configuration for new infrastructure was spotty, so some services don't have logging configured

Monitoring

• API and Reconciler uptime are monitored with Pingdom
• Public status page is (https://aws.amazon.com/pm/cloudwatch
• Pingdom downtime alerts are sent to Slack #status channel
• Elasticsearch CPU is monitored via Datadog. An alert is triggered in Slack #status when CPU usage reaches >80% for >5min