Spike: Metrics/Monitoring

[arein]

AC:

• export metrics somewhere (I remember @Harry saying that GoTrue is integrated w/Datadog)
• integrate with our Grafana instance
• define top-level metrics
• setup dashboards
• setup monitors

[HarryET]

Metrics can be done, and GoTrue uses OpenTracing. So I could edit our fork to add Prometheus support but that then starts to become a hard fork and that is not what we want to do. I will research today the possibility of using Datadog and check if we already have it...

[HarryET]

@arein, what metrics were we looking to get from this?

• Requests count
• 2xx vs 4xx vs 5xx

[arein]

Yup and latency. Ideally we have two top-level metrics: availability (derived from 2xx/5xx) and latency.
As discussed offline these can be gathered from other sources e.g. the load balancer

[HarryET]

Open Telem is now working with a sidecar container the IAM permissions just need to be looked at for the execution role as I'm not sure it has permissions to create new X-Ray/Cloudwatch traces

[arein]

https://github.com/WalletConnect/rs-relay/pull/156/files is adding that access (not really following least-privilege in that example tho)