feat: Add Prometheus metrics and health endpoints

[mhamzahkhan]

Hello

I'm not sure if this will be useful for anyone else, but I've added a few Prometheus metrics to the relay and thought I'd share. I'm pretty sure there are other ways to get these same metrics by adding external services but I prefer to just expose them internally. Feel free to reject the PR if it's not useful to anyone else, or if it is just horrendous.

Quick disclaimer: I'm not a developer by trade, I've just been messing around with Go on weekends, so please go easy on me. 😄

I borrowed/copied some of the code from Thanos, but I'm not 100% sure if what I did makes sense, or if there are better ways of doing it. I would be very happy to receive feedback for improvement.

What I Did:

• Added Prometheus metrics for the HTTP endpoints.
• Created health and readiness endpoints, mainly for use with K8S. Originally I had the probes going to /.well-known/nodeinfo, but this skews the counters for that handler. I figured having two separate endpoints, I could use the readiness to define if the application has started and is ready to accept traffic and the healthy endpoint I was planning to try to add simple checks (eg redis ping).

I'd also like to add metrics for the machinery task queue, for example, how many tasks are in the queue, how many succeeded/failed, how many tasks are actively being processed, and average completion times.

Plus, I want to expose some general relay stats (e.g., how many instances are subscribed, how many are still alive/failed, etc.). I haven't figured out how to do that yet.