Observability tools

[andrewm4894]

• add Prometheus & Grafana for custom metrics and visualization (/metrics endpoints and anything else we might want to add).
• add netdata for infrastructure monitoring and alerts (redis, postgres, containers, also prometheus metrics too etc)
• configure netdata to collect postgress, redis, and container metrics.
• configure Prometheus to scrape itself, backend, and inference-server.
• optional env var of NETDATA_CLAIM_TOKEN to claim to netdata cloud - makes it easier to work with infra and alerts to discord etc. I work there so am pretty sure can get us a free sponsored space that might be useful. Not trying to sell here or anything, just that it's a potential useful overlap given i work there :) .
• add initial sort of dummy fastapi custom dashboard in docker/grafana/dashboards. Idea is we can save dashboards as code in there (NOTE: needs much more work - anyone can add/improve dashboards as follow on PR's, my promql skills not great).
• add observability tools to observability docker compose profile (NOTE: not sure what best approach is here, would need some input from other more familiar with the docker set up).
• add Grafana on port 2000 instead of 3000 since app itself on 3000.
• add some README.md under each /docker folder.

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[github-actions]

❌ pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

[yk]

very cool, thank you so much. didn't know about netdata yet, interested to see how it does. eventually we'll need to get this stuff into the playbooks

[andrewm4894]

very cool, thank you so much. didn't know about netdata yet, interested to see how it does. eventually we'll need to get this stuff into the playbooks

Yep, was wondering about how to actually get it deployed as such. I see some stuff in /ansible but not 100% sure on that side of things yet in terms of how it hangs together.

[andrewm4894]

Merged anyway (so people can add better dashboards etc in grafana than mine) and maybe can add additional PR to add to the playbooks for deployment.