monitoring: cadvisor recording rules seem broken

[bobheadxi]

no data available anywhere. the dashboard queries themselves appear unchanged, and cadvisor doesnt seem down. the recording rule is reporting wonky names, so this might be some change in behaviour in cadvisor or prometheus:

[bobheadxi]

might be permissions issue: google/cadvisor#2270

[bobheadxi]

could be related to container runtime change to OCI (#15194)? though this seems to be an issue in docker-compose deployments too (e.g. demo.sourcegraph.com seems to have incomplete cadvisor names) separate issue: #17072

[bobheadxi]

Also confirmed that the services are being picked up, they just don't have a name, https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s-2/pull/702 experiments with setting it ourselves

Update: this doesn't seem to be setting the name correctly

[bobheadxi]

fix in k8s: https://k8s.sgdev.org/-/debug/grafana/explore?orgId=1&left=%5B%22now-5m%22,%22now%22,%22Prometheus%22,%7B%22expr%22:%22sum%20by(name)%20(container_cpu_usage_seconds_total%7Bname%3D~%5C%22%5Egitserver.*%5C%22%7D)%22,%22requestId%22:%22Q-71de0b6e-b74e-433f-aa1c-94cc0bc8f7db-0A%22,%22instant%22:true,%22range%22:false%7D%5D

[bobheadxi]

Status update

As of yesterday, the following patches provided a preliminary fix for this issue:

• monitoring: update cadvisor matcher #17096
• prometheus: set cadvisor metrics name labels deploy-sourcegraph#1572
• https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/4119
• https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s-2/compare/c2ce93928e[…]c912fc886a1a628f4916...4f59e3199b70b5980fb9637d03ceb551943e26e3

As of now, most dashboards seem to be back to normal in k8s.sgdev.org and Cloud.

There is one last problem, described and kind of fixed in sourcegraph/deploy-sourcegraph#1578, that needs to be addressed. There is also #17072, which should be fixed - I will verify and close that issue soon. These two issues are the only ones that might impact customers at the moment (unless they too have switched to OCI, in which case they will need this set of fixes)

There are still a few dashboards broken, most notably the fs-inode dashboards. It seems there's something different about the labelling on those - will require more investigation

[bobheadxi]

Some potentially related issues: google/cadvisor#2666, google/cadvisor#2215 - it seems that containerd support is not very fleshed out yet in cAdvisor, cc @daxmc99

[daxmc99]

Hmm from the linked issue google/cadvisor#2215 it looks like the preference is querying containerd directly. Not sure the best way to expose that yet