seccomp notifier metric cardinality

[pdf]

Looking at the recently added seccomp metrics, it looks like the syscalls label has essentially unbounded cardinality, since it appears that syscalls and counts are concatenated as a string value, e.g.:

container_runtime_crio_containers_seccomp_notifier_count_total{name="…",syscalls="chroot (1x)"} 1
container_runtime_crio_containers_seccomp_notifier_count_total{name="…",syscalls="chroot (2x), swapoff (2x)"} 1
container_runtime_crio_containers_seccomp_notifier_count_total{name="…",syscalls="chroot (1x), swapoff (2x)"} 1
container_runtime_crio_containers_seccomp_notifier_count_total{name="...",syscalls="swapoff (5x)"} 1

So, every possible combination of syscall and count may published to this label, making the possible value permutations infinite.

I'm not sure what the ideal structure for these metrics should be, but I don't think the current metrics should stand as they are.

[saschagrunert]

Yeah, some features are unlikely to produce many metrics and in this case we decided to add them as they are. We could switch them to a different format, like:

container_runtime_crio_containers_seccomp_notifier_count_total{name="abc",syscalls="chroot"} 2
container_runtime_crio_containers_seccomp_notifier_count_total{name="abc",syscalls="swapoff"} 1

The question is would that make anything easier to debug… 🤔

[pdf]

I don't think it would make things harder to debug, which is probably the more important consideration.

The real killer for cardinality is the counts - with just concatenated (sorted) syscall names, the label values are at least bounded by the permutations of syscall names, though I still don't love that.

If you stick to a single label per point it also opens up the possibility to query by syscall without having to resort to regexp, however useful that might or might not be.

[pdf]

@haircommander I had a look last night and the change itself looks straight-forward, but I had trouble getting reliable test runs locally so I think I'd have to build a dedicated env to get the test suite sorted, likely some days out before I can allocate the time.

[haircommander]

FYI, I'm looking to cut 1.26.0 early next week, and I'd love to have this fixed by then. If you need, you could use our CI as a way to check against the tests. not the best practice usually, but good in a pinch like this. WDYT?

[pdf]

I should have some time over the weekend, will try to sort it locally, otherwise I'll take the push it and see approach. Totally understand the desire to get it sorted before the current metric makes release.

[haircommander]

Thank you for taking this on and conforming to the now tight schedule 😄

[pdf]

@haircommander I couldn't work out the test suite in time, drafted a PR for testing on CI at #6456