You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We started the system metrics collector back in the day when applications ran on dedicated hosts! Now, almost everything runs inside containers and we should ensure that Kamon does its best in those cases. This means:
Fully implement process-specific metrics. Currently, we are only reporting CPU usage but it would be great to include memory and IO (network and disk) if possible
Automatically disable host metrics when running inside a container. Host metrics could (and probably) should be collected by other means when the host is meant to run containers. We should have a setting to force enabling host metrics in case folks want to have it anyways
Figure out if there is a way to use OSHI in read-only containers. We are unpacking the native libraries at runtime and loading them, but that requires a writable file system. I'm sure there is a way around this, just need time to investigate and share
As much as possible, align the metric/tag names with the ones in the OpenTelemetry semantic conventions for system and process metrics
Notes from early investigation:
We could "hack" reading the cgroup memory limits by reading /sys/fs/cgroup/memory.max (see Feature Request: expose cgroup memory limits when running in Docker oshi/oshi#893 for more info). In my local tests, the file contains the proper memory limit when a container is started with a memory limit, but contains max when there are no limits. In that case we might fallback to the global memory limit. This was a local docker container, though. I'll test further in EKS to see how it goes there.
So far, I'm not aware of any reliable way to detect whether the application is running inside a container or not. This needs research.
The text was updated successfully, but these errors were encountered:
We started the system metrics collector back in the day when applications ran on dedicated hosts! Now, almost everything runs inside containers and we should ensure that Kamon does its best in those cases. This means:
Notes from early investigation:
/sys/fs/cgroup/memory.max
(see Feature Request: expose cgroup memory limits when running in Docker oshi/oshi#893 for more info). In my local tests, the file contains the proper memory limit when a container is started with a memory limit, but containsmax
when there are no limits. In that case we might fallback to the global memory limit. This was a local docker container, though. I'll test further in EKS to see how it goes there.The text was updated successfully, but these errors were encountered: