[static systemd services] => PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/....

[HolgerHees]

I know that there are some bugreports related to cgroups but I think this is different, because it is related to short running static services.

On my system the services "systemd-hostnamed" and "snapperd" are triggered every hour and are running between 30 seconds and 1 minute. Every time when they succeeded I get a message like

Aug 04 15:00:02 marvin systemd[1]: Starting Hostname Service...
Aug 04 15:00:02 marvin systemd[1]: Started Hostname Service.
Aug 04 15:00:32 marvin systemd[1]: systemd-hostnamed.service: Succeeded.
Aug 04 15:00:33 marvin netdata[1509]: PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/systemd-hostnamed.service/blkio.throttle.io_serviced'
Aug 04 15:00:34 marvin netdata[1509]: PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/systemd-hostnamed.service/blkio.throttle.io_serviced'

or

Aug 04 15:00:02 marvin systemd[1]: Starting DBus interface for snapper...
Aug 04 15:00:02 marvin systemd[1]: Started DBus interface for snapper.
Aug 04 15:01:02 marvin systemd[1]: snapperd.service: Succeeded.
Aug 04 15:01:02 marvin netdata[1509]: PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/snapperd.service/blkio.throttle.io_serviced'
Aug 04 15:01:03 marvin netdata[1509]: PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/snapperd.service/blkio.throttle.io_serviced'

OS / Environment

os: openSuSE 15.3
kernel: 5.3.18-59.16-preempt
systemd: 246.13

Netdata version

Version: netdata v1.31.0
Configure options:  '--host=x86_64-suse-linux-gnu' '--build=x86_64-suse-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/lib' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-dependency-tracking' '--docdir=/usr/share/doc/packages/netdata-1.31.0' '--enable-plugin-nfacct' '--enable-plugin-freeipmi' '--enable-plugin-cups' '--with-zlib' '--with-math' '--with-user=netdata' 'build_alias=x86_64-suse-linux-gnu' 'host_alias=x86_64-suse-linux-gnu' 'CFLAGS=-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' 'CXXFLAGS=-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'
Features:
    dbengine:                YES
    Native HTTPS:            YES
    Netdata Cloud:           YES 
    Cloud Implementation:    Next Generation
    TLS Host Verification:   YES
Libraries:
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  YES
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    NO
    IPMI:                    YES
    NFACCT:                  YES
    perf:                    YES
    slabinfo:                YES
    Xen:                     YES
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES

[ilyam8]

Hi, @HolgerHees.

You can filter out those cgroups. Use the "enable by default cgroups matching" option in the netdata.conf. The filter syntax is simple patterns.

[HolgerHees]

thanks. I already found this parameter.

I want to avoid using it. Because then I have to repeat to long default pattern + my additional ones and if you change the default one in the future, I have to adapt this changes too.

maybe is there an option to extend the default parameter?

[ilyam8]

Unfortunately, there is no way to extend it but only replace it. Repeating is not a problem, because you just uncomment the option and prepend your patterns.

if you change the default one in the future, I have to adapt this changes too

That makes sense, you are correct. But I think that some imaginary problem from the future shouldn't stop you from solving the one you are having now.

[HolgerHees]

I changed it now on my system.

During investigation I found more similar cases

snapperd, systemd-hostnamed, logrotate and backup-rpmdb are affected. All of these a static triggered jobs and the error is related to blkio.

Maybe on a long term there is a more generic fix.

[ilyam8]

@HolgerHees Where did you get those logs from?

[HolgerHees]

from journalctl

and the netdata related config is "error log = syslog"

btw. next service I found with this issues is "systemd-tmpfiles-clean". also a static triggered job

[ilyam8]

Aug 04 15:01:02 marvin netdata[1509]: PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/snapperd.service/blkio.throttle.io_serviced'
Aug 04 15:01:03 marvin netdata[1509]: PROCFILE: Cannot open file '/sys/fs/cgroup/blkio/system.slice/snapperd.service/blkio.throttle.io_serviced'

@vlvkobal the problem is the severity level of the message ("error"). We log these messages on containers deletion. That is not an unexpected event. I think we don't even need to make it "info" level, but to not log at all ("errno 2, No such file or directory" case).

[HolgerHees]

any progress here? Just to avoid that this "small ugliness" is forgotten.

On my homeserver I follow the zero error message rule for all my logs. And the netdata error messages are the only leftovers ;-)

any my setup is not a small one. I run 27 containers. https://github.com/HolgerHees/smartserver

[ilyam8]

Hi, @HolgerHees. No, there is no progress. @vlvkobal let's discuss this issue when you are available.