systemd daemon-reload restarts snmpd

[3c2b2ff5]

Hi,

I am seeing a strange behavior of snmpd. When ever systemctl daemon-reload is triggered, snmpd is restarted.

I am running keepalived with snmp support to be able to monitor the state of my vrrp instances, but when snmpd is restarted, keepalived needs a restart as well, otherwise the connection to AgentX is lost.

How is this connected to systemd? could it be that the agent socket is managed by systemd? How can I avoid this behavior?

Thank you very much

[ubellavance]

What is the Linux distribution involved? Do you have commercial support?

[3c2b2ff5]

I am running Debian 10 with 5.10.0-0.bpo.3-amd64 Kernel. I am afraid we don't have a commercial support!

[ubellavance]

Does the problem occur when you execute systemctl daemon-reload manually, outside a Puppet context? If it does I think you would get better answers by writing to a Debian mailing list or forum.

[3c2b2ff5]

It happens when systemctl daemon-reload is executed within puppet context

[TuningYourCode]

@3c2b2ff5 do you have an example puppet report/output that we can see which resources are involved?

The only real connection between systemd and snmpd i identfied at a quick look is https://github.com/voxpupuli/puppet-snmp/blob/master/manifests/init.pp#L460-L463 or if snmpd config changes.

It would also be helpful if you can tell us your used versions of the modules (and/or update them to the lastest once if not already done)

[3c2b2ff5]

I modified /etc/systemd/system/consul.service.d/consul.conf file and executed systemctl daemon-reload manually, nothing happened, then I run the puppet agent to restore the config and in syslog you can see the restart of snmpd:

2021-07-21T15:06:02.077892+00:00 localhost puppet-agent[11582]: (/Stage[main]/consul/Systemd::Dropin_file[consul.conf]/File[/etc/systemd/system/consul.service.d/consul.conf]) Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
2021-07-21T15:06:02.312899+00:00 localhost systemd[1]: Stopping Consul Agent... 
2021-07-21T15:06:06.926292+00:00 localhost systemd[1]: consul.service: Succeeded.
2021-07-21T15:06:06.926936+00:00 localhost systemd[1]: Stopped Consul Agent.
2021-07-21T15:06:06.929790+00:00 localhost systemd[1]: Starting Consul Agent... 
2021-07-21T15:06:07.019185+00:00 localhost systemd[1]: Started Consul Agent.
2021-07-21T15:06:07.025687+00:00 localhost puppet-agent[11582]: (/Stage[main]/Consul::Run_service/Service[consul]) Triggered 'refresh' from 1 event
2021-07-21T15:06:07.026135+00:00 localhost puppet-agent[11582]: (Class[Systemd::Systemctl::Daemon_reload]) Scheduling refresh of Exec[systemctl-daemon-reload]
2021-07-21T15:06:07.036345+00:00 localhost systemd[1]: Reloading.
2021-07-21T15:06:07.188868+00:00 localhost puppet-agent[11582]: (/Stage[main]/Systemd::Systemctl::Daemon_reload/Exec[systemctl-daemon-reload]) Triggered 'refresh' from 1 event
2021-07-21T15:06:07.189873+00:00 localhost puppet-agent[11582]: (Class[Systemd::Systemctl::Daemon_reload]) Scheduling refresh of Service[snmpd]
2021-07-21T15:06:07.247316+00:00 localhost systemd[1]: Stopping Simple Network Management Protocol (SNMP) Daemon....
2021-07-21T15:06:07.248033+00:00 localhost snmpd[16564]: Received TERM or STOP signal...  shutting down...
2021-07-21T15:06:07.249842+00:00 localhost Keepalived_vrrp[25863]: AgentX master disconnected us, reconnecting in 15
2021-07-21T15:06:07.251901+00:00 localhost systemd[1]: snmpd.service: Succeeded.
2021-07-21T15:06:07.252706+00:00 localhost systemd[1]: Stopped Simple Network Management Protocol (SNMP) Daemon.. 
2021-07-21T15:06:07.254869+00:00 localhost systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon....
2021-07-21T15:06:07.257770+00:00 localhost systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon.. 
2021-07-21T15:06:07.262014+00:00 localhost puppet-agent[11582]: (/Stage[main]/Snmp/Service[snmpd]) Triggered 'refresh' from 1 event
2021-07-21T15:06:07.282456+00:00 localhost snmpd[12815]: Turning on AgentX master support. 
2021-07-21T15:06:07.283718+00:00 localhost snmpd[12815]: Turning on AgentX master support. 
2021-07-21T15:06:07.285059+00:00 localhost snmpd[12815]: NET-SNMP version 5.7.3
2021-07-21T15:06:14.498210+00:00 localhost puppet-agent[11582]: Applied catalog in 30.74 seconds

[3c2b2ff5]

Well I can update some modules at the next schedule, we need to coordinate this first. But for now the module versions are:

mod 'KyleAnderson/consul', '7.0.2'
mod 'puppet/snmp', '5.1.1'
mod 'puppet/keepalived', '3.1.0'

[ubellavance]

You're at the latest version of this module so I think you'll have to get the module modified for your use case, or submit a PR.

[3c2b2ff5]

Thank you very much @ubellavance

[3c2b2ff5]

You're at the latest version of this module so I think you'll have to get the module modified for your use case, or submit a PR.

How can I get it modified?

[ubellavance]

You're at the latest version of this module so I think you'll have to get the module modified for your use case, or submit a PR.

How can I get it modified?

You can wait until someone does, or try to find a puppet consultant that will do the PR for you.

[3c2b2ff5]

OK, sure thanks

[3c2b2ff5]

This an other incident I had yesterday:

2021-07-25T06:49:52.476020+00:00 localhost systemd[1]: Starting Daily apt upgrade and clean activities...
2021-07-25T06:49:54.718776+00:00 localhost systemd[1]: apt-daily-upgrade.service: Succeeded.
2021-07-25T06:49:54.719219+00:00 localhost systemd[1]: Started Daily apt upgrade and clean activities.
2021-07-25T06:50:01.103590+00:00 localhost CRON[6530]: (root) CMD (    ipset-save > /etc/ipset)
2021-07-25T06:50:01.104716+00:00 localhost CRON[6532]: (root) CMD (    /usr/local/sbin/iptables-save > /etc/iptables/rules.v4)
2021-07-25T06:51:47.296931+00:00 localhost Keepalived_vrrp[3099]: AgentX master disconnected us, reconnecting in 15
2021-07-25T06:51:47.297133+00:00 localhost Keepalived_vrrp[3099]: AgentX master agent failed to respond to ping.  Attempting to re-register.
2021-07-25T06:51:47.297191+00:00 localhost Keepalived_vrrp[3099]: scheduler: Error performing epoll_ctl DEL op for fd:15 (Bad file descriptor)
2021-07-25T06:52:02.297994+00:00 localhost Keepalived_vrrp[3099]: NET-SNMP version 5.7.3 AgentX subagent connected

[3c2b2ff5]

It seems the dependency here is solved in master, but there is no tagged release for the master yet.

[3c2b2ff5]

I believe this is now fixed with 6.0.0. I hope so at least.