Sysctl type fails on some systems when run from a backgrounded puppet agent

[jhoblitt]

I'm baffled as to why I'm seeing failures with the sysctl type on some el6.x systems in prod. I assume it has something to do the system setup, and not the provider, but I'm scratching my head as to the root cause. When the agent is run as puppet agent --test, even from ssh without a pty, it works, so I've been unable to reproduce the failure to debug it. I'm hoping someone has seen this error before or has a suggestion as to how to debug it.

puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv4.tcp_mtu_probing]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5961b8e220>
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.default.disable_ipv6]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5961b8dfa0>
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.all.disable_ipv6]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5961b8dff0>
puppet-agent[7585]: Finished catalog run in 90.24 seconds
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv4.tcp_mtu_probing]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5963580548>
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.default.disable_ipv6]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5963580250>
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.all.disable_ipv6]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f59635802c8>
puppet-agent[7585]: Finished catalog run in 89.20 seconds
puppet-agent[28159]: Ignoring --listen on onetime run
puppet-agent[28159]: Retrieving plugin
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_pci.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/postgres_default_version.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_http.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_hostgroups.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/megaraid.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_dns.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/tw.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_contactgroups.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_membase.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/nagios_moxi.rb
puppet-agent[28159]: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
puppet-agent[28159]: Caching catalog for dts-tu-podi.tuc.noao.edu
puppet-agent[28159]: Applying configuration version '1371495888'
puppet-agent[28159]: (/Stage[main]/Sdm::Users::Dts-tu-podi/User[pakzad]/ensure) removed
puppet-agent[28159]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.default.disable_ipv6]/value) changed live value from '0' to '1'
puppet-agent[28159]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.all.disable_ipv6]/value) changed live value from '0' to '1'
puppet-agent[28159]: Finished catalog run in 13.01 seconds
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv4.tcp_mtu_probing]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5961591778>
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.default.disable_ipv6]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f59615914f8>
puppet-agent[7585]: (/Stage[main]/Sdm::Site::Tu/Sysctl[net.ipv6.conf.all.disable_ipv6]) Could not evaluate: undefined method `value' for #<Puppet::Type::Sysctl::ProviderParsed:0x7f5961591548>
puppet-agent[7585]: Finished catalog run in 72.65 seconds

[domcleal]

It looks like a different provider is being used ("ProviderParsed") but with the sysctl type from augeasproviders, perhaps check for any other conflicting modules that supply a sysctl provider. You could try setting the provider explicitly to "augeas" too.

[jhoblitt]

I suspect your right and some part of the provider from duritong/puppet-sysctl, which I was previously using, is hanging around but I'm at a total loss to find it.

# locate Sysctl.rb sysctl.rb
/var/lib/puppet/lib/puppet/type/sysctl.rb

# grep -Ril sysctl /var/lib/puppet/lib
/var/lib/puppet/lib/puppet/type/sysctl.rb
/var/lib/puppet/lib/puppet/provider/sysctl/augeas.rb

# head /var/lib/puppet/lib/puppet/type/sysctl.rb
# Manages entries in /etc/sysctl.conf
#
# Copyright (c) 2012 Dominic Cleal
# Licensed under the Apache License, Version 2.0

Puppet::Type.newtype(:sysctl) do
  @doc = "Manages entries in /etc/sysctl.conf."

  ensurable

[domcleal]

Looking at the pids in the initial logs, it appears you have one long running process (daemonised?) and perhaps a puppet agent -t run too. Have you restarted the daemon since switching providers? It's likely it still has the old code loaded.

Other thoughts: if the agent is on the puppetmaster, check the module directories are clear of the old provider, since it could be loading the provider from here. Also check other environments, since there's no separation of plugins between environments. You could also try adding --trace to the command line, which might point out the file it's loading from.

[domcleal]

Closing, hopefully that fixed it.