chef-client 12.4.0 wrongly detects sysvinit instead of systemd on CentOS 7

[ameir]

I was running the httpd cookbook, which has previously worked just fine. Today, though, it wasn't. I thought that someone overwrote a version with bad changes, but I tried the fresh cookbook version from GitHub just to be safe, and had the same issues. I downgraded from chef-client 12.4.0 to 12.2.1 and things began to work just fine. In short, it looks like Ohai or whatever does the init system detection is being thrown off somewhere. Here's a snippet of the logs; I have the entire debug log if needed.

10.98.193.143     * template[default :create /etc/init.d/httpd-default] action create[2015-06-25T17:53:59+00:00] INFO: Processing template[default :create /etc/init.d/httpd-default] action create (/var/c
hef/cache/cookbooks/httpd/libraries/provider_httpd_service_rhel_sysvinit.rb line 27)
10.98.193.143 [2015-06-25T17:53:59+00:00] DEBUG: Providers for generic template resource enabled on node include: [Chef::Provider::Template]
10.98.193.143 [2015-06-25T17:53:59+00:00] DEBUG: Provider for action create on resource template[default :create /etc/init.d/httpd-default] is Chef::Provider::Template
10.98.193.143
10.98.193.143       ESC[0m
10.98.193.143       ================================================================================ESC[0m
10.98.193.143       ESC[31mError executing action `create` on resource 'template[default :create /etc/init.d/httpd-default]'ESC[0m
10.98.193.143       ================================================================================ESC[0m
10.98.193.143
10.98.193.143 ESC[0m      Chef::Exceptions::FileNotFoundESC[0m
10.98.193.143       ------------------------------ESC[0m
10.98.193.143       Cookbook 'httpd' (0.2.14) does not contain a file at any of these locations:
10.98.193.143 ESC[0m        templates/centos-7.1.1503/2.4/sysvinit/el-7/httpd.erb
10.98.193.143 ESC[0m        templates/centos/2.4/sysvinit/el-7/httpd.erb
10.98.193.143 ESC[0m        templates/default/2.4/sysvinit/el-7/httpd.erb
10.98.193.143 ESC[0m        templates/2.4/sysvinit/el-7/httpd.erb
10.98.193.143 ESC[0m
10.98.193.143 ESC[0m      This cookbook _does_ contain: ['/var/chef/cache/cookbooks/httpd/templates/default/2.2/mods/mpm.conf.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.2/scripts/a2enmod.e
rb','/var/chef/cache/cookbooks/httpd/templates/default/2.2/sysvinit/debian-7/apache2.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.2/sysvinit/el-5/httpd.erb','/var/chef/cache/cookbooks/httpd/
templates/default/2.2/sysvinit/el-6/httpd.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.2/sysvinit/ubuntu-10.04/apache2.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.2/sysvinit/ub
untu-12.04/apache2.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.4/mods/rhel/mpm.conf.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.4/scripts/a2enmod.erb','/var/chef/cache/cookboo
ks/httpd/templates/default/2.4/sysvinit/el-6/httpd.erb','/var/chef/cache/cookbooks/httpd/templates/default/2.4/sysvinit/ubuntu-14.04/apache2.erb','/var/chef/cache/cookbooks/httpd/templates/default/envvar
s.erb','/var/chef/cache/cookbooks/httpd/templates/default/httpd.conf.erb','/var/chef/cache/cookbooks/httpd/templates/default/magic.erb','/var/chef/cache/cookbooks/httpd/templates/default/module_load.erb'
,'/var/chef/cache/cookbooks/httpd/templates/default/mpm.conf.erb','/var/chef/cache/cookbooks/httpd/templates/default/rhel/sysconfig/httpd-2.2.erb','/var/chef/cache/cookbooks/httpd/templates/default/rhel/
sysconfig/httpd-2.4.erb','/var/chef/cache/cookbooks/httpd/templates/default/systemd/httpd.service.erb']ESC[0m
10.98.193.143
10.98.193.143 ESC[0m      Resource Declaration:ESC[0m
10.98.193.143       ---------------------ESC[0m
10.98.193.143       # In /var/chef/cache/cookbooks/httpd/libraries/provider_httpd_service_rhel_sysvinit.rb
10.98.193.143 ESC[0m
10.98.193.143 ESC[0m       27:             template "#{new_resource.name} :create /etc/init.d/#{apache_name}" do
10.98.193.143 ESC[0m       28:               path "/etc/init.d/#{apache_name}"
10.98.193.143 ESC[0m       29:               source "#{parsed_version}/sysvinit/el-#{elversion}/httpd.erb"
10.98.193.143 ESC[0m       30:               owner 'root'
10.98.193.143 ESC[0m       31:               group 'root'
10.98.193.143 ESC[0m       32:               mode '0755'
10.98.193.143 ESC[0m       33:               variables(apache_name: apache_name)
10.98.193.143 ESC[0m       34:               cookbook 'httpd'
10.98.193.143 ESC[0m       35:               action :create
10.98.193.143 ESC[0m       36:             end

[coderanger]

Guessing this is due to the change in ordering rules for provider resolution. The fix will probably be in the httpd cookbook, not Chef.

[btm]

@ameir what platform(s) are you running on?

It looks like the template resource is failing because you're running on enterprise linux 7, the cookbook uses platform version to choose the template source, and doesn't have files for el7.

https://github.com/chef-cookbooks/httpd/tree/master/templates/default/2.4/sysvinit

[coderanger]

I'm betting it is because of this https://github.com/chef-cookbooks/httpd/blob/master/libraries/provider_httpd_service_rhel_systemd.rb#L14-L16

Platform should probably be rhel, not redhat.

[btm]

@coderanger platform redhat looks correct, which is a different thing than platform_family rhel, which is obviously confusing.

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb#L24

[lamont-granquist]

platform should be redhat, its platform_family which is rhel

oracle should also be treated identically to redhat, and it looks like it should check platform_family of rhel instead (which won't work on chef 11, but provides lines on providers doesn't work in chef 11, neither does the block for the platform_version check).

[btm]

@lamont-granquist there's a comment there that it doesn't use platform_family to avoid amazon (chef-boneyard/chef-rfc#109)

@someara and @jdmundrawala are replicating the issue now to determine the best place for a fix.

[coderanger]

Hmm, then probably a provider resolver ordering issue, the provides looks correct on the systemd provider to me at least.

[lamont-granquist]

yeah, nm, i just read that.

still, oracle is wrong.

[lamont-granquist]

@ameir it would be useful to get fuller debug logs which include the provider_resolver output for when the Systemd or Sysvinit providers get selected. The debug log for the template resource that fails isn't useful.

Although it does look like the platform+platform_version are centos-7.1.1503 based on the template resource output so that's useful.

[lamont-granquist]

So the fix would be to make sure that this is only applied if platform_version < 7:

https://github.com/chef-cookbooks/httpd/blob/master/libraries/provider_httpd_service_rhel_sysvinit.rb#L13-L14

Both providers match on centos, and for some reason the NodeMap stuff is not considering the filter with the block to be of a 'higher priority' and returning it. Which is a bug in core chef AFAIK (probably something that @jkeiser needs to look at). But the problem can be resolved here by making the selection of the provider unambiguous. You always want to select sysvinit on centos/scientific/oracle/redhat where pv < 7.0 and always want to select systemd on those when pv >= 7.0 so that is exactly how the provides lines should be fixed to read.

[lamont-granquist]

In Chef 12.3.0 this code should have also spit out a WARN "ambiguous provider resolution" message.

[lamont-granquist]

Actually there's even better detection of sysvinit vs systemd in Chef 12 that the httpd cookbook should use via supports? code rather than doing pv checks.

[ameir]

Looking at the debug logs more, I do see that in the run with chef-client 12.4.0, php-fpm indeed used the right provider, but not httpd. That certainly suggests the issue is at the cookbook level, not in chef-client; my apologies.

Here are snippets from two runs with two different client versions:

With chef-client 12.4.0:

10.98.193.143 230 unit files listed.
10.98.193.143 [2015-06-25T17:53:59+00:00] DEBUG: Provider for action enable on resource service[php-fpm] is Chef::Provider::Service::Systemd
10.98.193.143 [2015-06-25T17:53:59+00:00] DEBUG: service[php-fpm] already enabled - nothing to do
10.98.193.143 (up to date)
10.98.193.143 Recipe: my_cookbook::web_helper
10.98.193.143 * httpd_service[default] action create[2015-06-25T17:53:59+00:00] INFO: Processing httpd_service[default] action create (my_cookbook::web_helper line 42)
10.98.193.143 [2015-06-25T17:53:59+00:00] DEBUG: Providers for generic httpd_service resource enabled on node include: [Chef::Provider::HttpdService::Rhel::Systemd, Chef::Provider::HttpdService::Rhel::Sysvinit]
10.98.193.143 [2015-06-25T17:53:59+00:00] DEBUG: Provider for action create on resource httpd_service[default] is Chef::Provider::HttpdService::Rhel::Sysvinit

With chef-client 12.2.1:

10.98.192.42 231 unit files listed.
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers that refused resource service[php-fpm] were: [Chef::Provider::Service::Init, Chef::Provider::Service::Redhat]
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers that support resource service[php-fpm] include: [Chef::Provider::Service::Systemd]
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers that survived replacement include: [Chef::Provider::Service::Systemd]
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: service[php-fpm] already enabled - nothing to do
10.98.192.42 (up to date)
10.98.192.42 Recipe: my_cookbook::web_helper
10.98.192.42 * httpd_service[default] action create[2015-06-25T19:35:43+00:00] INFO: Processing httpd_service[default] action create (my_cookbook::web_helper line 42)
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers for generic httpd_service resource enabled on node include: [Chef::Provider::HttpdService::Rhel::Systemd, Chef::Provider::HttpdService::Rhel::Sysvinit]
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers that refused resource httpd_service[default] were: []
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers that support resource httpd_service[default] include: [Chef::Provider::HttpdService::Rhel::Systemd, Chef::Provider::HttpdService::Rhel::Sysvinit]
10.98.192.42 [2015-06-25T19:35:43+00:00] DEBUG: providers that survived replacement include: [Chef::Provider::HttpdService::Rhel::Systemd]

I'd be happy to provide full debug logs of the two runs via gist or the like.

Thanks for looking into this!

-Ameir

[ameir]

I pasted the log output at https://gist.github.com/ameir/1f3af9674cde1a341b95 . Let me know if you need anything else to help debug this.

[jkeiser]

I am still traveling, so I haven't read everything here, but core chef is indeed supposed to consider things with blocks to be of higher priority than the same thing without a block. So os: 'linux' do ... end is higher priority than os: 'linux'. However, platform: 'debian' is higher priority than both of them.

[coderanger]

As a short term file you can manually specify the provider:

httpd_service '...' do
  provider Chef::Provider::HttpdService::Rhel::Systemd
  ...
end

[jaym]

Can somebody sanity check:
https://github.com/chef/chef/blob/12.4.0/lib/chef/node_map.rb#L184
that should be m[:block] instead of m[:filters][:block]

[coderanger]

@jdmundrawala I concur with your reading, should be m[:block].

[jaym]

it looks like changing that fixes the issue on my machine

[lamont-granquist]

Should add a unit test to that.

Cookbook should still also get fixed to not be ambiguous.

[jaym]

Fixed in #3599

[ameir]

Thanks for the prompt support, guys! I see that chef-client and the httpd cookbook have been updated, so a big 👍 .

[lamont-granquist]

Yeah this is all fixed or getting fixed and tracked by other bugs, I don't think there's anything more to do in this ticket.