Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

does not start at boot on CentOS 6 #235

Closed
petere opened this Issue Nov 13, 2015 · 9 comments

Comments

Projects
None yet
4 participants
@petere
Copy link

petere commented Nov 13, 2015

On CentOS 6, this cookbook registers consul as an upstart job. The consul process is started when Chef runs. I can also manually start and stop consul using something like sudo start consul. But after rebooting, consul is not automatically started. I can start it manually again, and everything is fine, but that's probably not the idea.

The configuration in /etc/init/consul.conf seems to indicate that it should automatically start at system start. But the upstart version CentOS 6 is so old, it's hard to debug anything.

Here is a small Vagrantfile to test this:

Vagrant.configure('2') do |config|
  config.vm.box = 'chef/centos-6.6'
  config.vm.provision 'chef_solo' do |chef|
    chef.add_recipe 'consul-cluster'
    chef.json = {
      'consul' => {
        'config' => {
          'enable_syslog' => true
        }
      },
      'consul-cluster' => {
        'config' => {
          'bootstrap_expect' => 1
        }
      },
    }
  end
end

After vagrant up, check that consul is running, then run vagrant reload, and it won't come back up.

@johnbellone

This comment has been minimized.

Copy link
Contributor

johnbellone commented Nov 13, 2015

What's the minor version of CentOS 6 that you're using?

@petere

This comment has been minimized.

Copy link
Author

petere commented Nov 13, 2015

6.6, as in the Vagrantfile shown above.

@johnbellone johnbellone added the bug label Dec 8, 2015

@johnbellone johnbellone added this to the 1.4 milestone Dec 8, 2015

@johnbellone johnbellone self-assigned this Dec 8, 2015

@johnbellone

This comment has been minimized.

Copy link
Contributor

johnbellone commented Jan 3, 2016

I was able to confirm that Consul isn't starting on reboot.

@legal90

This comment has been minimized.

Copy link
Collaborator

legal90 commented Jan 18, 2016

@petere FYI: it could be worked around by switching service provider to sysvinit (instead of upstart):

node.set['poise-service']['consul']['provider'] = 'sysvinit'

Or the same with poise_service_options resource:

poise_service_options node['consul']['service_name'] do
  _options['provider'] = 'sysvinit'
end
@johnbellone

This comment has been minimized.

Copy link
Contributor

johnbellone commented Jan 19, 2016

That was what I was going to suggest as well. I'm not comfortable using the version of upstart on CentOS 6 because it is ancient. I'll try and take some time to mess around with this, but this is probably the best route.

@johnbellone johnbellone removed this from the 1.4 milestone Jan 19, 2016

@petere

This comment has been minimized.

Copy link
Author

petere commented Jan 19, 2016

I had tried that as well, but it doesn't work. The first problem is that the generated init script does not pass the path (/usr/local/bin) for consul, so it doesn't start. And if you fix that (manually), consul starts, but the init script fails because it checks for the PID of /usr/bin/env. And you don't get any error messages about either of these. It become clear very quickly that that scenario is not tested very well in poise-service. You can easily reproduce this by adding the appropriate attributes to the default test in .kitchen.yml.

@johnbellone

This comment has been minimized.

Copy link
Contributor

johnbellone commented Jan 20, 2016

Alright, we can fix this by putting a custom sysvinit template into the project.

@griffint61

This comment has been minimized.

Copy link

griffint61 commented Jan 25, 2016

I saw this too, and managed to figure out what's going on. It's failing because with the upstart config on EL6, Consul is being started before the network is up, and then Consul itself bails out because, again, the network is not up. I would be nice if Consul was a little more patient in this scenario, but there's probably a way to build some patience in to the upstart config, or give it an explicit dependency on networking. Note that networking on EL6 is managed by a sysvinit script.

My workaround for now is have my wrapper cookbook add a :reboot cron job that watches for /var/lock/subsys/network, and then run "initctl start consul" if Consul is not already running.

@johnbellone

This comment has been minimized.

Copy link
Contributor

johnbellone commented Jan 27, 2016

@griffint61 That's interesting. I have a new sysvinit template that I wrote for the project, but if I configure the service to start on run levels 3 and 5 it should be fine?

Ginja added a commit to visioncritical/consul-cookbook that referenced this issue Jan 29, 2016

Fix sous-chefs#235 and force sysvinit on EL6 platform.
The upstart version in EL6 is ancient and causing problems.

Ginja added a commit to visioncritical/consul-cookbook that referenced this issue Jan 29, 2016

Fix sous-chefs#235 and force sysvinit on EL6 platform.
The upstart version in EL6 is ancient and causing problems.

Ginja added a commit to visioncritical/consul-cookbook that referenced this issue Jan 29, 2016

Fix sous-chefs#235 and force sysvinit on EL6 platform.
The upstart version in EL6 is ancient and causing problems.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.