eye ignores pid change and overwrites again with 'invalid' old pid

[hamann]

Hi!

We're monitoring/controlling haproxy instances with eye. The restart_command should do a so called 'soft restart' ('/usr/sbin/haproxy -D -f /etc/haproxy/haproxy_test.conf -sf {PID}'), which works so far and haproxy itself updates (as expected) the pidfile with the new pid. But then eye ignores the new pid entry and rewrites the pid file with the old instance pid. That's really bad, because if the old haproxy instance stops (when all connections on the old instance are closed), eye will restart haproxy again and again

proxy/haproxy_test.pid) changes by itself (pid:6751) => (pid:7118), not under eye control, so ignored
08.04.2014 17:10:51 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file(/var/run/haproxy/haproxy_test.pid) changes by itself (pid:6751) => (pid:7118), not under eye control, so ignored
08.04.2014 17:10:56 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file(/var/run/haproxy/haproxy_test.pid) changes by itself (pid:6751) => (pid:7118), > 120 ago, so rewrited (even if pid_file not under eye control)

Is it possible to add an option so that eye doesn't ignore a pid change?

according eye config:

# vi: set ft=ruby :

Eye.load("/etc/eye/root/config.rb")

Eye.application "haproxy_test" do
  working_dir "/"

  process "haproxy_test" do
    start_command '/usr/sbin/haproxy -D -f /etc/haproxy/haproxy_test.conf'
    stop_command '/bin/kill -SIGUSR1 {PID}'
    restart_command '/usr/sbin/haproxy -D -f /etc/haproxy/haproxy_test.conf -sf {PID}'
    pid_file '/var/run/haproxy/haproxy_test.pid'
    stdall '/var/log/haproxy.log'
  end
end

[kostya]

it looks like, that oldpid 6751 still alive, after restart-command, but eye wait when it die (2 minutes), like unicorn do.

[hamann]

Yes, the old pid is still alive. But why does it ignore the new pid entry and rewrites it?

[kostya]

Little unexpected case when pid_file changes and old process not die. Because eye dont know why pid_file was rewritten, may it just some fuckup.

[hamann]

When doing a soft restart, there will be two haproxy instances. The old instance exists as long as there are clients connected and kills itself if all clients are disconnected. In our case that could take several minutes or hours. Eye should monitor the new instance in the meantime

[kostya]

i guess, i will add option for that case.

[hamann]

👍

[hamann]

Thanks!

[kostya]

Can you test it with 0.6.pre?

[hamann]

18.04.2014 21:06:21 INFO  -- [haproxy_test:haproxy_test] <= restart
18.04.2014 21:06:26 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file (/var/run/haproxy/haproxy_test.pid) changed by itself (<1765> => <1931>), ignoring self-managed pid change
18.04.2014 21:06:31 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file (/var/run/haproxy/haproxy_test.pid) changed by itself (<1765> => <1931>), ignoring self-managed pid change
18.04.2014 21:06:36 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file (/var/run/haproxy/haproxy_test.pid) changed by itself (<1765> => <1931>), ignoring self-managed pid change
18.04.2014 21:06:41 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file (/var/run/haproxy/haproxy_test.pid) changed by itself (<1765> => <1931>), ignoring self-managed pid change
18.04.2014 21:06:46 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file (/var/run/haproxy/haproxy_test.pid) changed by itself (<1765> => <1931>), ignoring self-managed pid change
18.04.2014 21:06:51 WARN  -- [haproxy_test:haproxy_test] check_alive: pid_file (/var/run/haproxy/haproxy_test.pid) changed by itself (<1765> => <1931>), trusting this change, and now monitor <1931>

[hamann]

Nice! So you decided to go with that default?

[kostya]

yes this is default, because unicorn works with it well, and why not,
you can set grace to update pid with option auto_update_pidfile_grace, which is default 30s;