Permalink
Browse files

Avoid cases where server is restarted in Died_config_changed state (l…

…eading to lack of saved state due to version mismatch)

Summary:
A story out of OOM logs - 2 out of 10 servers I checked OOMed because they had no saved state, because they were running wrong version. This is due to a bug - `Died_config_changed` is a special sate that should only be exited by new client connecting (assuming that new client exercised all the version lookup logic, which will cause it to kill the server if version really doesn't match).

All the auto restarting logic was disregarding that and restarting the server anyway.

Reviewed By: ljw1004

Differential Revision: D12813597

fbshipit-source-id: 9ca906bd2752653436ab7be1611917a0755b7fa9
  • Loading branch information...
dabek authored and hhvm-bot committed Oct 28, 2018
1 parent c5aa612 commit be891779c2368f35b9f2010a669545bda3ebebc9
Showing with 6 additions and 0 deletions.
  1. +6 −0 hphp/hack/src/monitor/serverMonitor.ml
@@ -508,6 +508,12 @@ module Make_monitor (SC : ServerMonitorUtils.Server_config)
let max_watchman_retries = 3 in
let max_sql_retries = 3 in
match informant_report with
| Informant_sig.Move_along when env.server = Died_config_changed ->
env
| Informant_sig.Restart_server _ when env.server = Died_config_changed ->
Hh_logger.log "%s" @@ "Ignoring Informant directed restart - waiting for next client " ^
"connection to verify server version first";
env
| Informant_sig.Restart_server target_mini_state ->
Hh_logger.log "Informant directed server restart. Restarting server.";
HackEventLogger.informant_induced_restart ();

0 comments on commit be89177

Please sign in to comment.