Nomad Restart of Unhealthy Containers

[vrenjith]

Assuming that nomad is configured with a check block while registering with consul so that Consul does a health check of the containers.
In this scenario, if consul reports that one of the containers are not healthy, will nomad restart/reschedule those containers?

[diptanu]

@vrenjith Not yet, but this is on our roadmap.

[mattcl]

Any updates on where in the roadmap this is?

[sbvitok]

Any news?

[dadgar]

Hey, no update on this quite yet. We are refactoring the way we do Consul registrations in 0.5.X. This will make it easier to add new features like this.

[dmzaytsev]

Hi,
any news?
as 0.6.0 is coming I guess

[clstokes]

Related to #164.

[alxark]

it would be great to add some option to restart some of containers but leave another one in failing state. and add timeout for restart, for example during some long term operations when container should be not available to accept client connection but should be active and killed only after deadline.

[epetrovich]

Here is a workaround.
I have an autoscaled spot fleet in AWS with nomad agents and this feature was essential for me.
Our main application is written in Java and Java Machine do not fails by itself in "some cases". It just cannot respond to the health_check on http port.

What I've done:

1. I've added an inline template to the main task to trigger restart by changes in Consul KV.
   I've used a hostname in the key. Allocation index is possible also or whatever that identifies allocation. I'm OK with hostnames as operator = "distinct_hosts"

  template {
    data = <<EOH
          last_restart:    {{ key_or_default (printf "apps/backend-rtb/backend-rtb-task/%s" (env "attr.unique.hostname")) "no_signal"  }}
          EOH
    destination   = "local/nomad_task_status"
    change_mode   = "restart"
  }  

Consul check for the main app

  service {
    tags = ["backend-rtb"]
    port = "backend"
    check {
        type     = "http",
        port     = "backend"
        path     = "/system/ping"
        interval = "2s"
        timeout  = "1s"
      }
  }

2. I wrote a very simple watcher script and added it to the task group as a second task.
   It checks the status of the main job and writes a timestamp value into consul kv that triggers a restart. It queries a consul agent on the localhost. Here is just an example of params that i've used.

    command = "python"
    args    = [ "local/watcher.py",
                "--consul_host", "127.0.0.1",
                "--node","${node.unique.name}",
                "--consul_kv_key","apps/backend-rtb/backend-rtb-task/${attr.unique.hostname}",
                "--service","backend-rtb-backend-rtb-group-backend-rtb-task",
                "--filter","passing",
                "--check_interval","15",
                "--start_after","30",
                "--fails","6"
              ]
  }

With python and python-consul that was quite simple. Any custom restart logic is possible here.

[anishmashankar]

+1
Any news on the road map for this feature?