11/02/16 12:06:18 PDT Driver Failure fork/exec permission denied

[rajkashikar]

Nomad version

Output from nomad version v0.4.0

Operating system and Environment details

RHEL 6

Issue

Adding 'user = "eifadmin"' in the job spec results in the error permission denied. If username is not defined in the job spec, all works well. I am using raw_exec driver. The binary I am using "/users/eifadmin/automation/scripts/tinywebserver2" is a simple web server in golang which opens 2 ports.

Reproduction steps

[root@eifweb-master: /etc/consul.d/jobs]nomad status yodanew
ID          = yodanew
Name        = yodanew
Type        = service
Priority    = 50
Datacenters = sjc
Status      = dead
Periodic    = false

Allocations
ID        Eval ID   Node ID   Task Group  Desired  Status
a993216e  0d7b8f2e  62ec88eb  yodanew     run      failed
a0317831  5c23849b  745412fc  yodanew     stop     complete
c6ef6ae7  37369bdd  6bd0b908  yodanew     run      failed
a2e592d6  9652a60a  1092a15f  yodanew     run      failed
af12006a  3abc911b  1092a15f  yodanew     stop     complete
[root@eifweb-master: /etc/consul.d/jobs]nomad alloc-status a993216e

couldn't retrieve stats (HINT: ensure Client.Advertise.HTTP is set): Unexpected response code: 500 (unknown allocation ID "a993216e-11e7-c132-0b5b-313d788a0bf8")
ID            = a993216e
Eval ID       = 0d7b8f2e
Name          = yodanew.yodanew[0]
Node ID       = 62ec88eb
Job ID        = yodanew
Client Status = failed

Task "yodanew" is "dead"
Task Resources
CPU  Memory   Disk     IOPS  Addresses
25   250 MiB  300 MiB  0     http: 171.68.194.79:41166

Recent Events:
Time                   Type            Description
11/02/16 12:37:57 PDT  Not Restarting  Error was unrecoverable
11/02/16 12:37:57 PDT  Driver Failure  fork/exec /users/eifadmin/automation/scripts/tinywebserver2: permission denied
11/02/16 12:37:57 PDT  Received        Task received by client
[root@eifweb-master: /etc/consul.d/jobs]

Job file (if appropriate)

job "yodanew" {
  datacenters = ["sjc"]
  type        = "service"

  task "yodanew" {
    driver = "raw_exec"

    config {
      command = "/users/eifadmin/automation/scripts/tinywebserver2"
      args    = ["${NOMAD_PORT_http}", "${NOMAD_PORT_https}", "yodanew rules"]
    }

    service {
      tags = ["nomadservice", "marvel", "urlprefix-/yodanew"]
      name = "yodanew"
      port = "http"

      check {
        type     = "http"
        path     = "/"
        interval = "60s"
        timeout  = "5s"
      }
    }

    user = "eifadmin"

    resources {
      cpu    = 25  # 500 Mhz
      memory = 250 # 256MB

      network {
        mbits = 10

        port "http" {}
      }
    }

    constraint {
      attribute = "${attr.kernel.name}"
      value     = "linux"
    }
  }
}

[dadgar]

Are you running nomad as root?

[rajkashikar]

Yes, I am running nomad as root.
rkashika@eifweb-master: ~]ps -afe|grep nomad|grep agent
root 6554 1 0 Oct22 ? 02:04:43 /usr/bin/nomad agent -config=/etc/nomad/bin/nomad.hcl

[dadgar]

Is that user defined on the host

[rajkashikar]

Yes, the user is present. I tried with both, local user and YP user. I don't think the syntax of job is a problem either. WHen I change the user="eifadmin" to user = "root", the job gets scheduled properly.
I tried with static and dynamic ports.

Here is the output of the "nomad status yodanew" command:

Recent Events:
Time Type Description
11/08/16 11:04:05 PST Not Restarting Error was unrecoverable
11/08/16 11:04:05 PST Driver Failure fork/exec /users/eifadmin/automation/scripts/tinywebserver2: permission denied
11/08/16 11:04:05 PST Received Task received by client
[root@eifweb-master jobs]# vim yodanew.hcl
[root@eifweb-master jobs]#

[rajkashikar]

Found the problem. The /var/lib/nomad needs to have permissions 777.

Thanks Dadgar to helping me out

[dadgar]

Where /var/lib/nomad is the actual nomad executable?

Do you have any insight onto why permissions there were an issue?

[rajkashikar]

/var/lib/nomad is the datadir owned by root:root. It has permissions 755, Changing it to 777 fixed it.

Nomad binary is in /etc/nomad/bin

Here is the nomad.hcl file for reference.

[rkashika@capistrano: ~/bin]cat /etc/nomad/bin/nomad.hcl
region="global"
datacenter="sjc"
log_level="WARN"
data_dir = "/var/lib/nomad"
enable_syslog = true
syslog_facility="LOCAL6"
disable_update_check = true
enable_debug = true
bind_addr = "127.0.0.1"

consul {
address = "127.0.0.1:8500"
}

advertise {
rpc = "127.0.0.1:4647"
}

client {
enabled = true
servers = [ "sjc-consul1", "sjc-consul2", "sjc-consul3" ]

options {
"driver.raw_exec.enable" = 1

[dadgar]

This has been fixed. It was due to the alloc dir not having the exec bit set!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.