rkt driver requires iptables on PATH

[blalor]

Nomad version

Nomad v0.4.0-dev ('bc09a0444722617a3a0ee0daa28d24b93d9d3e5b+CHANGES')

Operating system and Environment details

• CentOS Linux release 7.2.1511 (Core)
• uname -a: Linux localhost 4.6.2-1.el7.elrepo.x86_64 Pass agent configuration into fingerprinter #1 SMP Wed Jun 8 14:49:20 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

This kernel comes from kernel-ml.

rkt version:

rkt Version: 1.8.0
appc Version: 0.8.4
Go Version: go1.6.2
Go OS/Arch: linux/amd64
Features: -TPM

Issue

The rkt driver does not preserve an appropriate $PATH to allow rkt to find iptables, which is required by the default stage1 for the default network config. As a result, rkt as executed by the nomad executor fails, but the same command executed by the user outside of nomad succeeds.

Reproduction steps

Run the below job config. Get the allocation. Rummage around in nomad fs <alloc_id> until you find the logs you need (which is totally nonintuitive).

Nomad Client logs

nomad fs <alloc_id> alloc/logs/redis.stderr.0

image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.8.0
image: using image from local store for url docker://redis:latest
networking: loading networks from /etc/rkt/net.d
networking: loading network default with type ptp
stage1: failed to setup network: failed to locate iptables: exec: "iptables": executable file not found in $PATH

nomad fs <alloc_id> redis/redis-executor.out

2016/06/16 13:47:30 [DEBUG] executor: launching command /usr/bin/rkt run --volume=redis,kind=host,source=/tmp/NomadClient985935330/dff7f32e-5075-0bb2-3197-84db399ce5f8/alloc --mount=volume=redis,target=/tmp/NomadClient985935330/dff7f32e-5075-0bb2-3197-84db399ce5f8/alloc docker://redis:latest --insecure-options=all --set-env=NOMAD_ALLOC_NAME=example.cache[0] --set-env=NOMAD_ALLOC_DIR=/tmp/NomadClient985935330/dff7f32e-5075-0bb2-3197-84db399ce5f8/alloc --set-env=NOMAD_ALLOC_ID=dff7f32e-5075-0bb2-3197-84db399ce5f8 --set-env=NOMAD_IP_db=127.0.0.1 --set-env=NOMAD_ALLOC_INDEX=0 --set-env=NOMAD_TASK_NAME=redis --set-env=NOMAD_TASK_DIR=/tmp/NomadClient985935330/dff7f32e-5075-0bb2-3197-84db399ce5f8/redis/local --set-env=NOMAD_CPU_LIMIT=500 --set-env=NOMAD_MEMORY_LIMIT=256 --set-env=NOMAD_ADDR_db=127.0.0.1:41325 --set-env=NOMAD_PORT_db=41325 --memory=268435456M --cpu=500m
2016/06/16 13:47:30 [INFO] executor: registering services
2016/06/16 13:47:30 [INFO] executor: de-registering services and shutting down consul service

Job file

job "example" {
    datacenters = ["dc1"]

    constraint {
        attribute = "${attr.kernel.name}"
        value = "linux"
    }

    update {
        stagger = "10s"
        max_parallel = 1
    }

    group "cache" {
        restart {
            attempts = 10
            interval = "5m"
            delay = "25s"
            mode = "delay"
        }

        task "redis" {
            driver = "rkt"

            config {
                image = "docker://redis:latest"
            }

            service {
                name = "${TASKGROUP}-redis"
                tags = ["global", "cache"]
                port = "db"
                check {
                    name = "alive"
                    type = "tcp"
                    interval = "10s"
                    timeout = "2s"
                }
            }

            resources {
                cpu = 500 # 500 Mhz
                memory = 256 # 256MB
                network {
                    mbits = 10
                    port "db" {
                    }
                }
            }
        }
    }
}

Workaround

Replace /usr/bin/rkt with a wrapper script that execs the real rkt binary. Wrapper contents:

#!/bin/bash

echo "==> starting at $( date )"
env

echo "path before: ${PATH}"
export PATH=${PATH}:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
echo "path after: ${PATH}"

exec /usr/local/bin/rkt-real "${@}"

Non-default network config for rkt may also be a solution. It doesn't appear that the Nomad rkt driver supports that type of config in the task at this time.

/cc @jrg72

[schmichael]

This was fixed in 0.4.1. Sorry for being so slow to close!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.