New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

entrypoint overwritten for docker containers #2373

Closed
a-nldisr opened this Issue Oct 7, 2015 · 16 comments

Comments

Projects
None yet
@a-nldisr

a-nldisr commented Oct 7, 2015

Hello,

Recently we updated from marathon 0.10.0 to 0.11.0
Today i tried to restart a Marathon / Docker config we had running earlier. The job however respawned every time as if it ran successfully. The stdout and stderr were empty and the docker containers ran for 1 second.

This is what i saw in docker ps -a

docker ps -a
CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS                           PORTS               NAMES
283a72643a36        buildserver.surfsara.nl:5000/hdfs-nn:latest   "/bin/sh -c ' '"         4 minutes ago       Exited (0) 4 minutes ago                             mesos-20150630-143338-1545979914-5050-6950-S83.7b6ca5d0-411a-4e02-aed8-20191378757a
48cbac75ad5d        buildserver.surfsara.nl:5000/hdfs-nn:latest   "/bin/sh -c ' '"         4 minutes ago       Exited (0) 4 minutes ago                             mesos-20150630-143338-1545979914-5050-6950-S83.c9be6392-552f-4750-96ef-8c6acf3a3e54

I was missing something, the part from the entrypoint i specified in the container. When i start the container from the command line i see this:

7335f29e4b78        buildserver.surfsara.nl:5000/hdfs-nn:latest   "/bin/sh -c '/opt/bin"   14 minutes ago      Exited (137) 10 minutes ago                          elated_brattain

How the entrypoint is specified in the Dockerfile:
ENTRYPOINT /opt/bin/pre-start-hadoop.sh && /usr/bin/supervisord

I suspect there is something going wrong with sanitizing the input. I can start the container when i specify the command above used in the entrypoint.

@gkleiman

This comment has been minimized.

Show comment
Hide comment
@gkleiman

gkleiman Oct 7, 2015

Contributor

Can you please add the json app definition that you get from GET /v2/apps/<app_name>?

Contributor

gkleiman commented Oct 7, 2015

Can you please add the json app definition that you get from GET /v2/apps/<app_name>?

@VFT

This comment has been minimized.

Show comment
Hide comment
@VFT

VFT Oct 12, 2015

with v0.10.0,marathon will set default cmd as that initial cmd from docker image,but v0.11.0 it will set cmd as "/bin/sh -c ' ' " if you set the "cmmond" textput to empty

so,you need to set cmd manually

VFT commented Oct 12, 2015

with v0.10.0,marathon will set default cmd as that initial cmd from docker image,but v0.11.0 it will set cmd as "/bin/sh -c ' ' " if you set the "cmmond" textput to empty

so,you need to set cmd manually

@aldipower aldipower added the bug label Oct 12, 2015

@DavidWittman

This comment has been minimized.

Show comment
Hide comment
@DavidWittman

DavidWittman Oct 14, 2015

I'm having this issue as well. It looks like when adding tasks via the UI, command is set to a single space, which is causing the entrypoint to be overridden as described here. Here's the output from GET /v2/apps:

{
    "apps": [
        {
            "acceptedResourceRoles": null,
            "args": null,
            "backoffFactor": 1.15,
            "backoffSeconds": 1,
            "cmd": " ",
            "constraints": [],
            "container": {
                "docker": {
                    "forcePullImage": false,
                    "image": "objectrocket/kibana:latest",
                    "network": "BRIDGE",
                    "parameters": [],
                    "portMappings": [
                        {
                            "containerPort": 5601,
                            "hostPort": 0,
                            "protocol": "tcp",
                            "servicePort": 10000
                        }
                    ],
                    "privileged": false
                },
                "type": "DOCKER",
                "volumes": []
            },
            "cpus": 1.0,
            "dependencies": [],
            "deployments": [],
            "disk": 256.0,
            "env": {
                "ELASTICSEARCH_URL": "<redacted>"
            },
            "executor": "",
            "healthChecks": [],
            "id": "/kibana",
            "instances": 1,
            "labels": {},
            "maxLaunchDelaySeconds": 3600,
            "mem": 512.0,
            "ports": [
                10000
            ],
            "requirePorts": false,
            "storeUrls": [],
            "tasksHealthy": 0,
            "tasksRunning": 0,
            "tasksStaged": 0,
            "tasksUnhealthy": 0,
            "upgradeStrategy": {
                "maximumOverCapacity": 1.0,
                "minimumHealthCapacity": 1.0
            },
            "uris": [],
            "user": null,
            "version": "2015-10-14T23:27:24.141Z",
            "versionInfo": {
                "lastConfigChangeAt": "2015-10-14T23:27:24.141Z",
                "lastScalingAt": "2015-10-14T23:27:24.141Z"
            }
        }
    ]
}

I'm having this issue as well. It looks like when adding tasks via the UI, command is set to a single space, which is causing the entrypoint to be overridden as described here. Here's the output from GET /v2/apps:

{
    "apps": [
        {
            "acceptedResourceRoles": null,
            "args": null,
            "backoffFactor": 1.15,
            "backoffSeconds": 1,
            "cmd": " ",
            "constraints": [],
            "container": {
                "docker": {
                    "forcePullImage": false,
                    "image": "objectrocket/kibana:latest",
                    "network": "BRIDGE",
                    "parameters": [],
                    "portMappings": [
                        {
                            "containerPort": 5601,
                            "hostPort": 0,
                            "protocol": "tcp",
                            "servicePort": 10000
                        }
                    ],
                    "privileged": false
                },
                "type": "DOCKER",
                "volumes": []
            },
            "cpus": 1.0,
            "dependencies": [],
            "deployments": [],
            "disk": 256.0,
            "env": {
                "ELASTICSEARCH_URL": "<redacted>"
            },
            "executor": "",
            "healthChecks": [],
            "id": "/kibana",
            "instances": 1,
            "labels": {},
            "maxLaunchDelaySeconds": 3600,
            "mem": 512.0,
            "ports": [
                10000
            ],
            "requirePorts": false,
            "storeUrls": [],
            "tasksHealthy": 0,
            "tasksRunning": 0,
            "tasksStaged": 0,
            "tasksUnhealthy": 0,
            "upgradeStrategy": {
                "maximumOverCapacity": 1.0,
                "minimumHealthCapacity": 1.0
            },
            "uris": [],
            "user": null,
            "version": "2015-10-14T23:27:24.141Z",
            "versionInfo": {
                "lastConfigChangeAt": "2015-10-14T23:27:24.141Z",
                "lastScalingAt": "2015-10-14T23:27:24.141Z"
            }
        }
    ]
}

@aquamatthias aquamatthias added the ready label Oct 16, 2015

@jstabenow

This comment has been minimized.

Show comment
Hide comment
@jstabenow

jstabenow Oct 17, 2015

Here an example to produce this problem:
spaces

curl marathon:8080/v2/apps/nginx

{
    "app": {
        "id": "/nginx",
        "cmd": " ",
        "args": null,
        ...

Here an example to produce this problem:
spaces

curl marathon:8080/v2/apps/nginx

{
    "app": {
        "id": "/nginx",
        "cmd": " ",
        "args": null,
        ...
@jstabenow

This comment has been minimized.

Show comment
Hide comment
@jstabenow

jstabenow Oct 17, 2015

I'm not a programmer, but that looks to me suspiciously ;-)

Marathon-UI > AppsFromModelPostProcess.js:

// Remove this hack, if there is a solution available.
// https://github.com/mesosphere/marathon/issues/2147
if (Util.isEmptyString(app.cmd)) {
   app.cmd = " ";
}

I'm not a programmer, but that looks to me suspiciously ;-)

Marathon-UI > AppsFromModelPostProcess.js:

// Remove this hack, if there is a solution available.
// https://github.com/mesosphere/marathon/issues/2147
if (Util.isEmptyString(app.cmd)) {
   app.cmd = " ";
}
@jstabenow

This comment has been minimized.

Show comment
Hide comment

Same issue?: #2147

@DavidWittman

This comment has been minimized.

Show comment
Hide comment
@DavidWittman

DavidWittman Oct 19, 2015

Yeah, this looks like a dupe.

edit: Or rather, an unintended side effect.

Yeah, this looks like a dupe.

edit: Or rather, an unintended side effect.

@abridgett

This comment has been minimized.

Show comment
Hide comment
@abridgett

abridgett Jan 20, 2016

#2147 looks done now, it looks as if we need to have the option to send this as "args" rather than "cmd" to the API (https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps). Currently AFAICT this breaks all images with an entrypoint (especially those without /bin/sh). Actually it's a bit more involved as args needs to be an array so you have to deal with splitting on whitespace, quoting etc.

#2147 looks done now, it looks as if we need to have the option to send this as "args" rather than "cmd" to the API (https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps). Currently AFAICT this breaks all images with an entrypoint (especially those without /bin/sh). Actually it's a bit more involved as args needs to be an array so you have to deal with splitting on whitespace, quoting etc.

@aquamatthias aquamatthias modified the milestones: 0.16.0, Next Jan 28, 2016

@aquamatthias

This comment has been minimized.

Show comment
Hide comment
@aquamatthias

aquamatthias Jan 28, 2016

Contributor

@pierlo-upitup can this one be closed?

Contributor

aquamatthias commented Jan 28, 2016

@pierlo-upitup can this one be closed?

@pierluigi

This comment has been minimized.

Show comment
Hide comment
@pierluigi

pierluigi Jan 28, 2016

If we intend this issue as a dupe of #2147 then, yes – that was fixed in 0.13, where the cmd field is not set to (whitespace) anymore. mesosphere/marathon-ui#460

If we intend this issue as a dupe of #2147 then, yes – that was fixed in 0.13, where the cmd field is not set to (whitespace) anymore. mesosphere/marathon-ui#460

@justinclayton

This comment has been minimized.

Show comment
Hide comment
@justinclayton

justinclayton Feb 26, 2016

I would say this should not be closed. Forcing entrypoint to /bin/sh -c for everyone is bad for a great number of reasons:

  1. it overrides the intended behavior of the container image creator
  2. it's overridden to something that does not exist in all containers
  3. it's overridden to something that does not allow for SIGTERM to be trapped by children

The most frustrating (and amusing) bit I discovered is that you can't opt-out of it. Try passing "parameters":[{"key":"entrypoint","value":"/my-original-entrypoint"}] and enjoy the lulz as you find that what it actually passed on the command line for docker run was --entrypoint=/my-original-entrypoint --entrypoint /bin/sh. Then it tosses -c to the front of the command section, which can also lead to very unexpected behavior.

Bottom line, Marathon shouldn't have a horse in this race IMHO; it should simply honor the container ecosystem's options for configuration and not force the user to jump through hoops to work around this.

I would say this should not be closed. Forcing entrypoint to /bin/sh -c for everyone is bad for a great number of reasons:

  1. it overrides the intended behavior of the container image creator
  2. it's overridden to something that does not exist in all containers
  3. it's overridden to something that does not allow for SIGTERM to be trapped by children

The most frustrating (and amusing) bit I discovered is that you can't opt-out of it. Try passing "parameters":[{"key":"entrypoint","value":"/my-original-entrypoint"}] and enjoy the lulz as you find that what it actually passed on the command line for docker run was --entrypoint=/my-original-entrypoint --entrypoint /bin/sh. Then it tosses -c to the front of the command section, which can also lead to very unexpected behavior.

Bottom line, Marathon shouldn't have a horse in this race IMHO; it should simply honor the container ecosystem's options for configuration and not force the user to jump through hoops to work around this.

@polym

This comment has been minimized.

Show comment
Hide comment
@polym

polym Feb 26, 2016

Met same problem.

root     21972  1.5  0.2 133684 15720 ?        Sl   10:29   0:00 docker -H 
unix:///var/run/docker.sock run --cpu-shares 102 --memory 33554432 
-e MARATHON_APP_VERSION=2016-02-26T15:02:10.751Z -e HOST=10.0.1.193 -e MARATHON_APP_RESOURCE_CPUS=0.1
-e MARATHON_APP_DOCKER_IMAGE=gitlab/gitlab-runner -e MESOS_TASK_ID=gitlab-ci-runner.ab1d0505-dc9d-11e5-9668-6eb5f3e16a90 
-e MARATHON_APP_RESOURCE_MEM=16.0 -e MARATHON_APP_RESOURCE_DISK=0.0 
-e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/gitlab-ci-runner 
-e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0.342b695d-7078-4c8a-9bbb-2098cd2281d9 
-v /var/run/docker.sock:/var/run/docker.sock:rw -v /etc/gitlab-runner:/srv/gitlab-runner/config:ro 
-v /usr/bin/docker:/usr/bin/docker.static:ro -v /tmp/mesos/slaves/80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0/frameworks/b0c582c1-1530-451f-95ed-537beaa5a251-0000/executors/gitlab-ci-runner.ab1d0505-dc9d-11e5-9668-6eb5f3e16a90/runs/342b695d-7078-4c8a-9bbb-2098cd2281d9:/mnt/mesos/sandbox --net bridge 

--entrypoint=/docker-entrypoint.sh --entrypoint /bin/sh

--name mesos-80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0.342b695d-7078-4c8a-9bbb-2098cd2281d9 gitlab/gitlab-runner -c run --user=root --working-directory=/home/gitlab-runner

polym commented Feb 26, 2016

Met same problem.

root     21972  1.5  0.2 133684 15720 ?        Sl   10:29   0:00 docker -H 
unix:///var/run/docker.sock run --cpu-shares 102 --memory 33554432 
-e MARATHON_APP_VERSION=2016-02-26T15:02:10.751Z -e HOST=10.0.1.193 -e MARATHON_APP_RESOURCE_CPUS=0.1
-e MARATHON_APP_DOCKER_IMAGE=gitlab/gitlab-runner -e MESOS_TASK_ID=gitlab-ci-runner.ab1d0505-dc9d-11e5-9668-6eb5f3e16a90 
-e MARATHON_APP_RESOURCE_MEM=16.0 -e MARATHON_APP_RESOURCE_DISK=0.0 
-e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/gitlab-ci-runner 
-e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0.342b695d-7078-4c8a-9bbb-2098cd2281d9 
-v /var/run/docker.sock:/var/run/docker.sock:rw -v /etc/gitlab-runner:/srv/gitlab-runner/config:ro 
-v /usr/bin/docker:/usr/bin/docker.static:ro -v /tmp/mesos/slaves/80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0/frameworks/b0c582c1-1530-451f-95ed-537beaa5a251-0000/executors/gitlab-ci-runner.ab1d0505-dc9d-11e5-9668-6eb5f3e16a90/runs/342b695d-7078-4c8a-9bbb-2098cd2281d9:/mnt/mesos/sandbox --net bridge 

--entrypoint=/docker-entrypoint.sh --entrypoint /bin/sh

--name mesos-80ea35d4-b0bb-4a17-8691-624e0491e3cb-S0.342b695d-7078-4c8a-9bbb-2098cd2281d9 gitlab/gitlab-runner -c run --user=root --working-directory=/home/gitlab-runner
@justinclayton

This comment has been minimized.

Show comment
Hide comment
@justinclayton

justinclayton Feb 26, 2016

@polym Hey, gitlab-runner is exactly what I'm trying to get working too! Nice.

So, I'm going to eat a little crow here instead of just deleting my earlier post. For others who are banging their heads against this same wall: it turns out this isn't an outstanding Marathon issue at all. At worst it's obscure and in need of clearer documentation.

If you want to preserve a container's custom entrypoint, you need to remove cmd from the top level of the app definition and add "args":[] to the "container" object.

Using cmd sets CommandInfo.shell to true when Marathon calls the Mesos containerizer, and this setting is what overrides the entrypoint to /bin/sh and prepends -c to the Docker container's CMD. Using args sets CommandInfo.shell to false, which results in simply passing args into CMD as is. The latter is probably what you want most of the time.

Pro tip: if you want to override the entrypoint yourself, you are now able to do that by passing parameters, like so:

{
  "id": "/gitlab-runner",
  "fetch": [
    { "uri": "https://path/to/custom/script/runner.sh", "executable": true }
  ],
  "container": {
    "type": "DOCKER",
    "args": [],
    "docker": {
      "image": "gitlab/gitlab-runner",
      "network": "BRIDGE",
      "parameters": [
        { "key": "entrypoint", "value": "/mnt/mesos/sandbox/runner.sh" }
      ]
    }
  }
}

I'm still having trouble with my SIGTERMs getting trapped, but at least I can now make apples to apples comparisons with my local setup.

@polym Hey, gitlab-runner is exactly what I'm trying to get working too! Nice.

So, I'm going to eat a little crow here instead of just deleting my earlier post. For others who are banging their heads against this same wall: it turns out this isn't an outstanding Marathon issue at all. At worst it's obscure and in need of clearer documentation.

If you want to preserve a container's custom entrypoint, you need to remove cmd from the top level of the app definition and add "args":[] to the "container" object.

Using cmd sets CommandInfo.shell to true when Marathon calls the Mesos containerizer, and this setting is what overrides the entrypoint to /bin/sh and prepends -c to the Docker container's CMD. Using args sets CommandInfo.shell to false, which results in simply passing args into CMD as is. The latter is probably what you want most of the time.

Pro tip: if you want to override the entrypoint yourself, you are now able to do that by passing parameters, like so:

{
  "id": "/gitlab-runner",
  "fetch": [
    { "uri": "https://path/to/custom/script/runner.sh", "executable": true }
  ],
  "container": {
    "type": "DOCKER",
    "args": [],
    "docker": {
      "image": "gitlab/gitlab-runner",
      "network": "BRIDGE",
      "parameters": [
        { "key": "entrypoint", "value": "/mnt/mesos/sandbox/runner.sh" }
      ]
    }
  }
}

I'm still having trouble with my SIGTERMs getting trapped, but at least I can now make apples to apples comparisons with my local setup.

@xiajidexue

This comment has been minimized.

Show comment
Hide comment
@xiajidexue

xiajidexue Aug 28, 2016

@justinclayton, i met the same problem using Jenkins on Mesos plugin, where should i set CommandInfo.shell to false? I did not know how to set this

xiajidexue commented Aug 28, 2016

@justinclayton, i met the same problem using Jenkins on Mesos plugin, where should i set CommandInfo.shell to false? I did not know how to set this

@ahmet2mir

This comment has been minimized.

Show comment
Hide comment
@ahmet2mir

ahmet2mir Dec 28, 2016

@justinclayton works well, but you have to use REST API, the UI (at least 1.3.6) will remove args and force cmd to " " in JSON mode

ahmet2mir commented Dec 28, 2016

@justinclayton works well, but you have to use REST API, the UI (at least 1.3.6) will remove args and force cmd to " " in JSON mode

@meichstedt

This comment has been minimized.

Show comment
Hide comment

@mesosphere mesosphere locked and limited conversation to collaborators Mar 27, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.