CMD works incorrectly on Windows when ENTRYPOINT is specified

[dermeister0]

Description

If ENTRYPOINT is in shell format and CMD is in JSON format, ENTRYPOINT should be ignored.

Steps to reproduce the issue:

#1
FROM microsoft/nanoserver
CMD ["echo", "hi"]

#2
FROM microsoft/nanoserver
ENTRYPOINT echo START
CMD ["echo", "hi"]

Describe the results you received:

#1
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container 434d705c4988241f36530291989c3c18de0786df44d9e302e8f9e4207d63b612 encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {"ApplicationName":"","CommandLine":"echo hi","User":"","WorkingDirectory":"C:\\","Environment":{},"EmulateConsole":false,"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}.

#2
START echo hi

Describe the results you expected:
#1 returns error. It's correct.

#2 prints message. It's wrong. It should return error, because CMD is in JSON format.

Additional information you deem important (e.g. issue happens only occasionally):

https://docs.docker.com/engine/reference/builder/#cmd

The CMD instruction has three forms:
CMD ["executable","param1","param2"] (exec form, this is the preferred form)
CMD ["param1","param2"] (as default parameters to ENTRYPOINT)
CMD command param1 param2 (shell form)

Note: If CMD is used to provide default arguments for the ENTRYPOINT instruction, both the CMD and ENTRYPOINT instructions should be specified with the JSON array format.

Output of docker version:

Docker version 17.05.0-ce, build 89658be

Output of docker info:

Containers: 67
 Running: 0
 Paused: 0
 Stopped: 67
Images: 161
Server Version: 17.05.0-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 15063 (15063.0.amd64fre.rs2_release.170317-1834)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 15.91GiB
Name: ANTON-PC
ID: XOFH:ZGJZ:Q7OO:LM7Z:NQER:HW7H:B7XQ:HUWB:G5CU:HSQM:IPG5:P4A7
Docker Root Dir: D:\Docker2
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 22
 System Time: 2017-05-24T19:12:26.0469256+07:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[thaJeztah]

Looks to be specific to Windows; I tried this on a Linux version of docker-ce 17.05, and got this;

$ docker build -t foo -<<EOF
FROM busybox
ENTRYPOINT echo START
CMD ["echo", "hi"]
EOF

$ docker run --rm foo
START

I tried on older versions (docker 1.10 and up), which showed the same behaviour as above.

Personally, I wouldn't have been against allowing mixed notation, and just accept JSON arguments as arguments to the ENTRYPOINT, but given that it looks like we always ignored them (no error as well), I think we should keep that behaviour.

Adding a warning sounds good to me though, so that the user is aware of the CMD being ignored.

ping @tonistiigi @duglin @johnstep @jhowardmsft

[duglin]

I've noticed this before and I suspect we didn't do the normal "CMD overrides the args of ENTRYPOINT" because it would have involved parsing the ENTRYPOINT string and understanding things like quotes, etc. While today we have parsers for that, originally we didn't, so we were stuck with this odd behavior.

Changing it would be a breaking change, but you have to wonder who we would break since CMD is ignored today, which means people probably aren't mixing like this.

We could try to parse ENTRYPOINT (when its a string) and ignore its args IFF CMD is present, and see how that works/feels.

A warning wouldn't be bad I guess, but I'd worry people will miss it in the flood of output. I'd almost prefer to make it an error so people are forced to 'fix' their Dockerfile - since ignoring user's inputs is probably a bad thing to do in general.

[thaJeztah]

@duglin the issue here is that the behaviour on Windows looks to be different than the behaviour on Linux;

Windows:

START echo hi

vs Linux:

START

That's something that needs to be addressed at least; the warning is more a "nice to have", but separate from this issue 👍

[duglin]

True - I assumed Windows would be fixed to give the same "odd" behavior as linux :-) so I focused on the more interesting aspect of it.

[thaJeztah]

Ah, yes, it should probably get the same "odd" behavior

[dermeister0]

https://docs.docker.com/engine/reference/builder/#entrypoint

The shell form prevents any CMD or run command line arguments from being used

https://docs.docker.com/engine/reference/builder/#shell-form-entrypoint-example

You can specify a plain string for the ENTRYPOINT and it will execute in /bin/sh -c. This form will use shell processing to substitute shell environment variables, and will ignore any CMD or docker run command line arguments.

I think if ENTRYPOINT is in shell form, container build process should fail if CMD statement presents. Also docker run should display warning if ENTRYPOINT is in shell form, but arguments were provided.

[dermeister0]

https://www.ctl.io/developers/blog/post/dockerfile-entrypoint-vs-cmd/

When using ENTRYPOINT and CMD together it's important that you always use the exec form of both instructions. Trying to use the shell form, or mixing-and-matching the shell and exec forms will almost never give you the result you want.

ENTRYPOINT /bin/ping -c 3
CMD localhost

/bin/sh -c '/bin/ping -c 3' /bin/sh -c localhost

ENTRYPOINT ["/bin/ping","-c","3"]
CMD localhost

/bin/ping -c 3 /bin/sh -c localhost
ENTRYPOINT /bin/ping -c 3
CMD ["localhost"]"

/bin/sh -c '/bin/ping -c 3' localhost

ENTRYPOINT ["/bin/ping","-c","3"]
CMD ["localhost"]

/bin/ping -c 3 localhost

[jelster]

I'm trying to use the exec form of both instructions with a powershell script, and I'm finding that the exec form for ENTRYPOINT simply doesn't parse correctly at all - here's a simple DOCKERFILE to demo:

# escape=`
FROM microsoft/windowsservercore:latest
SHELL ["powershell", "-Command"]
WORKDIR c:\\scripts
COPY .\Test-Entrypoint.ps1 c:\\scripts\
ENTRYPOINT [".\Test-Entrypoint.ps1"]
CMD ["-?"]

where Test-Entrypoint.ps1 can be simply contain something like param($path) gci $path
And here's the output from a simple docker run test/entrypoint:

+ [.\Test-Entrypoint.ps1] -?
+  ~
Missing type name after '['.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordEx
   ception
    + FullyQualifiedErrorId : MissingTypename

This SO Q&A seems to suggest that "shell" form does behave as expected - something that I've been able to verify in my own testing.

[lowenna]

You're missing escaping of the backslash in ENTRYPOINT. Otherwise it's not valid JSON. So it's trying to treat it as a direct command (non-JSON) which is why the open square brace is erroring.

Also you shouldn't need to escape in the COPY

[jelster]

Hmm. I think I'm missing part of how the commands are parsed and passed to the shell. Why does the backslash need to be escaped for my example, but in the Doc man it has this example with unescaped backslashes? Is it because I'm using

# escape=`

docker docs sample:

ENTRYPOINT ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]

EDIT:
Nvm on that - I'm being silly and mixing up my slashes. So if I use the escape character ` like this:

ENTRYPOINT [".`\Test-Entrypoint.ps1"]

It will parse correctly as the powershell command .\Test-Entrypoint.ps1?

[lowenna]

No. Double backslash. This is straight JSON

[jelster]

Thanks for being patient with me and repeating yourself until I finally understood the JSON escaping, @jhowardmsft! You're right as well about there not being a need to escape the COPY command - I had kept it in from an older DOCKERFILE I had cannibalized adapted for the sample.

Having corrected that problem, I now have another question. If I've specified SHELL ["powershell", "command"], shouldn't whatever command ENTRYPOINT points to get executed using the configured SHELL, even though it's in "exec" form?

I'm seeing this (abridged for brevity), which I speculate could mean that the command Test-Entrypoint.ps1 was passed as an argument to cmd.exe:

# escape=`
FROM microsoft/windowsservercore:latest
SHELL ["powershell", "-Command"]
WORKDIR c:\scripts
COPY . c:\scripts
ENTRYPOINT [".\\Test-Entrypoint.ps1"]
CMD ["-?"]

docker run --rm test-container:latest

C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: container   
encountered an error during CreateProcess: failure in a Windows system call: 
%1 is not a valid Win32 application. 
(0xc1) extra info: {"CommandLine":".\\Test-Entrypoint.ps1 -?","WorkingDirectory":"C:\\scripts", 
"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}.

ed: change container name to avoid confusion

[lowenna]

No I don't believe so and the error confirms that. No shell was used at all, and a .ps1 file isn't an executable without PS. I need to look at code though and am OOF ATM to absolutely confirm. A JSON entrypoint will ignore the shell regardless. If you exec form the entrypoint you should get the desired behaviour you want, and the shell will be used.

[jelster]

Hmmm, thanks for confirming that. I wanted the behavior of an executable, with arguments to docker run test-container <args> being passed to the ENTRYPOINT powershell command and have been trying to figure out the right combo of syntax and structure to make it work.

I originally came down this path because I was having problems importing/invoking a custom powershell module prior to specifying ENTRYPOINT. I think that I'll try to tackle this again with my new-found knowledge and see if I can figure out where I malfed up - thanks again!

[lowenna]

Why not just add "powershell", "-command" to the entrypoint?

[jelster]

good question and something I tried; the run-time parameter that users would pass is a string looking something like AuthType=Office365;Url=https://contoso.crm.dynamics.com/;Username=someone@contose.com;Password=****, that is passed as the connection string arg to an Xrm cmdlet. Running it in the normal powershell CLI, I can use a string literal (or a variable) with no problems.

In the scenario I've been working with, I've found that unless I wrap the argument in both a ' and a " set of quotes, the parameter gets eaten and messily regurgitated down the pipeline on its way to being passed to the cmdlet.

Here's a version of the CLI command that fails to preserve the argument:

docker run --rm -it test-container:latest 'AuthType=Office365;Url=https://contoso.crm.dynamics.com;Username=jsmith@contoso.onmicrosoft.com;Password=********'

Wrapping the string with an addition set of " results in the expected end-result.

here's DOCKERFILE snippet (note that the default CMD is correctly used when nothing is provided to docker run)

ENTRYPOINT import-module .\my-powershell-module.psd1;Test-Entrypoint
CMD ["-?"]

Could this have less to do with ENTRYPOINT and more to do with how docker run parses arguments passed after the image name?

I'm gonna need to go back and re-test my approach that used ENTRYPOINT ["powershell", "-Command"] and see if the quotation issue behaves the same (as I suspect it will)...

[jelster]

Wanted to leave an update with some explorations I've made in this area. I wasn't able to isolate any specific issues with ENTRYPOINT but I have gained some insights that I thought could be useful to other people looking to add to their understanding.

I'm really happy with the pattern I've settled upon which uses the shell form of both ENTRYPOINT and CMD:

The service I'm hosting in the container requires some runtime configuration, which is achieved with a powershell script that is set as the ENTRYPOINT instruction. Thanks again @jhowardmsft for the guidance!

Once the configuration parameters have been modified, a separate Start script is invoked (by the CMD instruction) which enters an infinite loop that returns event log entries.

The key is to ensure that a trailing ; is included in the ENTRYPOINT instruction. This allows the CMD to specify execution of the start script and obviates the need to pass in additional runtime parameters (save environment variables and the like).

In other words, the pattern looks like this (note the trailing ;):

Configure

ENTRYPOINT .\Set-RuntimeConfig.ps1 -installPath $env:ContainerDir -catalogServiceUri $env:CatalogUri;

Run

CMD .\Start.ps1

No matter how I start my container, I can be confident that the necessary setup tasks have been ran. If I want to debug or perform other operations, I simply pass a command to docker run, overriding the .\Start.ps1 with, say, powershell gci .\*.dll

[lowenna]

I'll take this. Needed as part of the containerd runtime changes.

[lowenna]

Aha, the answer to this is hilarious. After scanning the moby codebase, along with containerd and runc, there's nothing explicit in any of that code to strip the "CMD" part.

It's actually shell behaviour. So

ENTRYPOINT echo hello
CMD ["world"]

will generate a container config similar to below with both Cmd and Entrypoint populated:

When a container is run from that image, the OCI spec is process bit is populated as follows:

Note both the entrypoint and the cmd are both in there.

However, if you look at ps, you'll notice "world" isn't present.

It turns out, this is entirely shell behaviour on Linux (that the "CMD" part is dropped if the ENTRYPOINT is in shell form).

Which is completely different to how command line parsing works on Windows:

So to fix this, I may be able to do a Windows-targeted fix in moby and drop the CMD part from OCI if the entrypoint is shell form. I'm already heavily all-over that code for containerd integration on Windows, but might give it a go to fix building on what I already have. However, it might be a bit contentious as I will need to store the fact that ENTRYPOINT is shell form in the config so as to know to drop the CMD part when building the OCI spec prior to passing it to the runtime. IOW, make the OCI spec so that it doesn't have the "world" part in it. However, kind of low priority to do right now.

[giggio]

It seems that if you use an entrypoint/cmd with incorrect escape sequences the Dockerfile parser will add the shell to the entrypoint entry. And the escape directive (# escape=` ) is ignored, which seems like a bug.

So this is broken:

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8
ENTRYPOINT ["dotnet", "s\w.dll" ]

It will produce this entrypoint (check with docker inspect):

            "Entrypoint": [
                "powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; [\"dotnet\", \"s\\w.dll\"]"
            ],

While this works:

FROM mcr.microsoft.com/dotnet/framework/sdk:4.8
ENTRYPOINT ["dotnet", "s\\w.dll" ]

And will produce this entrypoint:

            "Entrypoint": [
                "dotnet",
                "s\\w.dll"
            ],

[thaJeztah]

@giggio could you open a separate ticket for that?

[giggio]

Ok, done: #39779