docker stack deploy Fails When Run From cmd.exe on Windows with CLI v23+

[twinter-amosfivesix]

Description

Trying to deploy a stack (docker stack deploy) when using version 23+ of the Docker CLI under a process started by Windows cmd.exe causes an error message similar to:

unexpected environment "=::=::\\"

This also affects docker stack config.

My System

The failure happens in Docker CLI 23.0.0 and 23.0.1. I'm using Windows 10 Enterprise 21H2 19044.2604.

The Key Part

cmd.exe has to be involved in starting the docker process in some manner. So you could run cmd.exe and then docker. Or cmd.exe and then pwsh.exe and then docker.

The Relevant Code

This appears to be the relevant code. Environment variables are read and iterated through. If one is found with a blank name/key, the error message is printed and docker aborts.

Why Does This Happen

The Old New Thing has the best explanation of this. cmd.exe basically keeps special environment variables with blank names to support old MS-DOS functionality. These are the environment variables that Docker stumbles over.

Stack Overflow question about this. Here you'll see talk of the =::=::\\ value that I commonly see.

Reproduce

1. Be using the Docker CLI version 23.0.0 or 23.0.1 on Windows.
2. Have a simple stack file named Stack.yml. See example below.
3. Start a Windows command prompt (cmd.exe) - This is key!
4. Run this: docker stack deploy -c Stack.yml Stack
5. The output is:

unexpected environment "=::=::\\"

Example Stack File:

services:
  hello:
    image: hello-world:latest

Expected behavior

The stack should deploy without error.

For comparison purposes, you can follow these steps to see the stack deploy correctly:

1. Be using the Docker CLI version 23.0.0 or 23.0.1 on Windows.
2. Have a simple stack file named Stack.yml. See example in the Reproduce section.
3. Start a PowerShell 7 session without going through cmd.exe. The reliable way I've found to do this is search for pwsh.exe from the Start menu. Using icons in the Start menu doesn't seem to work all the time.
4. Run this: docker stack deploy -c Stack.yml Stack
5. The output is:

Creating network Stack_default
Creating service Stack_hello

docker version

C:\Users\twinter>docker version
Client:
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:50:24 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:46:47 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.1
    Path:     C:\Users\twinter\.docker\cli-plugins\docker-sbom.exe

Server:
 Containers: 25
  Running: 0
  Paused: 0
  Stopped: 25
 Images: 17
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 1bis6jp8bw0eg3x73wfrq1px5
  Is Manager: true
  ClusterID: exugqkpkx2cn5bp38wcw73dxo
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.2.96.83
  Manager Addresses:
   10.2.96.83:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.18.0-425.3.1.el8.x86_64
 Operating System: Red Hat Enterprise Linux 8.7 (Ootpa)
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 7.071GiB
 Name: PRD-VIRT-RHL-56.AD.VERVEINDUSTRIAL.COM
 ID: IPGK:JLNH:EKXO:NZLH:O2O4:GQC5:PE7C:NYBC:BKQA:HNR2:Q36F:WCJ5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

Checking the Environment Like Docker CLI

Below is a simple golang program that uses the Docker CLI code to show you the environment variables that are failing Docker.

package main

import (
    "fmt"
    "os"
    "strings"
)

func main() {
  buildEnvironment(os.Environ())
}

func buildEnvironment(env []string) (map[string]string, error) {

    result := make(map[string]string, len(env))
	for _, s := range env {
		k, v, ok := strings.Cut(s, "=")
		if !ok || k == "" {
			fmt.Println("unexpected environment:")
			fmt.Println("  key:  ", k)
                        fmt.Println("  value:", v)
                        fmt.Println()
		}
		// value may be set, but empty if "s" is like "K=", not "K".
		result[k] = v
	}
	return result, nil
}

[twinter-amosfivesix]

Some related links:

• os: Environ doc comment does not describe keys containing "=" golang/go#49886
• x/build: spurious "=C:=C:\golang" in environment golang/go#49882

[twinter-amosfivesix]

Various ways I've tried to do the deploy. I'm assuming the ones that fail somehow involve cmd.exe though I don't know how to prove that beyond looking for the environment variables in question.

(wt and Terminal refer to Windows Terminal)

Start menu, type PowerShell.exe, enter -> Deploy fails
Start menu, type PowerShell 7 (x64), click "Open" -> Deploy fails
Start menu, type Windows PowerShell, click "Open" -> Deploy fails
Start menu, type pwsh.exe, enter -> Deploy OK
Start menu, type pwsh.exe, click "Open" -> Deploy OK
Start menu, type wt.ext, enter, start a PowerShell 7 profile -> Deploy OK
Start menu, type Terminal, click "Open", start a PowerShell 7 profile -> Deploy OK
Start menu, Visual Studio Code, click "Open", Terminal, New Terminal -> Deploy OK

[thaJeztah]

Thanks for your detailed report!

The Old New Thing has the best explanation of this. cmd.exe basically keeps special environment variables with blank names to support old MS-DOS functionality. These are the environment variables that Docker stumbles over.

I didn't know about these! Thanks for that info; looks like we should be able to work around this (haven't looked at the code yet though)

[twinter-amosfivesix]

One note for potential fixes: I've seen up to three different "nameless" environment variables, but not always all three. And their values can be different. And of course they have no name. So identifying the exact "MS-DOS functionality" ones would seem impossible.

[vvoland]

I checked it on my Windows installation and in general there are 2 kinds of these:

1. =ExitCode and =ExitCodeAscii which store the exitcode of the last command (former is hex format, later is ascii representation of the code).
2. Per drive working directories (=::=::\ and friends)
   There is one env variable for each separate drive letter that was accessed in the shell session and stores the working directory for that specific drive.
   The general format for these is:
   Variable name =<DRIVE_LETTER>: with a value being the path to the working directory on this drive.

A couple of examples:
=C:=C:\some\dir (key: =C:, value: C:\some\dir)
=D:=D:\some\other\dir (key: =C:, value: C:\some\dir)
=Z:=Z:\ (key: =Z:, value: Z:\)

I'm not sure how to interpret the =::=::\ (which is always present). I didn't manage to change it any way. Not sure what drive :: could actually mean.

Also, their value can be read by os.Getenv, so:
os.Getenv("=C:") => C:\some\dir
os.Getenv("=::") => ::\

A PR is on its way.

[thaJeztah]

Thanks again for the extra info! Let me leave a quick comment, as we were discussing this ticket in our standup this morning. I currently don't have a functional Windows environment to test on myself, but asked a colleague on my team team to have a look as well.

First the quick bits (without having given a close look at the code affected);

• it's a regression, and we're planning to do a 23.0.2 patch release soon with some bugfixes we already found, so we should try to have a fix into that release
• if needed, I'd be okay with a "quick fix" first, which could mean temporarily relaxing some validation if that's what gets us there, and work on a "better" fix after.

The longer bits;

• We try to avoid "too much validation" on environment variables, because even though there are documented "allowed" formats, those formats are only documented as SHOULD in the specs/RFCs. At some point we added validation to match the documented constraints, and it turned out that there's many uses of technically invalid environment variables, but software depending on them (as the kernel accepted them without error).
• So we removed most validation (and "normalizing"), and leave it up to the container / kernel to decide what it accepts (if that fails, we'll just return the error "as-is", leaving it up to the user to decide how to fix). Not ideal, but for some of these things it's the best we can do.
• At the same time, we will try to avoid common "footguns", one of them is situations where environment variable-names could be set using parameters/variables; think of $FOO=bar (where $FOO may not be set). We want to prevent those cases from being silently ignored, and were possible "fail early".

So I guess the thing to look at is;

• Distinguish user-error ($FOO=bar) from errors not controlled by the user (cmd.exe setting =A:)
• Which could mean we filter empty environment variable names from what os.Environ() gave us.
  • tricky bit there is that the user also can set variables (and use) env-vars for substituting (which would mean we can't detect the user-mistakes?)
  • we could limit some of this logic to Windows binaries only
  • (for a follow-up) look if Microsoft has official documentation on these special env-vars, so if we can make the filtering more specific
  • Ideally such logic should have a canonical implementation
  • ... which could be in a project maintained by Microsoft (such as https://github.com/microsoft/hcsshim or https://github.com/microsoft/go-winio)
  • ... or perhaps even part of Go's stdlib (should os.Environ() filter these?) or golang.org/x/sys/win
  • ^^ because I can expect other project may also run into these in some way or another, and having a canonical implementation would be good for that
• Given that these special vars are set by cmd.exe, I assume these would never be explicitly set by the user, and they would never make sense to (implicitly or otherwise) be passed on to (Windows) containers.

[thaJeztah]

let me also /cc @vvoland (on my team), and our friends at Microsoft, @kevpar @katiewasnothere (in case they have ideas 🤗)

[twinter-amosfivesix]

One thought that might go with a fix here: I'd highly recommend changing the error message from "unexpected environment ..." to "unexpected environment variable ...". When I ran into this error it took me a while to figure out that was meant by "environment". I was not doing any special environment variable stuff so that did not occur to me at first. I was trying to figure out what is a "docker environment". I know that sounds a little silly looking back. 😄 I had recently played with docker context and thought it was related to that. Then the error made me wonder if it was a YAML parsing thing. Seeing the same issue in docker stack config made me think it was. I'm not even sure now how I managed to find the cmd.exe environment variable stuff.