New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Whitespace characters (new-line & tab) in environment variables #11443

Closed
zeeraw opened this Issue Mar 17, 2015 · 15 comments

Comments

Projects
None yet
3 participants
@zeeraw

zeeraw commented Mar 17, 2015

Problem

When I specify variables including escaped whitespace characters (\n, \t) I'd expect a way to have them interpreted. The way I've seen this solved in other configuration tools (eg. YAML) is to interpret backslash escaped characters in strings specified with double quotes, so that's what I've tried without success.

Reproduce

-e

When I try to pass a single variable with double quotes, I get it back escaped.

$ docker run -e FOO="hello\n\tworld" --rm busybox:latest sh -c 'echo $FOO'
hello\n\tworld

What I would expect is for the escaped characters to have been interpreted.

$ docker run -e FOO="hello\n\tworld" --rm busybox:latest sh -c 'echo $FOO'
hello
    world

--env-file

# vars.env
BAR="hello\n\tworld"

What happens is that I get the variable value, including the double quotes.

$ docker run --env-file vars.env --rm busybox:latest sh -c 'echo $BAR'
"hello\n\tworld" # interesting...

What I would expect is for the double quotes to have been ignored and the escaped characters to have been interpreted.

$ docker run --env-file vars.env --rm busybox:latest sh -c 'echo $BAR'
hello
    world

Questions

  • Is this intended behaviour?
  • Is it the responsibility of docker to interpret the escaped characters?
  • Does escaped whitespace characters belong in environment variables?

Environment details

$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): darwin/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
$ uname -a
Darwin xxx 13.4.0 Darwin Kernel Version 13.4.0: Wed Dec 17 19:05:52 PST 2014; root:xnu-2422.115.10~1/RELEASE_X86_64 x86_64
$ docker info
Containers: 90
Images: 365
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 549
Execution Driver: native-0.2
Kernel Version: 3.18.5-tinycore64
Operating System: Boot2Docker 1.5.0 (TCL 5.4); master : a66bce5 - Tue Feb 10 23:31:27 UTC 2015
CPUs: 8
Total Memory: 1.961 GiB
Name: boot2docker
ID: QSGW:FDTO:LVT2:U3RA:RUQN:JKTV:UJIW:JZE4:N2F5:MDM3:PSJC:PHVI
Debug mode (server): true
Debug mode (client): false
Fds: 80
Goroutines: 96
EventsListeners: 0
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: xxx
Registry: [https://index.docker.io/v1/]
@zeeraw

This comment has been minimized.

Show comment
Hide comment
@zeeraw

zeeraw Mar 17, 2015

Repercussions of this in languages like Ruby or Python are double escaped strings.

# Sad
$ FOO="hello\nworld" ruby -e 'puts ENV["FOO"].inspect'
# "hello\\nworld"
$ FOO='hello\nworld' ruby -e 'puts ENV["FOO"].inspect'
# "hello\\nworld"
$ FOO=hello\nworld ruby -e 'puts ENV["FOO"].inspect'
# "hellonworld"
$ FOO=hello\\nworld ruby -e 'puts ENV["FOO"].inspect'
# "hello\\nworld"

# Happy
$ FOO=$'hello\nworld' ruby -e 'puts ENV["FOO"].inspect'
# "hello\nworld"

zeeraw commented Mar 17, 2015

Repercussions of this in languages like Ruby or Python are double escaped strings.

# Sad
$ FOO="hello\nworld" ruby -e 'puts ENV["FOO"].inspect'
# "hello\\nworld"
$ FOO='hello\nworld' ruby -e 'puts ENV["FOO"].inspect'
# "hello\\nworld"
$ FOO=hello\nworld ruby -e 'puts ENV["FOO"].inspect'
# "hellonworld"
$ FOO=hello\\nworld ruby -e 'puts ENV["FOO"].inspect'
# "hello\\nworld"

# Happy
$ FOO=$'hello\nworld' ruby -e 'puts ENV["FOO"].inspect'
# "hello\nworld"
@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Mar 17, 2015

Contributor

This might be fixed by: #10431

Contributor

duglin commented Mar 17, 2015

This might be fixed by: #10431

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Mar 17, 2015

Contributor

I just checked and #10431 does fix this with one caveat.... the quotes in the env-file need to be removed otherwise you'll see them in your output. Which makes sense because when you specified them on the first command, the shell removed them before docker saw them.

/cc @tiborvass

Contributor

duglin commented Mar 17, 2015

I just checked and #10431 does fix this with one caveat.... the quotes in the env-file need to be removed otherwise you'll see them in your output. Which makes sense because when you specified them on the first command, the shell removed them before docker saw them.

/cc @tiborvass

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Mar 17, 2015

Contributor

Closing as a dup - if you think I'm mistaken please let me know.

Contributor

duglin commented Mar 17, 2015

Closing as a dup - if you think I'm mistaken please let me know.

@duglin duglin closed this Mar 17, 2015

@zeeraw

This comment has been minimized.

Show comment
Hide comment
@zeeraw

zeeraw Mar 17, 2015

@duglin Just to clarify, when loading from the env file, escaped characters will be interpreted without using quotes?

zeeraw commented Mar 17, 2015

@duglin Just to clarify, when loading from the env file, escaped characters will be interpreted without using quotes?

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Mar 17, 2015

Contributor

@zeeraw I think so. Since there's no shell involved I think its safe to think of them as having an implicit double-quote around it.

Contributor

duglin commented Mar 17, 2015

@zeeraw I think so. Since there's no shell involved I think its safe to think of them as having an implicit double-quote around it.

@zeeraw

This comment has been minimized.

Show comment
Hide comment
@zeeraw

zeeraw Mar 17, 2015

@duglin Seems legit, do the test cases in #10431 cover this too?

zeeraw commented Mar 17, 2015

@duglin Seems legit, do the test cases in #10431 cover this too?

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Mar 17, 2015

Contributor

No because #10431 is just a docker build thing. The --env-file option is for docker run.

Contributor

duglin commented Mar 17, 2015

No because #10431 is just a docker build thing. The --env-file option is for docker run.

@hudon

This comment has been minimized.

Show comment
Hide comment
@hudon

hudon May 5, 2015

I'm on Docker 1.6 which presumably includes this fix #10431 , but the --env-file + newlines bug is still present

# .env
TEST=hello\nworld
$ docker run --env-file .env --rm container  ruby -e 'puts ENV["TEST"]'
hello\nworld

I'm expecting:

hello
world

hudon commented May 5, 2015

I'm on Docker 1.6 which presumably includes this fix #10431 , but the --env-file + newlines bug is still present

# .env
TEST=hello\nworld
$ docker run --env-file .env --rm container  ruby -e 'puts ENV["TEST"]'
hello\nworld

I'm expecting:

hello
world
@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin May 5, 2015

Contributor

See if my test is similar:

$ cat e
TEST=hello\nworld
$ docker run --env-file e ubuntu bash -c 'echo -e $TEST'
hello
world

notice I had to use echo -e and not just echo to get it to interpret the \n correctly. Could you need to do something similar?

Contributor

duglin commented May 5, 2015

See if my test is similar:

$ cat e
TEST=hello\nworld
$ docker run --env-file e ubuntu bash -c 'echo -e $TEST'
hello
world

notice I had to use echo -e and not just echo to get it to interpret the \n correctly. Could you need to do something similar?

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin May 5, 2015

Contributor

Not sure if this helps or not but just messing with the shell:

$ export FOO=bar\nbar
$ echo $FOO
barnbar
$ export FOO=bar\\nbar
$ echo $FOO
bar\nbar
$ env | grep FOO
FOO=bar\nbar

I can't help but think you may need to do something in your ruby code to get it to interpret \n the way you want.

Contributor

duglin commented May 5, 2015

Not sure if this helps or not but just messing with the shell:

$ export FOO=bar\nbar
$ echo $FOO
barnbar
$ export FOO=bar\\nbar
$ echo $FOO
bar\nbar
$ env | grep FOO
FOO=bar\nbar

I can't help but think you may need to do something in your ruby code to get it to interpret \n the way you want.

@hudon

This comment has been minimized.

Show comment
Hide comment
@hudon

hudon May 5, 2015

The issue is that there is no way for docker's --env-file option to pass environment variables with newline characters in them. It is only possible to pass backslashes and n's. I want my ruby code to receive a variable that actually contains a newline, like:

$ TEST=$'hello\nworld' ruby -e 'puts ENV["TEST"]'
hello
world

Here, $'\n' evaluates the two characters \n to a newline character and passes that in.

hudon commented May 5, 2015

The issue is that there is no way for docker's --env-file option to pass environment variables with newline characters in them. It is only possible to pass backslashes and n's. I want my ruby code to receive a variable that actually contains a newline, like:

$ TEST=$'hello\nworld' ruby -e 'puts ENV["TEST"]'
hello
world

Here, $'\n' evaluates the two characters \n to a newline character and passes that in.

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin May 5, 2015

Contributor

hmm

$ TEST=$'hello\nworld' docker run -e TEST ubuntu env
...
TEST=hello
world
...
Contributor

duglin commented May 5, 2015

hmm

$ TEST=$'hello\nworld' docker run -e TEST ubuntu env
...
TEST=hello
world
...
@hudon

This comment has been minimized.

Show comment
Hide comment
@hudon

hudon May 5, 2015

Yea, --env or -e work, but not --env-file

hudon commented May 5, 2015

Yea, --env or -e work, but not --env-file

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin May 5, 2015

Contributor

docker run -e TEST=$'hello\nworld' ubuntu env too.
So, I guess the issue is just the parsing of the env file.... still digging....

Contributor

duglin commented May 5, 2015

docker run -e TEST=$'hello\nworld' ubuntu env too.
So, I guess the issue is just the parsing of the env file.... still digging....

dbaba added a commit to CANDY-LINE/docker-mysql-backup-cron that referenced this issue Jan 19, 2017

knight-of-ni added a commit to ZoneMinder/zoneminder that referenced this issue Feb 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment