Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh ) #3352

Closed
keyolk opened this issue Apr 20, 2016 · 38 comments

Comments

@keyolk
Copy link

keyolk commented Apr 20, 2016

I've a application which has multi container. to easy installation, I decide to use package image which contains docker-comopse.yaml file.
what I expected is like below

$ docker run my_application install | sh
this will pull all related image

$ docker run my_application up | sh
this will do "docker-compose up" and some intial job for application using "docker-compose exec"

But I got below error

$ echo "docker-compose exec cassandra cqlsh -e 'desc keyspaces'" | sh
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 57, in main
  File "compose/cli/main.py", line 108, in perform_command
  File "compose/cli/main.py", line 353, in exec_command
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

and I confiremd that below two commands works well

$ echo "docker exec my_application_cassandra_1 cqlsh -e 'desc keyspaces'" | sh

system_traces  system

$ sh -c "docker-compose exec cassandra cqlsh -e 'desc keyspaces'"

system_traces  system

Does someone have any idea why the error comes?

below is my version info and logs from docker-compose

$ docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Sat Mar 12 19:18:57 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Sat Mar 12 19:18:57 2016
 OS/Arch:      linux/amd64
$ docker-compose version
docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013

$ sh -c "docker-compose --verbose exec cassandra cqlsh -e 'desc keyspaces'"
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'dtr.test.org', username=u'keyolk')
docker.auth.auth.parse_auth: Found entry (registry=u'test.org', username=u'$oauthtoken')
compose.cli.command.get_client: docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.3.2-0-vanilla, Os=linux, BuildTime=2016-03-12T19:18:57.542338402+01:00, ApiVersion=1.22, Version=1.10.3, GitCommit=20f81dd, Arch=amd64, GoVersion=go1.5.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (filters={u'label': [u'com.docker.compose.project=insator', u'com.docker.compose.service=cassandra', u'com.docker.compose.oneoff=False', u'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker exec_create <- (u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253', ['cqlsh', '-e', 'desc keyspaces'], tty=True, stdin=True, privileged=False, user=None)
compose.cli.verbose_proxy.proxy_callable: docker exec_create -> {u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}
compose.cli.verbose_proxy.proxy_callable: docker exec_start <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}, tty=True, socket=True)
compose.cli.verbose_proxy.proxy_callable: docker exec_start -> <socket object, fd=7, family=1, type=1, protocol=0>
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
 u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
 u'DetachKeys': u'',
 u'ExitCode': None,
 u'ID': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725',
 u'OpenStderr': True,
 u'OpenStdin': True,
 u'OpenStdout': True,
 u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
                    u'entrypoint': u'cqlsh',
...
compose.cli.verbose_proxy.proxy_callable: docker exec_resize <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}, width=190, height=42)
                                                                                                                                                                    compose.cli.verbose_proxy.proxy_callable: docker exec_resize -> None

system_traces  system

compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
 u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
 u'DetachKeys': u'',
 u'ExitCode': 0,
 u'ID': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725',
 u'OpenStderr': True,
 u'OpenStdin': True,
 u'OpenStdout': True,
 u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
                    u'entrypoint': u'cqlsh',
...
echo "docker-compose --verbose exec cassandra cqlsh -e 'desc keyspaces'" | sh                                   [2/22580]
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'dtr.test.org', username=u'keyolk')
docker.auth.auth.parse_auth: Found entry (registry=u'test.org', username=u'$oauthtoken')
compose.cli.command.get_client: docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.3.2-0-vanilla, Os=linux, BuildTime=2016-03-12T19:18:57.542338402+01:00, ApiVersion=1.22, Version=1.10.3, GitCommit=20f81dd, A$ch=amd64, GoVersion=go1.5.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (filters={u'label': [u'com.docker.compose.project=insator', u'com.docker.compose.service=cassandra', u'com.docker.compose.oneo$f=False', u'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker exec_create <- (u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253', ['cqlsh', '-e', 'desc keyspaces'], tty=True, stdin=True$ privileged=False, user=None)
compose.cli.verbose_proxy.proxy_callable: docker exec_create -> {u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}
compose.cli.verbose_proxy.proxy_callable: docker exec_start <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}, tty=True, socket=True)
compose.cli.verbose_proxy.proxy_callable: docker exec_start -> <socket object, fd=7, family=1, type=1, protocol=0>
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
 u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
 u'DetachKeys': u'',
 u'ExitCode': None,
 u'ID': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01',
 u'OpenStderr': True,
 u'OpenStdin': True,
 u'OpenStdout': True,
 u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
                    u'entrypoint': u'cqlsh',
...
compose.cli.verbose_proxy.proxy_callable: docker exec_resize <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}, width=190, height=42)
compose.cli.verbose_proxy.proxy_callable: docker exec_resize -> None
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 57, in main
  File "compose/cli/main.py", line 108, in perform_command
  File "compose/cli/main.py", line 353, in exec_command
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1
@kajmagnus
Copy link

kajmagnus commented May 8, 2016

I'm having the same problem, when piping docker-compose output via gzip to a file. When I run the command myself in Bash, it works just fine. However, as a cron job, I'm getting the same error as you — and the same stack trace file & line numbers & function names, as you, and return code -1.

Here's the command:

/usr/local/bin/docker-compose exec postgres pg_dumpall --username=postgres \
   | gzip > $postgres_backup_path

(I have that line in a backup script that I run sometimes manually (works fine) and sometimes as a cron job (won't work)).


Now I've worked around my problem by calling Docker directly like so:

/usr/bin/docker exec edm_postgres_1 pg_dumpall --username=postgres \
    | gzip > $postgres_backup_path

— works fine from both Cron and manually in Bash.

@tkaefer
Copy link

tkaefer commented May 20, 2016

I do have the same issue when using docker-compose exec in a cronjob:

something like

cd ~/piwik && docker-compose exec piwik sudo -u www-data /usr/bin/php5 /app/console core:archive --url=https://piwik-host/ > ~/piwik-archive.log

does not work - neither as a one-liner in the crontab, nor as a shell script called from crontab

I could create a workaround with bash script like:

#!/bin/bash
piwikid=$(docker ps -f "ancestor=piwik" -q)

/usr/bin/docker exec ${piwikid} /usr/bin/php5 /app/console core:archive --url=https://piwik-host/ > ~/piwik-archive.log

@michaelarnauts
Copy link

michaelarnauts commented May 25, 2016

I could get it to work by adding the -T parameter to not create a Pseudo-TTY.

docker-compose exec -T container_name ./build.sh for example.

@yannisc
Copy link

yannisc commented Jul 15, 2016

The -T option worked for me, but the script doesn't return to command prompt even after it finishes the last command.

@knaperek
Copy link

knaperek commented Oct 1, 2016

I'm having the same issue when running:
docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content

Everything seems to work after all (the command executes as expected), it's just that nasty crash that docker-compose gives me afterwards:

$ docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 61, in main
  File "compose/cli/main.py", line 113, in perform_command
  File "compose/cli/main.py", line 441, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo $?
255

When I use the -T flag as recommended though, my command doesn't work and I even get stuck on the docker-compose command (doesn't return to my shell but just hangs instead).

@zhao-ji
Copy link

zhao-ji commented Nov 13, 2016

Traceback (most recent call last):
File "", line 3, in
File "compose/cli/main.py", line 62, in main
File "compose/cli/main.py", line 114, in perform_command
File "compose/cli/main.py", line 442, in exec_command
File "site-packages/dockerpty/pty.py", line 334, in start
File "site-packages/dockerpty/pty.py", line 370, in _hijack_tty
File "site-packages/dockerpty/io.py", line 164, in do_write
OSError: [Errno 32] Broken pipe
docker-compose returned -1

@paolomainardi
Copy link

paolomainardi commented Nov 28, 2016

I have the same problem, but just by launching this from jenkins (it fails just there):
docker-compose exec drupal bash -c "bin/behat --profile=ci", now changed to docker exec ${COMPOSE_PROJECT_NAME}_drupal_1 bash -c "bin/behat --profile=ci".

@noamelf
Copy link

noamelf commented Jan 4, 2017

Same here. docker-compose exec fails when running as part of a Jenkins job (works locally), but docker exec works. Thanks @paolomainardi !

@michaelarnauts
Copy link

Have you tried adding the -T flag to docker-compose exec?

@noamelf
Copy link

noamelf commented Jan 4, 2017

Tried it now, adding -T to docker compose exec solve this for me as well.

@TheLQ
Copy link

TheLQ commented Jan 9, 2017

-T isn't working for me. My only workaround is to use docker exec directly but means I have to parse the container ID from docker ps which isn't ideal.

To test I have a basic python script that just prints stdin

$ cat myScript.py
#!/usr/bin/python
import sys
print "started"
for line in sys.stdin:
    print line

$ echo test | docker-compose exec -T myContainer /usr/src/myScript.py
.. nothing happens, just waits ..

$ echo test | docker-compose exec myContainer /usr/src/myScript.py
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 65, in main
  File "compose/cli/main.py", line 117, in perform_command
  File "compose/cli/main.py", line 462, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo test | docker exec -i d6380314bb89  /usr/src/myScript.py
started
test

@michaelarnauts
Copy link

michaelarnauts commented Jan 9, 2017 via email

@jsoriano
Copy link
Contributor

jsoriano commented Jan 9, 2017

I'd say that the problem is that -T in docker-compose exec is mostly equivalent to not using any of -t or -i flags in docker run, so with -T there is not tty but also not attached stdin, and it seems that there is no way of attaching stdin without tty in docker-compose.

docker-compose exec should probably have different flags for tty and for attaching stdin as docker run has.

@dylanninin
Copy link

dylanninin commented Feb 7, 2017

env: ansible playbook + ubuntu(vagrant)

error:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 65, in main
    command()
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 117, in perform_command
    handler(command, command_options)
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 462, in exec_command
    pty.start()
  File "/usr/local/lib/python2.7/dist-packages/dockerpty/pty.py", line 338, in start
    io.set_blocking(pump, flag)
  File "/usr/local/lib/python2.7/dist-packages/dockerpty/io.py", line 32, in set_blocking
    old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)

add -T fix the issue.

@endzyme
Copy link

endzyme commented Mar 6, 2017

for all those curious about getting around this while still getting information from docker-compose...

docker exec -i $(docker-compose ps -q cassandra) < someexample.cql

The -i keeps stdin open while not allotting a tty and the ps -q on the docker-compose provides us with the container ID we want.

I assume this will provide multiple IDs if you have many running, you'll have to do some munging to use this method.

It would appear docker-compose isn't implicitly using a -i when passing the exec along.

@OnkelTem
Copy link

OnkelTem commented Mar 10, 2017

Just want to leave this minimal test-case for docker-compose:

echo 123 | docker-compose exec container_name tee

It triggers error:

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 88, in main
  File "compose/cli/main.py", line 140, in perform_command
  File "compose/cli/main.py", line 486, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose

@paolomainardi
Copy link

I can reproduce this bug just when docker-compose gets called from jenkins or gitlab-ci

@smakhtin
Copy link

Same here in Jenkins.

@tnguyen14
Copy link

What is docker-compose -q app?

@vshih
Copy link

vshih commented Sep 14, 2017

I think he means you can use docker-compose ps -q {app} to get the container ID. For example,

docker exec $(docker-compose ps -q {app}) bash -c ...

@Hubbitus
Copy link

docker-compose exec container_name cat logfile > logfile

also fails for me, but with -T

docker-compose exec -t container_name cat logfile > logfile

it just produce empty file, so no any output! Even there no error, it is unusable.

felixhummel pushed a commit to felixhummel/docker-etherpad-lite that referenced this issue Sep 20, 2017
felixhummel pushed a commit to felixhummel/kimai-in-docker that referenced this issue Sep 20, 2017
@quolpr
Copy link

quolpr commented Nov 15, 2017

So, as I understand, docker exec is the solution for now?

@arno01
Copy link

arno01 commented Nov 22, 2017

echo 123 | docker exec -i $(docker-compose ps -q service) cat - for now...

lightster added a commit to watchpoint-io/watchpoint-docker that referenced this issue Jan 8, 2018
`docker-compose exec` throws an exception after running a command if
`docker-compose` is ran via a cron. A workaround is to run the command
via `docker exec` instead.

docker/compose#3352
Schnitzel added a commit to uselagoon/lagoon that referenced this issue Jan 15, 2018
@shin- shin- added this to the 1.19.0 milestone Jan 19, 2018
@taragurung
Copy link

I would add a bit of changes for someone who have a container-name set .Based on @michaelarnauts
This worked for me, docker-compose exec -T app ./build.sh where app is the service defined in the docker-compose

ryanprior added a commit to ryanprior/conjur-puppet-demo that referenced this issue Feb 8, 2018
This prevents alloction of a pseudo-TTY when it is not necessary, as suggested
in docker/compose#3352
bennadel added a commit to bennadel/DailyPrime.me that referenced this issue Apr 17, 2018
While the `renew_certificate.sh` script worked fine when I ran it manually, it
didn't appear to be running via the `crontab` (based on the lack of StatsD
metrics going to my DataDog agent). After piping the output of the `crontab` to
a file, I found two issues. First, the output was:

> the input device is not a TTY

This was caused by the `-it` options in my `docker run` command. The `-t`
requires a TTY interface to be allocated; which, is not allocated when running
via `crontab`.

Once I removed the `-it` options, I was faced with a second error in the log
output:

```
Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 124, in perform_command
  File "compose/cli/main.py", line 467, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose
```

Apparently, this is a known issue with `docker-compose` being run inside a
script, like `crontab`. To work around this, adding the `-T` option tells
docker not to allocate a Pseudo-TTY for `docker-compose`.

Source: https://stackoverflow.com/questions/43099116/the-input-device-is-not-a-tty
Source: docker/compose#3352
hemberger added a commit to smrealms/dockerize that referenced this issue May 5, 2018
Without it, the following error occurs when run with cron:

ValueError: file descriptor cannot be a negative integer (-1)

See docker/compose#3352.
@bscheshirwork
Copy link

now I try in cron (again)
crontab -e

*/10 * * * * time -p /usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec php ./yii command

now I have same problem in cron (again)

[17362] Failed to execute script docker-compose
Traceback (most recent call last):
  File "site-packages/dockerpty/pty.py", line 334, in start
  File "site-packages/dockerpty/pty.py", line 367, in _hijack_tty
  File "site-packages/dockerpty/io.py", line 59, in select
  File "site-packages/dockerpty/io.py", line 351, in fileno
  File "site-packages/dockerpty/io.py", line 103, in fileno
  File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 519, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
  File "site-packages/dockerpty/io.py", line 351, in fileno
  File "site-packages/dockerpty/io.py", line 103, in fileno
  File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.
Command exited with non-zero status 255
real 2.09
user 0.98

after add into crontab -e

COMPOSE_INTERACTIVE_NO_CLI=1
# Edit this file to introduce tasks to be run by cron.

https://stackoverflow.com/a/10657111/7670492

I try again with -T

/usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec -T php ./yii command

success

Code0x58 added a commit to Code0x58/ops-examples that referenced this issue Jul 19, 2018
lwander added a commit to lwander/community that referenced this issue Feb 19, 2019
See the [helm docs](https://docs.docker.com/v17.09/compose/reference/exec/) for the `-T` flag explanation.

When running this as a startup-script as shown in the [Mender solution doc](https://cloud.google.com/community/tutorials/cloud-iot-mender-ota) we run into [this error](docker/compose#3352). Disabling pseudo-tty creation fixes it.
ToddKopriva pushed a commit to GoogleCloudPlatform/community that referenced this issue Feb 22, 2019
See the [helm docs](https://docs.docker.com/v17.09/compose/reference/exec/) for the `-T` flag explanation.

When running this as a startup-script as shown in the [Mender solution doc](https://cloud.google.com/community/tutorials/cloud-iot-mender-ota) we run into [this error](docker/compose#3352). Disabling pseudo-tty creation fixes it.
szschaler added a commit to automated-assessment/nexus_monorepo that referenced this issue Dec 12, 2019
ederuiter added a commit to thisisdevelopment/laravel-base-dev that referenced this issue Mar 11, 2020
merc1er added a commit to merc1er/bch-toolkit that referenced this issue Oct 5, 2020
tobybellwood pushed a commit to uselagoon/lagoon-service-images that referenced this issue Mar 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests