New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dsub 0.4.3 crashes on ubuntu 20.04 #219
Comments
Hi @hylkedonker, That is a very interesting error, and one that I haven't ran across before. When using the local provider, it writes a timestamp to a tmp file named 'end-time.txt' in /tmp. The error seems to indicate that that either that file is empty, or was read incorrectly. There are a few things you could do to help debug:
|
Thanks for getting in touch.
$ cat /tmp/dsub-local/echo--hylke--201221-115840-24/task/end-time.txt
$
python3 -m venv dsub_libs
source dsub_libs/bin/activate
pip3 install dsub
dsub \
--provider local \
--logging "${TMPDIR:-/tmp}/dsub-test/logging/" \
--output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \
--command 'echo "Hello World" > "${OUT}"' \
--wait I get a slightly different error:
with the stderr output:
I hope this helps. |
Hi @hylkedonker, do you have Docker installed? The Hello World example uses the Going back to your first error, I imagine looking at The error we are seeing here is certainly not helpful. In the past, I recall seeing errors about Docker not being installed. I will work on a fix about improving the error handling here. |
I do have docker installed on my system.
…On Mon, 21 Dec 2020, 18:37 Willy Nojopranoto, ***@***.***> wrote:
Hi @hylkedonker <https://github.com/hylkedonker>, do you have Docker
installed? The Hello World example uses the local provider, and installing
Docker is necessary
<https://github.com/DataBiosphere/dsub#post-installation-steps> if using
the local provider.
Going back to your first error, I imagine looking at cat
/tmp/dsub-local/echo--hylke--201220-165144-45/task/stderr.txt may also
reveal information about whether Docker is installed.
The error we are seeing here is certainly not helpful. In the past, I
recall seeing errors about Docker not being installed. I will work on a fix
about improving the error handling here.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALRD5VBYQIEEY6UHK6OMHNTSV6BV5ANCNFSM4VDFZCMQ>
.
|
I was able to get a machine with ubuntu 20.04 running, and was able to reproduce your error regardless of whether Docker was installed. The root issue is that the local provider makes a call to
|
I tried the command and was able to verify that |
I'm unable to reproduce the "no such container" error, and doing a couple searches mostly shows that its a transient issue, which isn't too helpful here. There are a couple things we could do to debug further:
|
As for the error $ cat /tmp/dsub-local/echo--hylke--201223-131919-18/task/docker.env
DATA_ROOT=/mnt/data
TMPDIR=/mnt/data/tmp
OUT=/mnt/data/output/file/tmp/dsub-test/output/out.txt
$ Output of log.txt
$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
$ Also, I haven't experienced any problems with Docker lately. No problems building, running and pushing containers. $ docker --version
Docker version 19.03.11, build dd360c7
I would be more inclined to think that some arguments may be empty, resulting in arguments being passed in the incorrect order. |
The missing docker.env file looks to be the most suspicious to me, and I'm not sure why we're seeing that. It is a little bit interesting that it is the last file I believe the expectation is that because we're using |
Thanks for the pointer. I'll try to take a closer look after the Christmas
holidays.
…On Thu, 24 Dec 2020, 00:35 Willy Nojopranoto, ***@***.***> wrote:
The missing docker.env file looks to be the most suspicious to me, and I'm
not sure why we're seeing that. It is a little bit interesting that it is
the last file dsub writes
<https://github.com/DataBiosphere/dsub/blob/master/dsub/providers/local.py#L377>
before executing the runner.sh script. I wonder if the file is still
buffered and not flushed or synced before command to run docker is executed.
I believe the expectation is that because we're using with
open(env_fname, 'wt') as f: construct, Python guarantees that the file
descriptor close method is called, which flushes the file. This should
mean that the subprocess for runner.sh should be able to read the
docker.env file, but I'm not sure...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALRD5VHL6CACEQB6EM24C43SWJ5FJANCNFSM4VDFZCMQ>
.
|
It seems that the problem is indeed related to Docker, which does not like the /tmp folder. $ docker run \
--env-file=/tmp/docker.env \
"ubuntu:14.04"
docker: open /tmp/docker.env: no such file or directory.
See 'docker run --help'. But if I copy $ docker run \
--env-file=$HOME/docker.env \
"ubuntu:14.04"
$ Solution: sudo snap remove docker and installed the apt Docker package instead (after adding the Docker apt repository) sudo apt-get install docker-ce docker-ce-cli containerd.io Now it seems to be working as expected (I am now running $ dsub \
--provider local \
--logging "${TMPDIR:-/tmp}/dsub-test/logging/" \
--output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \
--command 'echo "Hello World" > "${OUT}"' \
--wait
***WARNING: No Docker image specified. The default, `ubuntu:14.04` will be used.
***WARNING: For reproducible pipelines, specify an image with the `--image` flag.
Job properties:
job-id: echo--hylke--201230-204230-26
job-name: echo
user-id: hylke
Launched job-id: echo--hylke--201230-204230-26
To check the status, run:
dstat --provider local --jobs 'echo--hylke--201230-204230-26' --users 'hylke' --status '*'
To cancel the job, run:
ddel --provider local --jobs 'echo--hylke--201230-204230-26' --users 'hylke'
Waiting for job to complete...
Waiting for: echo--hylke--201230-204230-26.
echo--hylke--201230-204230-26: SUCCESS
echo--hylke--201230-204230-26 |
I'm happy to see you were able to get it working. I am not familiar with On our end we'll update the documentation to make this clear, and provider better error handling in the local provider. |
When I run the hello world example
it crashes as follows:
Any ideas how to get the Hello World example to work?
Versions:
dsub: 0.4.3
Ubuntu: 20.04
Python: 3.8.5
Docker: 19.03.11
Thanks,
Hylke
The text was updated successfully, but these errors were encountered: