FileNotFoundError occurs in python operator in docker

[kanga333]

HI.
I am running the digdag server in the docker container with the following version.

# docker --version
Docker version 1.12.6, build 78d1802

# java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15)
OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode)

# digdag --version
0.9.16

The digdag server mounts docker.sock and makes sibling containers possible.

Attempting to execute the python operator in the sibling container with the following dig file will cause FileNotFoundError.

python-with-docker.dig

timezone: UTC

_export:
  docker:
    image: python:3
    
+print_python_version:
  sh>: python --version

+hello:
  py>: scripts.myclass.MyClass.hello

myclass.py

class MyClass(object):
    def __init__(self):
        pass

    def hello(self):
        print("hello world")

log

2017-09-22 09:28:28.673 +0000 [INFO] (101622@[0:python]+python-with-docker+print_python_version) io.digdag.core.agent.OperatorManager: sh>: python --version
Python 3.6.2
2017-09-22 09:28:31.106 +0000 [INFO] (101622@[0:python]+python-with-docker+hello) io.digdag.core.agent.OperatorManager: py>: scripts.myclass.MyClass.hello
Traceback (most recent call last):
  File "<stdin>", line 12, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '.digdag/tmp/digdag-py-in-6591803356036202388.tmp'
2017-09-22 09:28:32.193 +0000 [ERROR] (101622@[0:python]+python-with-docker+hello) io.digdag.core.agent.OperatorManager: Task failed with unexpected error: Python command failed with code 1
java.lang.RuntimeException: Python command failed with code 1
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCode(PyOperatorFactory.java:153)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runTask(PyOperatorFactory.java:91)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:312)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:254)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:36)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2017-09-22 09:28:33.833 +0000 [INFO] (101622@[0:python]+python-with-docker^failure-alert) io.digdag.core.agent.OperatorManager: type: notify

Since the shell operator is working well, I guess it's a problem in running the python operator inside docker.

[hiroyuki-sato]

Hello, @kanga333

I'm not familiar Python.
This script seems work as you expected on my docker environment.
What OS are you using?

My file layout is

.
|-- README.md
|-- python-with-docker.dig
`-- scripts
    |-- __init__.py
    `-- myclass.py

See detail.

https://github.com/hiroyuki-sato/digdag-support/tree/master/core_649_python_and_docker

[hiroyuki-sato]

Oh I tested local mode. I’ll test sever mode later

[hiroyuki-sato]

Server mode also worked correctly as you expected.
https://github.com/hiroyuki-sato/digdag-support/tree/master/core_649_python_and_docker#server-mode-test

[kanga333]

Thank you for your prompt response.

I am trying to run on CoreOS 1298.7.0.

The version is slightly different from the one I actually encountered, but I prepared the reproduction environment below.
https://github.com/kanga333/issue-report/tree/master/digdag/649

Given the difference between your test and my situation, I think that digdag server is running inside docker is a condition of reproduction.

Outline drawing

               _export:
                 docker:
                   image: python:3

               +-------------+
               |             |
               |             |
           +---+----+   +----v---+
           |        |   |        |
docker run | digdag |   | python |
 +---------> server |   |        |
 |         |        |   |        |
 |         +--------+   +--------+
 |
 |       +-------------------------+
 |       |         docker          |
 |       +-------------------------+
 |       +-------------------------+
 +-------+     CoreOS(hostOS)      |
         +-------------------------+

[hiroyuki-sato]

Can you try
docker run -d -p 65432:65432 -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock digdag
instead of
docker run -d -p 65432:65432 -v /var/run/docker.sock:/var/run/docker.sock digdag

I added `-v /tmp:/tmp'

Thank you mapk0y
https://twitter.com/mapk0y/status/911900646017814528

[kanga333]

Thank you for your advice!!
By mounting /tmp it was possible to avoid errors.

As for me, this workaround has no problem.
However, since it is not a fundamental solution, I thought that the issue should remain open.
What do you think?

[yoyama]

I think this is docker in docker problem and not issue on Digdag itself.
If we have document on Digdag server on docker, this information may be useful.

[yoyama]

Close the old issue. Please reopen or file new one if you need.