Deadlock with podman (at least 4.5.0) and simple workflow

[jmfernandez]

A simple workflow which works both in docker and in singularity mode (example)

cwltool --singularity https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello.yml

hangs forever in podman mode

cwltool --podman https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello.yml

Expected Behavior

The workflow, when it is run in podman mode, should finish almost immediately.

Actual Behavior

The execution hangs forever, so it has to be killed through Ctrl+C or other means.

Workflow Code

It is available at https://github.com/inab/hello-workflows/tree/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl

Full Traceback

This is what appears running in debug mode and having to kill cwltool through Ctrl+C

INFO /home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/bin/cwltool 3.1.20230719185429
DEBUG [workflow ] initialized from https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl#cosifer_workflow
INFO [workflow ] start
DEBUG [workflow ] inputs {
    "an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
INFO [workflow ] starting step hello
DEBUG [step hello] job input {
    "https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl#cosifer_workflow/hello/an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
DEBUG [step hello] evaluated job input to {
    "https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-workflow.cwl#cosifer_workflow/hello/an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
INFO [step hello] start
DEBUG [job hello] initializing from https://raw.githubusercontent.com/inab/hello-workflows/227d4b7b0c925b3baf8241fecd3dd29b1d586a41/cwl/hello-tool.cwl#hello as part of step hello
DEBUG [job hello] {
    "an_input": {
        "class": "File",
        "location": "https://raw.githubusercontent.com/etc/passwd",
        "basename": "passwd",
        "nameroot": "passwd",
        "nameext": ""
    }
}
DEBUG [job hello] path mappings is {
    "https://raw.githubusercontent.com/etc/passwd": [
        "/tmp/tmpvpigbwfe",
        "/var/lib/cwl/stg87be0467-ee91-4839-a37d-b92c42a978b5/passwd",
        "File",
        true
    ]
}
DEBUG [job hello] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "bash"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "hello-world.bash"
    }
]
INFO ['podman', 'pull', 'bash:5.2.15']
Trying to pull docker.io/library/bash:5.2.15...
Getting image source signatures
Copying blob 5f1191cc3d45 skipped: already exists  
Copying blob 7264a8db6415 skipped: already exists  
Copying blob 735bc6fd85d9 skipped: already exists  
Copying config 3c04497fad done  
Writing manifest to image destination
Storing signatures
3c04497fad88ca32bf78b3447ca26c703e0d8afc9ed99e233cf575a290a86fc4
DEBUG [job hello] initial work dir {
    "_:5251f4d6-eeb1-4c82-9983-273730720695": [
        "echo \"Hello world!!!!!!!\"\necho\necho \"Computing the md5 of {\"basename\": \"passwd\", \"class\": \"File\", \"dirname\": \"/var/lib/cwl/stg87be0467-ee91-4839-a37d-b92c42a978b5\", \"location\": \"https://raw.githubusercontent.com/etc/passwd\", \"nameext\": \"\", \"nameroot\": \"passwd\", \"path\": \"/var/lib/cwl/stg87be0467-ee91-4839-a37d-b92c42a978b5/passwd\"}\"\nmd5sum /var/lib/cwl/stg87be0467-ee91-4839-a37d-b92c42a978b5/passwd\necho\necho \"Showing the inherited environment\"\nenv",
        "/pFFdxw/hello-world.bash",
        "CreateFile",
        true
    ]
}
INFO [job hello] /tmp/aeoxp4um$ podman \
    run \
    -i \
    --userns=keep-id \
    --mount=type=bind,source=/tmp/aeoxp4um,target=/pFFdxw \
    --mount=type=bind,source=/tmp/dads960j,target=/tmp \
    --mount=type=bind,source=/tmp/tmpvpigbwfe,target=/var/lib/cwl/stg87be0467-ee91-4839-a37d-b92c42a978b5/passwd,readonly \
    --workdir=/pFFdxw \
    --read-only=true \
    --log-driver=none \
    --user=10012:100 \
    --rm \
    --cidfile=/tmp/iz0wm18b/20230812023719-711370.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/pFFdxw \
    bash:5.2.15 \
    bash \
    hello-world.bash > /tmp/aeoxp4um/hello_output.txt
^CTraceback (most recent call last):
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/bin/cwltool", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/main.py", line 1452, in run
    retval = main(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/main.py", line 1298, in main
    (out, status) = real_executor(
                    ^^^^^^^^^^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/executors.py", line 61, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/executors.py", line 144, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/executors.py", line 240, in run_jobs
    job.run(runtime_context)
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 841, in run
    self._execute(runtime, env, runtimeContext, monitor_function)
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 334, in _execute
    rcode = _job_popen(
            ^^^^^^^^^^^
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 980, in _job_popen
    monitor_function(sproc)
  File "/home/jmfernandez/projects/hello-workflows/cwl/.cwlenv/lib/python3.11/site-packages/cwltool/job.py", line 859, in docker_monitor
    time.sleep(1)
KeyboardInterrupt

The issue seems to be related to the terminal (or the lack of it) when podman is run from within cwltool, or some kind of internal deadlock, because the very same podman command built by cwltool worked from a shell.

Your Environment

• cwltool version: 3.1.20230719185429
• podman version:

Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.20.5
Git Commit:   75e3c12579d391b81d871fd1cded6cf0d043550a
Built:        Sun Jun 25 00:09:49 2023
OS/Arch:      linux/amd64