Task using WorkingDirectory cache ends with an error: Unable to execute WorkingDirectory post actions

[anna-geller]

Explain the bug

Is cache only intended for use with Process runner - @loicmathieu? if so, we can close the issue

reproducer:

id: python_cached_dependencies
namespace: dev

tasks:
  - id: working_dir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: python_script
        type: io.kestra.plugin.scripts.python.Script
        warningOnStdErr: false
        beforeCommands:
          - python -m venv venv
          - source venv/bin/activate
          - pip install pandas
        script: |
          import pandas as pd
          print(pd.__version__)
    cache:
      patterns:
        - venv/**
      ttl: PT1H

stack trace:

Command succeed with code 0
2023-10-04 12:04:29.766Cache files changed, we update the cache
2023-10-04 12:04:29.918Unable to execute WorkingDirectory post actions
2023-10-04 12:04:29.918java.io.IOException: Is a directory
	at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at java.base/sun.nio.ch.FileDispatcherImpl.read(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
	at java.base/sun.nio.ch.FileChannelImpl.read(Unknown Source)
	at java.base/sun.nio.ch.ChannelInputStream.read(Unknown Source)
	at java.base/sun.nio.ch.ChannelInputStream.read(Unknown Source)
	at java.base/sun.nio.ch.ChannelInputStream.read(Unknown Source)
	at java.base/java.nio.file.Files.read(Unknown Source)
	at java.base/java.nio.file.Files.readAllBytes(Unknown Source)
	at io.kestra.core.tasks.flows.WorkingDirectory.postExecuteTasks(WorkingDirectory.java:326)
	at io.kestra.core.runners.Worker.handleTask(Worker.java:173)
	at io.kestra.core.runners.Worker.lambda$run$2(Worker.java:137)
	at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

changing to PROCESS runner works:

Environment Information

• Kestra Version: 0.13.0
• Operating System and Java Version (if not using Kestra Docker image):

[tchiotludo]

We must support docker runner

[loicmathieu]

Does it work? I think it has something to do with file write access on some Docker installation so it can be ignored.

[anna-geller]

thanks. I shared a reproducer, can you try it? I ran it here on the preview https://preview-oss.kestra.io/ui/executions/dev/python_cached_dependencies/16dIqspaJWkf1eDO3s8i0o/logs?level=TRACE&page=1

and it leads to the error shown in the issue body

[tchiotludo]

we don't display error if it's work for the end users, so if it work, hide the log

[loicmathieu]

I can confirm it didn't work

[loicmathieu]

I checked and this is caused by the container being run as root.
I can lower the message from ERROR to WARNING.
By the way, the same issue occurs for cleaning the temporary directory (which is not related to this task).

This is caused by the Docker daemon being run as root I think.
The python official image runs as root but I'm not sure changing that will make any difference.

[anna-geller]

Interesting, so it means that the files are properly cached and the error is only due to writing those files with the root user?

if so, it seems reasonable to change to WARNING

[loicmathieu]

Unfortunately this is when creating the cache file so caching will not work.
The file has been created as root so it cannot be run with the user running Kestra.

[anna-geller]

we don't have a solution for this atm as it's more a Docker issue than a Kestra issue - for now, we should extend task documentation for the WorkingDirectory task saying that the cache property can only be used with PROCESS runner

[walker-philips]

I am getting the same issue but have specified a PROCESS runner for all tasks. My overall Kestra deployment is using Docker with user as ROOT. Is the solution to have Kestra utilize a different system user in its docker compose to resolve the issue?

[anna-geller]

Hi @walker-philips, sorry for the late response — I think we don't know yet what's the right way to resolve the issue, that's why it's still open. We might solve it as part of the new script runner for Docker #3153

[loicmathieu]

@anna-geller I made a small adjustement by ignoring directories and files that cannot be read (like executables) and I can make it works. I'm not sure but depending on the system you may need to set the docker user to 1000 for ex to avoid permission issue.

See #3422

[anna-geller]

nice! let's keep the issue open to QA your change thoroughly first on the develop image