Encountered error while running prefect.deployments.steps.set_working_directory - FileNotFoundError

[tekumara]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

When the flow runs, Prefect tries to set a working directory to the dir on the host, used when creating the deployment, and fails.

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 28, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/tekumara/code/prefect-demo'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 203, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory

My prefect.yaml does not include any prefect.deployments.steps.set_working_directory step:

name: prefect-demo
prefect-version: 2.11.0

# build section allows you to manage and build docker images
build: null

# the deployments section allows you to provide configuration for deploying flows
deployments:
  - name: main
    version: snapshot
    tags: ["prefect-yaml"]
    description: deployment with flow inside docker container
    entrypoint: flows/param_flow.py:param
    parameters:
      i: 1
    work_pool:
      name: kubes-pool
      job_variables:
        image: prefect-registry:5000/flow:latest
        image_pull_policy: Always
        service_account_name: prefect-flows
        finished_job_ttl: 300

I wouldn't expect this to implicitly happen if I hadn't defined a step to do so.

Reproduction

See above

Error

No response

Versions

❯ prefect version
Version:             2.11.0
API version:         0.8.4
Python version:      3.10.12
Git commit:          eeb9e219
Built:               Thu, Jul 20, 2023 4:34 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         server

Additional context

No response

[tekumara]

As I workaround I have to explicitly tell prefect to set the working dir to that of my docker image:

    pull:
    # required see https://github.com/PrefectHQ/prefect/issues/10285
    - prefect.deployments.steps.set_working_directory:
        directory: /opt/prefect

Could prefect just use the the docker image's default WORKDIR instead?