How to run a ShellTask only after the upstream task was finished in prefect?

[lucascr91]

I have a Flow similar to this:

from prefect.tasks.shell import ShellTask
from mypipelines.tasks import download_ufs
from mypipelines.schedules import every_month

create_dirs = ShellTask(command="bash create_dir.sh", stream_output=True)
append_partitions = ShellTask(command="bash append_partitions.sh", stream_output=True)

with Flow("download_data") as download_data:
    create_dirs()
    download_ufs()
    append_partitions()

download_data.run_config = KubernetesRun(image=constants.DOCKER_IMAGE.value)
download_data.schedule = every_month

The download_ufs function is a regular prefect task defined with the decorator @task in a separate tasks.py file.

To test this flow, I import the flow in another file and then run flow.run()

However, the flow doesn't run in the intended order. Instead, it runs create_dirs() and append_partitions() first and only then it runs the task download_ufs.

Is there a way to force a Shell Task to behave like a regular task and follow the order established in the flow?

[zanieb]

Hi! You shouldn't count on the order of the tasks in the flow to determine their execution order. Instead, you should indicate that they are dependent on each other.

See https://docs.prefect.io/orchestration/faq/faq.html#how-do-i-set-a-task-to-be-an-upstream-task

e.g.

with Flow("download_data") as download_data:
    create = create_dirs()
    download = download_ufs(upstream_tasks=[create])
    append = append_partitions(upstream_tasks=[download])