Running/Managing Prefect Flows Clarification

[Vonatzki]

Use Case

Please provide a use case to help us understand your request in context

I am a bit confused on how Prefect Core would run and manage its Flows.

Based from what I read and the Medium articles by Prefect HQ, the Prefect Core library might need a bit of "help" from Prefect Cloud to manage and monitor its flows. However, there are use cases where this is not feasible (strict company policies requiring no outside communication, costs, etc.).

Aside from running multiple Python processes with flow.run() in the code, is my assumption correct that I can't really manage several Prefect flows without resorting to Prefect Cloud?

If Prefect Cloud is the only way to get these features, will there ever be a free and offline version of Prefect Cloud?

Kind of a dealbreaker for me if there is no sane way to manage several flows offline. Some tips on how to go about it in the docs might help.

Solution

Please describe your ideal solution

Documentation should be clear on this. So far, it only tells us to include flow.run() in our code for every Prefect Flow but it doesn't have a guide or instruction when you have several flows to manage.

For those exploring this tool, it places you in the "so what now?" state.

I am thinking that a crude way of doing this would be to containerize each Prefect flow and you have to create volume mapping for the logs for monitoring purposes.

Hope someone can clarify this once and for all.

[Vonatzki]

I just read about the .deploy() method found in the Prefect Cloud portion of the docs. I see that you can actually use a Docker environment to execute your Prefect Flow. But it seems it needs to communicate with Prefect Cloud online to work.

Can I use this without contacting Prefect Cloud API?

[jlowin]

Hi @Vonatzki,

Your assumption is not correct, though we are aware some of the documentation is sparse and will be shoring it up soon.

Prefect Core (this library) is an engine for defining, testing, and executing workflows. Prefect Cloud is a platform for workflow orchestration. There is no part of Prefect Core, except its Cloud client of course, that relies on Prefect Cloud to function.

This is because they are two products with very different use cases and requirements, and we do not conflate them. While we obviously think Prefect Cloud is the best way to orchestrate Prefect workflows, it's not a requirement and we're working with many companies that have happily deployed Core to production with homegrown approaches.

What makes this possible is that every component of Core has been modularized and exposed as a hook. You can use them yourself, just as we do from Prefect Cloud. Want to build a docker container of your flow? Simply instantiate a prefect.environments.storage.Docker object and call its .build() method. This will automatically serialize your flow into a container with any dependencies you specify, build the image, and push it to a private registry. If you look at the .deploy() code, you'll see that's really all the Cloud client does.

Prefect Core will happily execute all of your workflows. If you'd like an out-of-the-box orchestration solution, we'd invite you to check out Cloud's free tier when it launches. If instead you'd like to write your own system, all of Core's hooks are available to you without limitation. And if you require your orchestration system to be open-source, we recommend you downgrade to Airflow.

[Vonatzki]

Thank you for your guidance @jlowin !

One last question: Does this mean that there will be no offline web UI in the works (ala Airflow) for Prefect Core? Not that it should have one, but it would certainly help in getting buy-in.

Thank you so much for pointing out the Core's hooks, would definitely explore that for my own use.

[jlowin]

That's correct! Our UI is driven by our API, and consequently requires a connection to Cloud. In short while we'll have more docs out which will hopefully make this all more clear!