Dynamically creating flows and how to create deployments for them

[jacksund]

TLDR

I've built up a python package that implements class-based workflows, and these workflows can be wrapped into prefect flow objects. However, with the new depolyments api, I am unsure how to register these dynamically-created workflows.

Full topic

Hi all, I've been posting around on slack and several github issues, but I thought I would open up this discussion thread to localize things. I'm sorry if this is overkill, but I could really use some help with the new deployments format.

Here's a minimal example of how my package currently works. Let's say we have an abstract base workflow class:

from prefect.flows import Flow


class AbstractClassBasedFlow:
    """
    An example abstract class that shows how you can dynamically create prefect
    flows.
    """

    factor: int = None
    extra: int = None
    
    @classmethod
    def run_config(cls, x, y):
        """
        this is what you would normally have as a function and decorate with
        prefect's @flow decorator, but here we want to access class methods and
        attributes!
        """
        a = cls.example_method(x)
        b = cls.example_method(y)
        return a + b + cls.extra

    @classmethod
    def example_method(cls, value):
        """
        a method that the 
        """
        return value ** cls.factor
    
    @classmethod
    def to_prefect_flow(cls):
        """
        dynamically creates a prefect flow for us!
        """
        return Flow(
            fn=cls.run_config,
            name=cls.__name__,
        )

Having a class like this make it really powerful to build workflows that share methods and also build out other higher-level features. Our current workflow doesn't do much, but it does show us how inheritance can become useful:

class Example1(AbstractClassBasedFlow):
    factor = 3
    extra = 3

class Example2(AbstractClassBasedFlow):
    factor = 2
    extra = 78

class Example3(AbstractClassBasedFlow):
    factor = 5
    extra = 123
    
    @classmethod
    def example_method(cls, value):
        """ example of overriding the parent method"""
        return value // cls.factor


# I've now created three "complex" but similar flows using inheritance
flow1 = Example1.to_prefect_flow()
flow2 = Example2.to_prefect_flow()
flow3 = Example3.to_prefect_flow()

# and all workflows behave as normal prefect flows
result1 = flow1(x=3, y=4)
result2 = flow2(x=3, y=4)
result3 = flow3(x=3, y=4)

# I was able to dynamically create deployments for these in prefect v1 and v2beta
# but am now not sure how. Is there any way to do this?

How would I create deployments for flow1, flow2, and flow3 shown above? These are also in modules and not lone scripts, so that adds another layer of confusion for me. This is far from the average use-case for Prefect, but I really think this approach can be super powerful and I therefore don't want to abandon the approach.

In practice, the actual Workflow class I use is located here, where there's a lot of other functionality going on -- including the automatic handling of deployments. It was easy to deploy my workflows before (using the built-in method).

[anna-geller]

@jacksund we are working on further improving the deployment UX and making it clearer how to apply it to various use cases and deployment patterns/environments; I'd recommend waiting for official recipes coming in the next days and weeks before committing to a deployment story just yet.

Flows are intended to be used as decorators - could you explain what do you mean when you say generating flows dynamically? do you mean running flows dynamically or building some factory function to avoid boilerplate for flows with similar functionality but e.g. requiring different parameters?

[jacksund]

I'd recommend waiting for official recipes coming in the next days and weeks

Hi @anna-geller. Thanks for the reply 😄 and sounds good. I'll let this sit until the next release then.

Flows are intended to be used as decorators

Yep and I always stick to using the decorator, except in these class-based flows where I create them dynamically. Here, I use prefect's Flow class in my example above (I could switch this to use the decorator but the syntax isn't as nice to read).

could you explain what do you mean when you say generating flows dynamically? do you mean running flows dynamically or building some factory function to avoid boilerplate for flows with similar functionality but e.g. requiring different parameters?

It's the latter: building some factory function to avoid boilerplate for flows with similar functionality. But it's more than "requiring different parameters" -- the functionality also handles calls to abstract methods + class attributes and also builds in higher-level functionality for each workflow. In the example I wrote above, the to_prefect_flow method is what creates the final prefect flow "dynamically".

[anna-geller]

check out response here https://discourse.prefect.io/t/we-are-still-working-on-the-deployment-ux-and-ci-cd-workflows-to-package-your-flows-and-code-depependencies-to-your-production-systems/1304

[jacksund]

thanks for putting this together -- and I am sorry my posts have been so panicked and persistent haha. I'll keep an eye on the prefect PRs and the next release before revisiting this topic. Until then, keep up the great work 🥳

[YasmineHamdy]

@anna-geller is there a way to deploy the flow using its arg name only instread of passing the script of it ??

[MiniEggz]

I put together a little deployment maker class that works at the minute but contains things that are to be deprecated (namely FlowScripts) but might be something that can be used in the interim while things are progressed/finalised.

https://github.com/MiniEggz/prefect-deployment-maker

[anna-geller]

check out response here https://discourse.prefect.io/t/we-are-still-working-on-the-deployment-ux-and-ci-cd-workflows-to-package-your-flows-and-code-depependencies-to-your-production-systems/1304

[jacksund]

This is now possible in v2.1.0. You'd use the Deployment.build_from_flow method (api docs). Big thanks to the prefect team for adding this!