Storage Wars

[cicdw]

Thanks for contributing to Prefect!

Please describe your work and make sure your PR:

• adds new tests (if appropriate)
• updates CHANGELOG.md (if appropriate)
• updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

Note that your PR will not be reviewed unless all three boxes are checked.

What does this PR change?

This PR builds off @joshmeek 's work and beefs up our Storage classes as follows:

• adds PIN 7 (closes Proposal PIN: Storage & Execution #921 )
• adds a new Memory storage option for local testing
• implements a new and expanded Storage interface for storing multiple flows, and accessing flows contained in Storage
• adds storage submodule to our exposed API documentation

I also removed a few things from the initial implementation for the sake of simplicity / exposing a cleaner interface until we know how we want these things to be used.

Assuming this PR is approved and merged, it would be very easy from here to implement a LocalEnvironment and a DaskClusterEnvironment using this interface. I avoided taking it that far because it's already a large PR.

For the full details of what this PR is trying to achieve, please read the PIN-7 that is included.

[codecov]

Codecov Report

Merging #956 into master will decrease coverage by 0.31%.
The diff coverage is 83.47%.

[joshmeek]

This was also a really good idea!

[joshmeek]

What's the intention behind removing the entire full dockerfile option? Too many extra required steps i.e. healthcheck, copying flows, etc...?

[cicdw]

So the intent was to simplify the interface for now, and the only difference between create_dockerfile_object_from_dockerfile and the current method is that the current method's docker file is:

FROM {base_image}

# DO pip stuff

so I figured when we need this feature we can easily generalize our current method (and avoid having two different methods).

[joshmeek]

This is awesome!

[jlowin]

I think this looks great!

One suggestion is perhaps to change the signature of get_flow_location(flow: Flow) to get_flow_location(flow_name: str). The reason is that in almost all circumstances, get_flow_location will be called from a different process than the one that built the storage, and so the Flow object will be necessarily different (in a "Python ID" sense) than the one that was stored. Therefore a string attribute like its name or some other key will ultimately be the only way to identify it to the Storage. (Memory seems to be an exception to this rule, but would be compatible with using the name as a key instead of the flow itself). Otherwise this will require instantiating a Flow object just to use Storage which is supposed to contain the Flow... and that feels circular.

[cicdw]

@jlowin happy to update the signature of that method. Just for the record, the only reason I left it like that was in case new types of storage relied on other permanent attributes of Flows other than name, but I do think it's better to keep it simple for now.

[jlowin]

This is such a clean implementation and advances so many of our goals that we thought would take longer, so let's start with the simpler version to get off the ground

[cicdw]

@jlowin and @joshmeek : I've addressed all of your feedback and updated Docker storage to serialize its flows attribute. That way, knowledge of containment survives serialization of flows / storage objects without any additional work! Additionally, all the agent will have to do when it picks up a flow run is something like:

environment = flow.environment
storage = flow.storage
environment.setup(storage)
environment.execute(storage, storage.flows[flow.name])

[cicdw]

Actually, writing that makes me wonder whether get_flow_location is even a necessary method at this point, since the flows attribute has been elevated. Thoughts?