Embedding config into context

[cicdw]

Thanks for contributing to Prefect!

Please describe your work and make sure your PR:

• adds new tests (if appropriate)
• updates CHANGELOG.md (if appropriate)
• updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

Note that your PR will not be reviewed unless all three boxes are checked.

What does this PR change?

This PR is subtle. Superficially, it converts most references of prefect.config to prefect.context.config via embedding the config object into context. In practice, however, it does something deeper: whatever Prefect-specific configuration options are set on the main Flow Runner process through environment variables will be respected on all machines that Tasks are submitted to run on. This is accomplished by embedding config into prefect.context, which is always kept up-to-date with the context present in the main Flow Runner process. Note that:

• set_temporary_config still works for unit-testing because dictionaries are tracked by-reference in Python
• this introduces some hidden "gotchas" when referencing config during execution that all developers should be aware of (this isn't something users will encounter, but developers might)

Why is this PR important?

Many Cloud users ran into the following problem: executing Flows against external Dask clusters required all dask workers to have Prefect Cloud specific environment variables set. This is unsatisfactory, as Prefect should handle those execution details internally. This PR ensures that as long as all workers have an up-to-date version of Prefect, no Prefect-specific configurations need to be set on the workers.

Notes for reviewers

@joshmeek / @jlowin : I've tested this using local agents in various forms, and verified that my local setup failed without this PR. Note that, at worst, it preserves current behavior as the first config to populate context will persist. That being said, please review this carefully as it is a somewhat deep change.

cc: @bnaul @joeschmid

[codecov]

Codecov Report

Merging #1481 into master will increase coverage by 0.05%.
The diff coverage is 100%.

[jlowin]

because the logging hooks in Prefect are created / called prior to the actual byte-code payload being deserialized on the worker

Feels like we could explore configuring logging handlers in a context manager, possibly even config itself.

with prefect.utilities.logging.configure_logging():
    do_stuff()
do_stuff_with_regular_logging()

This would solve some of my other frictions from last week, so it's on my mind

[jlowin]

I think this is a very important change and it's amazing how short it is (other than the prefect.config -> prefect.context.config updates)

I have a couple comments on how the new config is loaded into the existing one (which might break the set_temporary_config because they make a more explicit copy, not sure). Depending on what we want to do there (if anything) I'll give a second more complete review.

[joeschmid]

👏 Love this change! Much better experience for Prefect Cloud users running their own external Dask clusters.

[jlowin]

Two minor suggestions and a comment that doesn't require any action IMO.

Question: should we remove config from the top level of Prefect? Currently prefect.config is a thing, which IMO encourages using it. I'm sure we have documentation that references it as well. Perhaps it should live in prefect.configuration.config and the "first class" recommendation becomes prefect.context.config.

[jlowin]

This is actually a really subtle detail -- but if a value in the config were itself a dictionary, this would replace that dictionary with a Config. Now the config really shouldn't have any dictionaries in it, as its TOML defined and the only possible dictionaries are keys, not values, but it's just something to call out.

[cicdw]

Yea, I thought about this some; because we don't hedge against it in our normal config, I wasn't concerned here. Ultimately I'd argue users should only use config by setting a finite list of environment variables which act as levers for affecting cloud functionality (or local flow-building functionality, i.e., default settings). For all other dynamic values, users can always provide context values directly when creating a flow run (whether locally or in Cloud).

[jlowin]

Yeah I agree with you.

[cicdw]

Whether to remove prefect.config from the top level is a good question; I don't think we should hide it / protect it, but I do think we should document that it is not intended to be interacted with directly and is always placed in context during execution. For all other dynamic data, context is what users should interact with.

[jlowin]

👍