Dynamic context via prefect.runtime

[cicdw]

Many users rely on various pieces of run / deployment configuration for dynamically specifying pieces of code external to a flow function; for example, one might want to configure a task runner based on either tags or parameters passed to a deployment run.

Today, users must query the API themselves for information outside of a flow function; within a flow / task function, they can use get_run_context but that is intended for internal purposes more than as a user facing entity (there are many attributes here that would be confusing for users to access).

This PR introduces a new submodule, prefect.runtime, for accessing such information in an intuitive and graceful way - when accessed outside of a known context, all attributes return empty values. This submodule can easily be expanded upon following the patterns developed here in ways that are memory efficient and independent of implementation details of the engine. Note that the attribute access implemented here was inspired by PEP 562.

Example

Imagine using a RayTaskRunner and wanting to tag the Ray cluster based on customer name, which is also a parameter to your flow:

from prefect.runtime import deployment

customer_tag = deployment.parameters.get("customer_name", "dev")

@flow(
    task_runner=RayTaskRunner(init_kwargs={
        "customer_tag": customer_tag
    })
)
def main(*, customer_name: str):
    # [...]

Previously this was not achievable in a first class way.

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• This pull request includes tests or only affects documentation.
• This pull request includes a label categorizing the change e.g. fix, feature, enhancement

[netlify]

✅ Deploy Preview for prefect-docs ready!

Name	Link
🔨 Latest commit	4f0fc43
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-docs/deploys/6410913b20a02c000827b0ae
😎 Deploy Preview	https://deploy-preview-8790--prefect-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[desertaxle]

This looks cool! Just a few small suggestions/questions from me

[desertaxle]

LGTM!

[billpalombi]

I'm so psyched that you knocked this out so quickly. Thanks for updating the API reference. We'll want other docs on how to use this. I'll start thinking about that today.

[jbnitorum]

This is a much more elegant solution than get_run_context. Can I suggest adding more available attributes? Specifically flow_name, flow_run_name, and deployment_name. Obviously we can get these using the API once we have the already available IDs but it would be much more straight forward if they were available as attributes.

For people transitioning from v1 (like us) it would make things easier if the v1 context items (https://docs-v1.prefect.io/api/latest/utilities/context.html#context-2) were all available as attributes in prefect.runtime in V2. Or at least all the ones that are still applicable in v2.

[zanieb]

@jbnitorum yep we will continue to add attributes following this first pass; if you want to open an issue it might help facilitate other contributions.

[sckott]

+1 on flow_name

[zanieb]

#8864 adds some more attributes