Release 2.16.7

💻 Introducing prefect shell for observing CLI commands

You can now observe CLI commands as a Prefect flow. For example:

» prefect shell watch "curl http://wttr.in/Chicago?format=3"
17:32:39.562 | INFO | prefect.engine - Created flow run 'powerful-mushroom' for flow 'Shell Command'
17:32:40.171 | INFO | Flow run 'powerful-mushroom' - Chicago: ⛅️ +50°F
17:32:40.315 | INFO | Flow run 'powerful-mushroom' - Finished in state Completed()

See these docs to learn how to:

• run a shell command as a Prefect flow on-demand with watch
• schedule a shell command as a recurring Prefect flow using serve

See the PR for implementation details: #11998

Other Enhancements 🌟

• Integrate composite triggers with the DeploymentTrigger YAML representation — #12413
• Add JSON Artifacts — #12295
• Add auto-provisioning option for Cloud Run V2 push work pools — #12422

... and numerous 🐛 fixes!

Full Changelog: 2.16.6...2.16.7

See the release notes for more!

Release 2.16.6

Fix new behavior in typer 0.10.0 that broke the prefect CLI

See the following pull request for implementation details:

• Handle new typer.Option behavior — #12398

All changes: 2.16.5...2.16.6

Release 2.16.5

Multi-select deletion of flow runs

It is now easier to bulk select and delete flow runs through the UI. Listings of filterable and selectable flow runs (e.g. on the flow runs, flow, and deployment pages) now include a top-level checkbox for (de)selecting all currently filtered flow runs for bulk deletion.

See the following pull request for implementation details:

• #12356
• PrefectHQ/prefect-ui-library#2227
• #12285

Visualize state changes and artifacts in the UI

Additionally, the flow run graph UI enhancements for visualizing state changes and artifacts added in 2.16.4 are now enabled by default. See the release notes in 2.16.4 for more details!

Full Changelog: 2.16.4...2.16.5

---

See the release notes for more!

Release 2.16.4

Release 2.16.4

Flow Run Graph updates

The Flow Run Graph has been updated to display additional layers of information! Interactive and real-time state changes and artifacts are now visible in context on the graph.

These new layers are available for opt-in usage via the PREFECT_EXPERIMENTAL_ENABLE_ARTIFACTS_ON_FLOW_RUN_GRAPH and PREFECT_EXPERIMENTAL_ENABLE_STATES_ON_FLOW_RUN_GRAPH settings.

Agents

A year ago, we released workers as a replacement for agents. Workers significantly enhance the experience of deploying flows and simplify the specification of each flow's infrastructure and runtime environment.

With this release we are adding a six month (September 14) deprecation warning to agents and related concepts. Please note that:

• Deprecation will not impact or break any work running with agents and agent-related concepts - although we encourage users to upgrade to workers because they provide a better deployment experience, you can continue to use existing agents and related concepts after deprecation
• After September 14, Prefect Cloud users will not be able to create new agent work pools or infrastructure blocks
• After September 14, new minor versions of the Prefect Python package will not include agents

Like agents, workers support creating deployments through the Prefect CLI and through Python, but require different syntax. For more information, please refer to the Upgrade from Agents to Workers guide.

Enhancements

• Give better client-side feedback on websocket authT/authZ issues - #12221
• Allow table artifact cells to render markdown content - #2190
• Add an 'AzureBlobStorageContainer' block - #139
• API for task run counts by state - #12244
• Improved UI handling of custom flow run states. Badges for a state with a custom name will now more closely resemble their underlying state - PrefectHQ/prefect-ui-library#2210 and PrefectHQ/prefect-ui-library#2208

Fixes

• Fix support for legacy schedule in build_from_flow - #12257
• Remove extra field from create endpoints - #12240
• Prevent invalid link problem for prefect deployment run - #12267
• Hydrate run input when resuming a flow run - #12259
• Fix state select in notifications #2216
• Adds porting of network configuration to generated base job templates #392

Experimental

• Expose serve method on tasks - #12239
• Avoid relative module path problem with subscription keys - #12227

Deprecations

• Deprecate PrefectAgent and prefect agent command group - #12273
• Deprecate Infrastructrue blocks - #12282
• Deprecate Deployment class and deployment build and apply commands - #12283
• Deprecate the packaging module - #12291
• Deprecate storage blocks with implementations in Prefect integration packages - #12299
• Do not create default-agent-pool in empty state server - #12214

Typing

• Refactor: Inject Client - #12258
• Fix: Variables methods type-hinting - #12278

Documentation

• Fix formatting in concept -> work-pools.md - #12230
• Add get_run_logger to imports in interactive workflow examples - #12284
• Add on_running state change hook to State -> Concepts docs - #12293
• Fix some argument names in deploy guide - #12301

New Contributors

• @hainenber made their first contribution in #12232

Full Changelog: 2.16.3...2.16.4

Release 2.16.3

This release includes enhanced deployment parameters in the Prefect UI. We’ve refined the deployment parameters UI to significantly enhance default value handling and list management.

🛠️ This release introduces support for tuple-type parameters and a revamped list UI, freeing users from the constraints of JSON for list inputs.

🔄 You now have the flexibility to utilize JSON or Prefect variables for any parameter value, enhancing the versatility of deployment configurations.

💪 Moreover, we’ve synchronized validation across the UI and deployment schemas, ensuring that user inputs are consistently checked against the defined parameter requirements, which simplifies the deployment process and minimizes configuration errors.

✅ These improvements are aimed at providing a more efficient and user-friendly interface for managing deployment parameters. 🌟

This release also includes many other enhancements and fixes, including a new on_running state change hook 🎣, two memory leak fixes 🔧, and much more.

See the release notes for details! 📜🚀

Release 2.16.2

⛑️ Fixing a logging bug introduced by 2.16.1

In 2.16.1, a regression related to logging non-str objects (when a PREFECT_API_KEY was set) was introduced. The fix is included in 2.16.2.

See #12151 for implementation details.

🎯 Use a module path entrypoint when using .serve or .deploy

from prefect_dbt.cloud.jobs import trigger_dbt_cloud_job_run_and_wait_for_completion

from prefect.deployments import EntrypointType

if __name__ == "__main__":
    trigger_dbt_cloud_job_run_and_wait_for_completion.deploy(
        name="demo-deploy-from-module",
        work_pool_name="olympic",
        entrypoint_type=EntrypointType.MODULE_PATH,
        job_variables={"env": { "EXTRA_PIP_PACKAGES": "prefect-dbt" } }
    )

See #12134 for implementation details.

Full Changelog: 2.16.1...2.16.2

---

See the release notes for more!

Release 2.16.1

Enhanced multiple schedule support 📅

prefect.yaml now supports specifying multiple schedules via the schedules key.

This allows you to define multiple schedules for a single deployment, and each schedule can have its own cron, interval, or rrule configuration:

 ...
 schedules:
    - cron: "0 0 * * *"
      active: false
    - interval: 3600
      active: true
    - rrule: "FREQ=YEARLY"
      active: true

In addition you can specify multiple schedules via arguments to prefect deploy:

prefect deploy ... --cron '4 * * * *' --cron '1 * * * *' --rrule 'FREQ=DAILY'

More detail

We've also added support for multiple schedules to flow.serve, flow.deploy and prefect.runner.serve. You can provide multiple schedules by passing a list to the cron, interval, or rrule arguments:

import datetime
import random

from prefect import flow


@flow
def trees():
    tree = random.choice(["🌳", "🌴", "🌲", "🌵"])
    print(f"Here's a happy little tree: {tree}")

if __name__ == "__main__":
    trees.serve(
        name="trees",
        interval=[3600, 7200, 14400],
    )

This will create a deployment with three schedules, one that runs every hour, one that runs every two hours, and one that runs every four hours. For more advanced cases, use the schedules argument.

    trees.serve(
        name="trees",
        schedules=[
            IntervalSchedule(interval=datetime.timedelta(minutes=30)),
            {"schedule": RRuleSchedule(rrule="FREQ=YEARLY"), "active": True},
            MinimalDeploymentSchedule(schedule=CronSchedule(cron="0 0 * * *"), active=False),
        ]
    )

Dive into these new scheduling capabilities today and streamline your workflows like never before.

For implementation details, see the following pull request:
- #12107

New Contributors 🌟

• Shoutout to @jrbourbeau for their first contribution!

---

See the release notes for more details!

Release 2.16.0

🕢 Deployments now support multiple schedules 🕐

With today’s release, we’re excited to roll out initial support for using multiple schedules with Deployments! You can now use multiple schedules in the following ways:

• Specifying schedules in a Deployment YAML file
• Creating Python-based Deployments with the Deployment class
• New CLI commands: prefect deployment schedule <create, delete, pause, resume, ls, clear>
• New UI components aware of multiple schedules

Coming soon, we’ll round out support for multiple schedules in other areas, such as:

• When running a flow with flow.serve() and flow.deploy()
• When using prefect deploy

The easiest way to get started with multiple schedules is to try out the new CLI commands:

$ prefect deployment schedule ls happy-flow/my-deployment
                       Deployment Schedules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ID                                   ┃ Schedule        ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 │ cron: 0 0 1 * 1 │ True   │
└──────────────────────────────────────┴─────────────────┴────────┘

$ prefect deployment schedule create happy-flow/my-deployment --interval 60
Created deployment schedule!

$ prefect deployment schedule ls happy-flow/my-deployment
                         Deployment Schedules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ID                                   ┃ Schedule           ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 3638ed58-cab2-4462-a680-2f92fcf6c797 │ interval: 0:01:00s │ True   │
│ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 │ cron: 0 0 1 * 1    │ True   │
└──────────────────────────────────────┴────────────────────┴────────┘

Enhancements

• Add support for deploying to a process work pool using flow.deploy and deploy — #12017
• Add support for multiple schedules to Prefect server and CLI — #11971
• Add CLI command to read runs in a work queue — #11989

Fixes

• Update the text for the CLI command deployment run --help so it renders for more args — #11960
• Fix Flow.with_options logic for retries, retry_delay_seconds, flow_run_name — #12020
• Fix memory leaks related to cancellation scopes and async contextvar usage — #12019
• Revert the recent change that runs on the main thread while we investigate a concurrency issue — #12054
• Add a more readable error if Docker is not running — #12045

Documentation

• Improve language and formatting in Profiles and Configuration guide — #11996
• Improves docs formatting consistency and adds some minor content updates — #12004
• Updates formatting for guide: creating-interactive-workflows.md — #11991
• Add import statement for wait_for_flow_run — #11999
• Add deep dive on overriding job variables — #12033
• Remove extraneous trailing quotation marks in concepts/artifacts docs — #12040
• Add links to overriding job variables guide — #12043
• Update scheduling docs to include information about multiple schedules — #12064

Experimental

• Only allow using Task.submit() for autonomous task submission — #12025

Contributors

• @NodeJSmith

• @hamzamogni made their first contribution in #12000
• @eladm26 made their first contribution in #12045

Full release notes: https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md

Release 2.15.0

🔧 Task runs now execute on the main thread

We are excited to announce that task runs are now executed on the main thread!

When feasible, task runs are now executed on the main thread instead of a worker thread. Previously, all task runs were run in a new worker thread. This allows objects to be passed to and from tasks without worrying about thread safety unless you have opted into concurrency. For example, an HTTP client or database connection can be shared between a flow and its tasks now (unless synchronous concurrency is used). Some asynchronous and sequential use cases may see performance improvements.

Consider the following example:

import sqlite3
from prefect import flow, task

db = sqlite3.connect("threads.db")

try:
    db.execute("CREATE TABLE fellowship(name)")
except sqlite3.OperationalError:
    pass
else:
    db.commit()

db.execute("DELETE FROM fellowship")
db.commit()

cur = db.cursor()


@task
def my_task(name: str):
    global db, cur

    cur.execute('INSERT INTO fellowship VALUES (?)', (name,))

    db.commit()


@flow
def my_flow():
    global db, cur

    for name in ["Frodo", "Gandalf", "Gimli", "Aragorn", "Legolas", "Boromir", "Samwise", "Pippin", "Merry"]:
        my_task(name)

    print(cur.execute("SELECT * FROM fellowship").fetchall())

    db.close()


if __name__ == "__main__":
    my_flow()

In previous versions of Prefect, running this example would result in an error like this:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 7977619456 and this is thread id 6243151872.

But now, with task runs executing on the main thread, this example will run without error! We're excited this change makes Prefect even more intuitive and flexible!

See the following pull request for implementation details:
- #11930

🔭 Monitor deployment runs triggered via the CLI

You can monitor the status of a flow run created from a deployment via the CLI. This is useful for observing a flow run's progress without navigating to the UI.

To monitor a flow run started from a deployment, use the --watch option with prefect deployment run:

prefect deployment run --watch <slugified-flow-name>/<slugified-deployment-name>

See the following pull request for implementation details:
- #11702

Enhancements

• Enable work queue status in the UI by default — #11976 & PrefectHQ/prefect-ui-library#2080

Fixes

• Update vendored starlette version to resolve vulnerability in python-mulipart — #11956
• Fix display of interval schedules created with a different timezone than the current device - PrefectHQ/prefect-ui-library#2090

Experimental

• Prevent RUNNING -> RUNNING state transitions for autonomous task runs — #11975
• Provide current thread to the engine when submitting autonomous tasks — #11978
• Add intermediate PENDING state for autonomous task execution — #11985
• Raise exception when stopping task server — #11928

Documentation

• Update work pools concepts page to include Modal push work pool — #11954
• Add details to run_deployment tags parameter documentation — #11955
• Add Helm chart link in Prefect server instance docs — #11970
• Clarify that async nested flows can be run concurrently — #11982
• Update work queue and flow concurrency information to include push work pools — #11974

Contributors

• @zanieb

All changes: 2.14.21...2.15.0

Release 2.14.21

Introducing work queue status

We're excited to unveil the new status indicators for work queues in Prefect's UI, enhancing your ability to oversee and control flow run execution within our hybrid work pools.

Work queues will now display one of three distinct statuses:

• Ready - one or more online workers are actively polling the work queue
• Not Ready - no online workers are polling the work queue, signaling a need for intervention
• Paused - the work queue is intentionally paused, preventing execution

With the introduction of work queue status, you'll notice the absence of deprecated work queue health indicators in the UI.

See the documentation on work queue status for more information.

For now, this is an experimental feature, and can be enabled by running:

prefect config set PREFECT_EXPERIMENTAL_ENABLE_WORK_QUEUE_STATUS=True

See the following pull request for implementation details:
- #11829

Fixes

• Remove unnecessary WARNING level log indicating a task run completed successfully — #11810
• Fix a bug where block placeholders declared in pull steps of the deployments section of a prefect.yaml file were not resolved correctly — #11740
• Use pool_pre_ping to improve stability for long-lived PostgreSQL connections — #11911

Documentation

• Clarify Docker tutorial code snippet to ensure commands are run from the correct directory — #11833
• Remove beta tag from incident documentation and screenshots — #11921
• Update Prefect Cloud account roles docs to reflect renaming of previous "Admin" role to "Owner" and creation of new "Admin" role that cannot bypass SSO — #11925

Experimental

• Ensure task subscribers can only pick up task runs they are able to execute — #11805
• Allow a task server to reuse the same task runner to speed up execution — #11806
• Allow configuration of maximum backlog queue size and maximum retry queue size for autonomous task runs — #11825

All changes: 2.14.20...2.14.21