Nightly Release Candidate 3.0.0rc6

What's Changed

Fixes

• Make Task.serve sync compatible by @bunchesofdonald in #14184

Documentation

• React -> automate by @cicdw in #14185

Uncategorized

• Consolidate task runner content by @billpalombi in #14178
• Updates to cache policies by @cicdw in #14164
• Updates caching docs to be all about the new style of caching by @cicdw in #14165
• Added and edited Workspaces to 3.0 by @stellardave in #14085
• Separate pause and cancel by @billpalombi in #14179
• Move cancel runs by @billpalombi in #14191

Full Changelog: 3.0.0rc5...3.0.0rc6

Release 3.0.0rc5

What's Changed

Enhancements

• Log warning when a Concurrent, Dask, or Ray versions of PrefectFuture are garbage collection before resolution by @desertaxle in #14148

Fixes

• Fixes passing PrefectFutures to flows by @desertaxle in #14152
• Move timeout context into slot acquisition service by @zzstoatzz in #14121
• Add ability to defer result persistence by @cicdw in #14155
• Silence runtime errors for ephemeral shutdown by @jlowin in #14163

Documentation

• Add OSS to the quickstart by @djsauble in #14093
• chore(docs): add RayTaskRunner doc example using runtime_env by @parkedwards in #14138
• Remove references to ConcurrentTaskRunner from docs and ensure examples use .wait() by @zangell44 in #14144
• prefect-dbt - update integration docs by @seanpwlms in #14133
• Touch up flows section of docs by @cicdw in #14162
• Consolidates Dask content in Dask integration page by @billpalombi in #14177
• quickstart-tweaks by @billpalombi in #14142
• fix links by @stellardave in #14153
• Make some updates to the contributing sections of the docs by @cicdw in #14161
• Try to clean up task runner noise by @cicdw in #14169
• Consolidate Ray content by @billpalombi in #14171
• Remove separate logic section entirely by @cicdw in #14168

Refactor

• Move DeploymentImage to prefect.docker by @aaazzam in #14151
• relocate EntrypointType by @aaazzam in #14149
• Refactor FlexibleScheduleType by @aaazzam in #14158
• Remove remaining registry references by @aaazzam in #14173

Breaking

• Remove PrefectObjectRegistry [Taylor's Version] by @aaazzam in #14172

Integrations

prefect-docker

• add cached docker build and push steps by @zzstoatzz in #13286

prefect-kubernetes

• Fixes KubernetesClusterConfig import by @desertaxle in #14154

Full Changelog: 3.0.0rc4...3.0.0rc5

Release 3.0.0rc4

What's Changed

Enhancements

• Add pagination endpoint for deployments by @pleek91 in #14077
• Turn on task persistence by default by @desertaxle in #14102
• Add support for generator flows by @jlowin in #14061
• Adds a private status endpoint to TaskWorker by @chrisguidry in #14089

Fixes

• Fix bug where paused work queues getting unpaused and set to active on worker poll by @collincchoy in #14082
• Fix stale deployment statuses by @collincchoy in #14095
• Fix layout shift in home component by @pleek91 in #14112
• Addresses two race conditions in background task runs by @chrisguidry in #14115
• Move log to after task run name is set by @jlowin in #14063
• Avoid mutating collections unless absolutely necessary by @jlowin in #14048
• Acquire/release token on TaskWorker without errors by @jakekaplan in #14084
• Closes another race condition in task waiting by @chrisguidry in #14124
• Disables server-side PENDING task rescheduling by default by @chrisguidry in #14128
• Downgrade missing url log from warning to info by @jlowin in #14108

Documentation

• Updates scripts and folders to create SDK docs for Prefect and all integration libraries using Mkdocs by @discdiver in #14076
• Remove unnecessary pages and fix broken links by @discdiver in #14078
• Update PR template to reflect docs process changes by @discdiver in #14081
• Rename 'workflow' to 'flow' in the Prefect docs nav by @djsauble in #14049
• Update community to contribute in the docs by @discdiver in #14086
• Update integration links and content by @discdiver in #14096
• Update install instructions by @discdiver in #14119
• Update nearly 100 broken links by @discdiver in #14126
• Update install instructions and simplify README by @discdiver in #14135

Breaking

• Removes the deprected KubernetesClusterConfig block from the core library by @chrisguidry in #14107

Full Changelog: 3.0.0rc3...3.0.0rc4

Release 3.0.0rc3

Exciting New Features 🎉

• Add events to the flow run graph — #13875
• Add support for decorating instance/class/static methods with @task and @flow — #13944

Enhancements

• Update flow runs on Runs page to use pagination rather than infinite scrolling — #13839
• Raise helpful error when task_worker runs against ephemeral server — #13848
• Add new flows pagination endpoint — #13846
• Add better task retry logging — #13870
• Expose arg for max workers on ThreadPoolTaskRunner — #13866
• Remove dynamic key from non-flow task run name — #13902
• Add not_any_ filters to flow run state name/type — #13920
• Surface failed action error_detail for validation errors — #13940
• Add default store for transactions — #13983
• Prevent duplicate run creation from double button presses — #13986
• Add support for composite triggers — #13975
• Add transaction record expirations — #14035
• Add a url_for utility for generating URLs for Prefect objects — #13885
• Add synchronous client method for creating artifacts — #13931
• Add message for submitted tasks that are not waited — #13845

Fixes

• Add shield when re-enqueuing task runs — #13883
• Remove the limit and page from the history request — #13901
• Add circuit breaker when a task depends on itself — #13882
• Fix flow run retries for deployment flow runs — #13884
• Fix ThreadPoolTaskRunner concurrency by copying max_workers on duplication — #13943
• Fix prefect deploy with dynamic @flow decorator args — #13967
• Fix bug crashing task runs with remote storage — #13990
• Fix variables schema dump for work pool base job templates — #13971
• Prevent execution of code in if __name__ == "__main__": blocks when running prefect deploy — #13922
• Fix typing on GlobalConcurrencyLimitUpdate — #14005
• Bump TaskRunWaiter cache size and expiration time and pin task worker thread pool workers to passed limit — #14030
• Fix Kubernetes guide to avoid suggesting you need cloud — #14040
• Require a PREFECT_API_URL to start a worker — #13942
• Remove LiteralResult and handle singleton values the same — #13905
• Resolve an infinite loop distributing events to a websocket subscriber — #13877

Documentation

• Remove gerunds and redundant prefixes from CLI command help text — #13788
• Remove script tag — #13852
• Quickstart 3.0rc polish — #13879
• Update pydantic syntax in docs — #13888
• Improves integrations docs formatting — #13934
• Simplifies integration docs — #13926
• Fix code block formatting in automations concepts docs — #13959
• Add react docs — #13965
• Update Quickstart install instructions for 3.0 — #13988
• Fix docs formatting in integrations page — #13999
• Removed references to SequentialTaskRunner — #13996
• Removes extraneous docs files — #14016
• Remove extra arrow in top button — #14052
• Adds cards showing all integration libraries to the integrations page — #14051
• Update contribute section of the docs — #14055
• Add documentation about supported function types — #14060

Breaking Changes

• Remove unused attributes from EngineContext — #13917

Integrations

prefect-dbt

• Add back logging of dbt commands — #13853

prefect-databricks

• Support job parameters for Databricks jobs — #13642
• Restore jobs_runs_submit_by_id_and_wait_for_completion — #13958

Contributors

• @LarryUllman
• @Waiwait
• @padbk
• @stellardave

All changes: 3.0.0rc2...3.0.0rc3

Release 2.19.5

Release 2.19.5

Fixes

• Fix prefect deploy when using dynamic @flow parameters - #13991
• Fix prefect deploy to prevent code execution inside main guard - #14004
• Fix typing on GlobalConcurrencyLimitUpdate - #14006

Integrations

• Restore jobs_runs_submit_by_id_and_wait_for_completion to prefect-databricks - #13957

Documentation

• Fix formatting in automations concept docs - #13962

All changes: 2.19.4...2.19.5

Release 3.0.0rc2

We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc2 and check out the docs here.

Run tasks independently of flows

You can now run and serve tasks outside of flows and inside of other tasks.

from prefect import task

@task
def my_background_task(name: str):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_background_task.delay("ford")

Transactional semantics

Use rollback and commit hooks to facilitate idempotent python code.

from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
    print('first')

@first_task.on_rollback
def roll(txn):
    print('rolling back')

@task
def second_task():
    raise RuntimeError("oopsie")

@flow
def txn_flow():
    with transaction():
        first_task()
        second_task()
if __name__ == "__main__":
    txn_flow()

Open source Events and Automations

Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.

More flexible variables and new artifact types

Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.

Faster and richer CLI

Improved CLI speed and several added commands and conveniences.

Updated navigation, styling, and interaction design

The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.

Changes since 3.0.0rc1

Enhancements

• Adds deferred kwarg to Task.map to executed mapped tasks on a task server by @desertaxle in #13756
• Add pydantic version to cli version command by @pleek91 in #13715
• conditionally log retryable errors by @jakekaplan in #13773
• Update Variable.set return value + docs by @jakekaplan in #13557
• Add new flow runs pagination endpoint by @pleek91 in #13808
• Updates PrefectDistributedFuture to use an event websocket to determine task completion by @desertaxle in #13763
• Enhancement: Artifacts documentation by @dylanbhughes in #13724
• Migrate prefect-kubernetes to pydantic 2 by @zzstoatzz in #13687
• Add endpoint for bulk task run counts for flow runs by @pleek91 in #13759
• Copy pared down prefect-redis to integrations by @bunchesofdonald in #13758
• Cache key policies by @cicdw in #13791
• Refactor task run engine by @jlowin in #13793
• Cache Policy Upgrades by @cicdw in #13821
• Put AwsCredentials first to match default factory by @bunchesofdonald in #13819

Fixes

• Fix load_flow_argument_from_entrypoint to work with async flows by @elisalimli in #13716
• Add handling for positional only and keyword only arguments when parsing a function signature from source code by @desertaxle in #13774
• Load assignments when safeloading a namespace by @desertaxle in #13775
• Coerce work pool env values to str by @zzstoatzz in #13782
• Support retry_delay_seconds in task engine by @zzstoatzz in #13815
• Rename RedisFilesystem to RedisStorageContainer to match naming semantics of other filesystem classes by @bunchesofdonald in #13771

Documentation

• Fixes over 500 broken links in docs by @discdiver in #13734
• Remove agents concept page and refs for 3.0 by @serinamarie in #13631
• WIP 3.0rc docs new IA (on Mintlify) by @LarryUllman in #13783
• Fixes a doc build from an admonition change and note managed execution not in beta. by @discdiver in #13789
• Update version instruction install docs page by @discdiver in #13795
• Move integration package docs out of versioning by @discdiver in #13760
• Update button text at top of page by @discdiver in #13797
• Update get-started/install page by @LarryUllman in #13822
• Porting over task guide, mdx bugfixes by @WillRaphaelson in #13754

New Contributors

• @elisalimli made their first contribution in #13716
• @LarryUllman made their first contribution in #13783

Full Changelog: 3.0.0rc1...3.0.0rc2

2.19.4

Fixes

• Fix Could not find flow '[...]' in '[...].py' errors when using prefect deploy to deploy async flows - #13769
• Fix parameter schema generation for flows with positional only and keyword only arguments when using prefect deploy - #13778
• Fix use of dynamic models in flow typing when using prefect deploy - #13781
• Allow clients to provide task run ID during creation to enable compatibility between 2.x servers and 3.0.0rc clients - #13683

Documentation

• Add Docker deployment steps page to centralized integration docs - #13720

All changes: 2.19.3...2.19.4

Release 3.0.0rc1

Release 3.0.0rc1

We're excited to announce the release candidate of Prefect 3.0. It's the most flexible, powerful, fastest version of Prefect yet. Prefect 3.0 includes several exciting new features. Install it by running pip install prefect==3.0.0rc1 and check out the docs here.

Run tasks independently of flows

You can now run and serve tasks outside of flows and inside of other tasks.

from prefect import task

@task
def my_background_task(name: str):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_background_task.delay("ford")

Transactional semantics

Use rollback and commit hooks to facilitate idempotent python code.

from prefect import flow, task
from prefect.transactions import transaction
@task
def first_task():
    print('first')

@first_task.on_rollback
def roll(txn):
    print('rolling back')

@task
def second_task():
    raise RuntimeError("oopsie")

@flow
def txn_flow():
    with transaction():
        first_task()
        second_task()
if __name__ == "__main__":
    txn_flow()

Open source Events and Automations

Trigger actions, such as sending notifications, pausing schedules, starting flow runs and more in response to Prefect events.

More flexible variables and new artifact types

Variables can now be any JSON compatible type including dicts, lists, and integers. Progress and Image artifacts make it easy to add visual annotations to your flow run graph.

Faster and richer CLI

Improved CLI speed and several added commands and conveniences.

Updated navigation, styling, and interaction design

The new Runs page displays both flow and task run information, and an improved sidebar and switcher makes navigating Prefect simpler than ever.

Enhancements

• Create artifact for unsuccessful dbt task runs — #13348
• Add filter on task_run.expected_start_time — #13491
• Add utilities to serialize context to a dictionary and hydrate context from a dictionary — #13529
• Add API endpoints for deployment count and next flow run — #13544
• Allow flow parameter schema generation when dependencies are missing — #13315
• Change the default value for enforce_parameter_schema from False to True — #13594
• Migrate schemas to pydantic v2 — #13574
• Removes block auto-instrumentation — #13407
• Migrate all uses of the banned characters validation to a self-validator — #13370
• Ignore and warn on unrecognized settings - #13624

Fixes

• Remove unnecessary flow run infrastructure override access checks — #13401
• Enforce False case when flow run id is null — #13464
• Fix workspace variable hydration to allow for JSON data — #13548
• Remove unused settings/experimental work pool flags: PREFECT_EXPERIMENTAL_ENABLE_WORK_POOLS and PREFECT_EXPERIMENTAL_WARN_WORK_POOLS — #13144
• Pin pydantic>=2.7 for Secret — #13613
• Skip on cancellation hooks if runner can't load flow — #13660
• Refactor lazy imports to avoid accidental eager imports — #13296
• Allow block registration to use client schemas for server model creation — #13602
• Replace our customized Duration types with plain timedeltas — #13603

Experimental

• Add prefect.yaml and cli support for new schedule fields — #13318

Documentation

• Transition documentation hosting from Netlify to Mintlify — #13634
• Add Python 3.12 to list of Docker images — #13321
• Update index.md — #13353
• Improve tutorial section — #13297
• Fix jinja template in automations doc — #13422
• Update development section docs — #13247
• Update Ray integration docs — #13467
• Update Variables docs to include JSON types — #13493
• Update quickstart guide for usability — #13562
• Remove deployments-block-based concept page and refs for 3.0 — #13626
• Remove infrastructure concept page and refs for 3.0 — #13629
• Update docs image paths and remove outdated images — #13666
• Remove references to prefect.software from docs — #13382
• Update host.md — #13351
• Simplify rate limits page — #13689
• Removing references to deprecated block types and add disclaimer — #13651
• Update guides — #13253
• Remove storage concept page and refs - #13630

Integrations

• Migrate prefect-dbt to pydantic v2 - #13718
• Migrate prefect-email to pydantic v2 — #13654
• Migrate prefect-slack to pydantic v2 — #13673
• Migrate prefect-shell to pydantic v2 — #13675
• Migrate prefect-gcp to pydantic v2 — #13650
• Migrate prefect-github to pydantic v2 — #13655
• Migrate prefect-gitlab to pydantic v2 — #13656
• Migrate prefect-docker to pydantic v2 - #13697
• Migrate prefect-sqlalchemy to pydantic v2 - #13700
• Add PrefectDistributedClient to prefect-dask — #13537
• Update RayTaskRunner for compatibility with new engine — #13575
• Update DaskTaskRunner for compatibility with the updated engine — #13555
• prefect-dbt artifact consolidation and markdown fixes — #13379
• prefect-dbt - Cause unsuccessful dbt tasks to fail — #13405
• DBT Tasks extra_command_args Fix — #13308
• Update dbt-core dependency — #13394

Breaking Changes

• Remove prefect deployment build CLI from main — #13366
• Remove prefect agent CLI from main — #13365
• Remove prefect deployment apply CLI from main — #13367
• Remove PrefectAgent class — #13374
• Remove prefect.software — #13375
• Remove deployments module — #13373
• Remove EcsTask from main — #13417
• Remove AzureContainerInstanceJob from main — #13418
• Remove VertexAICustomTrainingJob from main — #13419
• Remove CloudRunJob from main — #13420
• Remove infrastructure blocks from main — #13424
• Remove Infrastructure, BlockWorker from main — #13430
• Remove deprecated storage blocks from main — #13410
• Remove prefect-agent as a possible work pool type — #13444
• Remove old engine — #13542
• Remove Python 3.8 support — #13331
• Remove deprecated module and its references — #13345
• Remove old task runners and futures modules — #13593
• Remove is_state — #13569
• Remove deprecated options from prefect work-queue and refs to agents - #13638

Contributors

• @bsignoret

• @jaraics made their first contribution in #13144

All changes: 2.19.0...3.0.0rc1

See release notes for details.

Release 2.19.3

New method for generating parameter schemas without dependencies

prefect deploy now works even when dependencies are missing from the current environment. This can speed up deployment via CI by removing the need to install dependencies before deploying your flows.

See the following pull requests for implementation details:

• #13620
• #13315

Enhancements

• Provide URL in CLI output upon work pool creation — #13597

Fixes

• Ensure graceful cancellation of flow runs corresponding to deleted deployments — #13669

Integrations

• Add loading state to concurrency limits table in the Prefect UI — PrefectHQ/prefect-ui-library#2483
• Remove old schema properties from deployments in the Prefect UI — PrefectHQ/prefect-ui-library#2482
• Add handling for multi-word dbt CLI commands — #13616

All changes: 2.19.2...2.19.3

Release 2.19.2

This release includes a few bug fixes, ensuring:

• 🛠️ runs created from the deployments/{id}/create_flow_run endpoint hydrates workspace variables
• 🔢 proper integer value display on the Variables page of the Prefect UI — PrefectHQ/prefect-ui-library#2454
• ⚙️ "Run a deployment" automation action parameter display input configuration persists during editing — PrefectHQ/prefect-ui-library#2458
• 📌 requests dependency pinned to <2.32.0 in requirements-dev.txt - #13538
• 📄 Jinja template example is renderable in automations documentation — #13421