Contrib Work Makes the Dream Work

Changelog

0.12.3

Released on July 8, 2020.

Enhancements

• Update flow.slugs during flow.replace - #2919
• flow.update accepts the optional kwarg merge_parameters that allows flows to be updated with common Parameters - #2501
• Added poke handler to notify agent process of available flow runs - #2914
• Add Cancelling state for indicating a flow-run that is being cancelled, but may still have tasks running - #2923

Task Library

• Add ReadAirtableRow task - #2843
• Add container_name kwarg to CreateContainer Docker task - #2904
• Adds an extra_docker_kwargs argument to CreateContainer Docker task - #2915

Fixes

• Fix issue with short-interval IntervalClocks that had a start_date far in the past - #2906
• When terminating early, executors ensure all pending work is cancelled/completed before returning, ensuring no lingering background processing - #2920

Contributors

• Bradley McElroy
• Itay Livni
• Matthew Alhonte
• Panagiotis Simakis
• Sandeep Aggarwal

Maps? Maps.

Changelog

0.12.2

Released on June 30, 2020.

Features

• Add apply_map, a function to simplify creating complex mapped pipelines - #2846

Enhancements

• Make storage location inside Docker storage configurable - #2865
• Send heartbeats on each iteration of the Cloud task runner's retry loop - #2893

Task Library

• Add option to BigQueryTask to return query as dataframe - #2862

Fixes

• Add more context keys when running locally so that templating is consistent between local and Cloud runs - #2662
• Fix Fargate agent not parsing string provided containerDefinitions - #2875
• Fix Fargate agent providing empty parameters if not set - #2878
• Fix issue with Queued task runs flooding agents with work - #2884
• Add missing prefect register flow to CLI help text - #2895

Contributors

• James Lamb
• nathaniel-md

Hot Reloading Flows

0.12.1

Released on June 25, 2020.

Features

• Task slugs are now stable across rebuilds of the same Flow - #2531
• Support configuring executors for LocalEnvironment, KubernetesJobEnvironment, and FargateTaskEnvironment - #2805
• Flows can now be stored and executed using file-based storage - #2840

Enhancements

• Add option to set repositoryCredentials on Fargate Agent containerDefinitions - #2822
• Update GraphQL endpoint to /graphql - #2669
• Allow Cloud Flow Runners to interact properly with Queued runs - #2741
• Add Result serializers - #2755
• Simplify DaskExecutor internals - #2817
• Set task names in LocalDaskExecutor - #2819
• Flows registered without an image set will default to all_extras - #2828
• Improve error message when sending unauthorized requests to Cloud - #2810
• Forward state change status back to core - #2839
• Add GitHub storage for storing flows as files in a GitHub repo - #2840
• Add prefect register flow CLI command for registering flows from files - #2840
• Add default GITHUB_ACCESS_TOKEN secret - #2840
• Create utility function for getting Kubernetes client - #2845

Task Library

• Adds a MySQL task using pymysql driver - #2124
• Add some tasks for working with Google Sheets - #2614
• Add support for HTML content in the EmailTask - #2811

Server

• Failing to set a state raises errors more aggressively - #2708

Fixes

• Fix all_extras tag not being set during CI job to build image - #2801
• Quiet no candidate Cached states were valid debug logging - #2815
• Fix LocalEnvironment execute function's use of the flow object - #2804
• Properly set task names when using DaskExecutor - #2814
• Fix the LocalDaskExecutor to only compute tasks once, not multiple times - #2819
• Generate key names for mapped tasks that work better with Dask's dashboard - #2831
• Fix FlowRunTask when running against locally deployed Server - #2832
• Make sure image from Docker storage is always used with KubernetesJobEnvironment - #2838
• Change Environment.run_flow() to prefer executor from flow's environment - #2849

Deprecations

• Deprecate RemoteEnvironment in favor of LocalEnvironment - #2805
• Deprecate RemoteDaskEnvironment in favor of LocalEnvironment with a DaskExecutor - #2805
• Deprecate executor_kwargs in KubernetesJobEnvironment and FargateTaskEnvironment in favor of executor - #2805

Breaking Changes

• Remove previously deprecated SynchronousExecutor - #2826

Contributors

• manesioz
• Alex Cano
• James Lamb
• Matthew Alhonte
• Paweł Cieśliński

Into the Depths

Changelog

0.12.0

Released on June 17, 2020.

Features

• Depth First Execution with Mapping on Dask - #2646
• Support use of cloud storage with containerized environments - #2517,#2796

Enhancements

• Add flag to include hostname on local storage - #2653
• Add option to set image_pull_secret directly on DaskKubernetesEnvironment - #2657
• Allow for custom callables for Result locations - #2577
• Ensure all Parameter values, included non-required defaults, are present in context - #2698
• Use absolute path for LocalResult location for disambiguation - #2698
• Retry client requests when receiving an API_ERROR code in the response - #2705
• Reduce size of serialized tasks when running on Dask - #2707
• Extend run state signatures for future development - #2718
• Update set_flow_run_state for future meta state use - #2725
• Add an optional flow argument to merge to support using it when not inside a flow context - #2727
• Add option to set service account name on Prefect jobs created by Kubernetes agent - #2547
• Add option to set imagePullPolicy on Prefect jobs created by Kubernetes agent - #2721
• Add option to set API url on agent start CLI command - #2633
• Add CI step to build prefecthq/prefect:all_extras Docker image for bundling all Prefect dependencies - #2745
• Move Parameter to a standalone module - #2758
• Validate Cached states based on hashed inputs - #2763
• Add validate_configuration utility to Fargate Agent for verifying it can manage tasks properly - #2768
• Add option to specify task targets as callables - #2769
• Improve State.__repr__ when there is no message - #2773
• Add support for db argument at run time in the SQLiteQuery and SQLiteScript - #2782
• Add support for mapped argument in control flows - #2784
• Use pagination in kubernetes resource manager to reduce memory usage - #2794

Task Library

• Adds a task to expose Great Expectations checkpoints as a node in a Prefect pipeline - #2489

Fixes

• Fix flow.visualize cleanup of source files when using filename - #2726
• Fix S3Result handling of AWS credentials provided through kwargs - #2747
• Fix DaskKubernetesEnvironment requiring that an env block is set when using custom specs - #2657
• Fix PostgresExecute task auto commit when commit is set to False - #2658
• Remove need for {filename} in mapped templates - #2640
• Fix issue with Results erroring out on multi-level mapped pipelines - #2716
• Fix issue with dask resource tags not being respected - #2735
• Ensure state deserialization works even when another StateSchema exists - #2738
• Remove implicit payload size restriction from Apollo - #2764
• Fix issue with declared storage secrets in K8s job environment and Dask K8s environment - #2780
• Fix context handling for Cloud when working with in-process retries - #2783

Deprecations

• Accessing prefect.core.task.Parameter is deprecated in favor of prefect.core.parameter.Parameter - #2758

Breaking Changes

• Environment setup and execute function signatures now accept Flow objects - #2796
• create_flow_run_job logic has been moved into execute for DaskKubernetesEnvironment and KubernetesJobEnvironment - #2796

Contributors

• Alex Cano
• David Haines
• Paweł Cieśliński

Please Sign Here

Changelog

0.11.5

Released on June 2, 2020.

Enhancements

• Allow for manual approval of locally Paused tasks - #2693
• Task instances define a __signature__ attribute, for improved introspection and tab-completion - #2602
• Tasks created with @task forward the wrapped function's docstring - #2602
• Support creating temporary dask clusters from within a DaskExecutor - #2667
• Add option for setting any build kwargs on Docker storage - #2668
• Add flow run ID option to get logs CLI command - #2671
• Add ID to output of get command for flows and flow-runs - #2671

Fixes

• Fix issue with Google imports being tied together - #2661
• Don't warn about unused tasks defined inline and copied - #2677
• Remove unnecessary volume mount from dev infrastructure Docker compose - #2676
• Fix issue with instantiating LocalResult on Windows with dir from other drive - #2683
• Fix invalid IP address error when running server start on Ubuntu using rootless Docker - #2691

Deprecations

• Deprecate local_processes and **kwargs arguments for DaskExecutor - #2667
• Deprecate address='local' for DaskExecutor - #2667

Contributors

• Alex Cano

Revert GraphQL Endpoint Change

Changelog

0.11.4

Released on May 27, 2020.

Fixes

• Revert GraphQL endpoint change - #2660

It Is A Hodgepodge My Dudes

Changelog

0.11.3

Released on May 27, 2020.

Enhancements

• Add option to set volumes on server start CLI command - #2560
• Add case to top-level namespace - #2609
• Use host IP for hostname label in cases where LocalAgent is in container using host network - #2618
• Add option to set TLS configuration on client created by Docker storage - #2626
• The start_time of a Paused state defaults to None - #2617
• Raise more informative error when Cloud Secret doesn't exist - #2620
• Update GraphQL endpoint to /graphql - #2651

Fixes

• Kubernetes agent resource manager is more strict about what resources it manages - #2641
• Fix error when adding Parameter to flow under case statement - #2608
• Fix S3Result attempting to load data when checking existence - #2623

Deprecations

• Deprecate private_registry and docker_secret options on DaskKubernetesEnvironment - #2630

Breaking Changes

• Kubernetes labels associated with Prefect flow runs now have a prefect.io/ prefix (e.g. prefect.io/identifier) - #2641

Contributors

• Bartek Roszak
• James Lamb

A Fix in the Changelog is Worth Two in the Code

Changelog

0.11.2

Released on May 19, 2020.

Enhancements

• Allow log configuration in Fargate Agent - #2589
• Reuse prefect.context for opening Flow contexts - #2581
• Show a warning when tasks are created in a flow context but not added to a flow - #2584

Server

• Add API healthcheck tile to the UI - #2395

Fixes

• Fix type for Dask Security in RemoteDaskEnvironment - #2571
• Fix issue with log_stdout not correctly storing returned data on the task run state - #2585
• Ensure result locations are updated from targets when copying tasks with task_args - #2590
• Fix S3Result exists function handling of NoSuchKey error - #2585
• Fix confusing language in Telemetry documentation - #2593
• Fix LocalAgent not registering with Cloud using default labels - #2587
• Fix flow's run_agent function passing a set of labels to Agent instead of a list - #2600

Contributors

• Joe Schmid

Fix Duplicate Agent Label Parsing

Changelog

0.11.1

Released on May 15, 2020.

Fixes

• Fix duplicate agent label literal eval parsing - #2569

Improved APIs, Who Dis?

Changelog

0.11.0

Released on May 14, 2020.

Features

• Introducing new Results interface for working with task results - #2507

Enhancements

• Allow slack_task to accept a dictionary for the message parameter to build a specially-structured JSON Block - #2541
• Support using case for control flow with the imperative api - #2546
• flow.visualize is now able to accept a format argument to specify the output file type - #2447
• Docker storage now writes flows to /opt dir to remove need for root permissions - #2025
• Add option to set secrets on Storage objects - #2507
• Add reserved default Secret names and formats for working with cloud platforms - #2507
• Add unique naming option to the jobs created by the KubernetesJobEnvironment - #2553
• Use ast.literal_eval for configuration values - #2536
• Prevent local cycles even if flow validation is deferred - #2565

Server

• Add "cancellation-lite" semantic by preventing task runs from running if the flow run isn't running - #2535
• Add minimal telemetry to Prefect Server - #2467

Task Library

• Add tasks to create issues for Jira and Jira Service Desk #2431
• Add DbtShellTask, an extension of ShellTask for working with data build tool (dbt) - #2526
• Add prefect.tasks.gcp.bigquery.BigQueryLoadFile - #2423

Fixes

• Fix bug in Kubernetes agent deployment.yaml with a misconfigured liveness probe - #2519
• Fix checkpointing feature not being able to be disabled when using server backend - #2438

Deprecations

• Result Handlers are now deprecated in favor of the new Result interface - #2507

Breaking Changes

• Allow for setting docker daemon at build time using DOCKER_HOST env var to override base_url in docker storage - #2482
• Ensure all calls to flow.run() use the same execution logic - #1994
• Moved prefect.tasks.cloud to prefect.tasks.prefect - #2404
• Trigger signature now accepts a dictionary of [Edge, State] to allow for more customizable trigger behavior - #2298
• Remove all uses of credentials_secret from task library in favor of PrefectSecret tasks - #2507
• Remove Bytes and Memory storage objects - #2507

Contributors

• Alvin Goh
• Daniel Kapitan
• Mark McDonald
• Jie Lou