Zero One Four! It's a big one!

Changelog

0.14.0

Released on December 16, 2020.

Features

• New flow run configuration system based on RunConfig types, see here for more info

Enhancements

• Kubernetes Agent now reports events for pending pods created by prefect jobs - #3783
• Using --rbac for Kubernetes Agent install command now includes the events resource - #3783
• Add orchestration-based dependencies to the prefecthq/prefect Docker image - #3804
• Add a slimmed down prefecthq/prefect:core Docker image that only contains base dependencies - #3804
• Docker storage now installs all orchestration-based dependencies when using default image - #3804
• Add warning on flow registration if flow.executor is set but the flow is using the legacy flow.environment configuration system - #3808
• Echoing prefect config file as JSON to be able to parse it with jq in the terminal - #3818
• Produce artifact for RunGreatExpectationsValidation even if validation fails - #3829
• execute flow-run command now sends flow run log in the case of an error - #3832
• Changed name of logs raised by the Kubernetes Agent if they stem from infrastructure events - #3832
• Add tini to the official Prefect docker images - #3839
• Remove task run level heartbeats for performance - #3842

Task Library

• Fix mising job_id in DatabricksRunNow task initialization - #3793

Fixes

• Fix Azure result byte decoding of blob data - #3846
• Prefect kubernetes agent no longer relies on existence of any fields in configured Kubernetes Job Template - #3805
• Accept old envvar style configuration for Kubernetes agent for --service-account-name/--image-pull-secrets options - #3814
• Pass as_user=False when using client.get_cloud_url in StartFlowRun - #3850
• Fix AWS boto3 utility passing duplicate kwargs to client initialization - #3857

Deprecations

• Storage classes have been moved from prefect.environments.storage to prefect.storage, the old import paths have been deprecated accordingly - #3796
• Executor classes have been moved from prefect.engine.executors to prefect.executors, the old import paths have been deprecated accordingly - #3798
• Deprecated use of storage_labels boolean kwarg on local agent - #3800
• Deprecated use of --storage-labels option from agent start CLI command - #3800
• Deprecates all Environment classes - users should transition to setting flow.run_config instead of flow.environment - #3811
• Deprecate the Fargate Agent in favor of the ECS Agent - #3812

Breaking Changes

• Using in-cluster installs of the Kubernetes Agent now requires RBAC for the events resource - #3783
• Removed setting of default labels on storage objects and the local agent - #3800
• Remove deprecated RemoteEnvironment/RemoteDaskEnvironment - #3802
• Remove deprecated executor_kwargs argument to KubernetesJobEnvironment/FargateTaskEnvironment - #3802
• Remove deprecated prefect run cloud/prefect run server CLI commands - #3803
• Remove deprecated prefect execute cloud-flow CLI command - #3803
• Stop building the prefecthq/prefect:all_extras image and switch flow deployment default to using prefecthq/prefect:{core_version} - #3804
• Flows now use RunConfig based deployments by default - legacy Environment based deployments are now opt-in only - #3806
• Remove deprecated prefect.contrib module - #3813
• Remove all references to result handlers and safe results - #3838
• Remove option to enable deprecated Kubernetes resource manager in agent install CLI command - #3840

Contributors

• Christian Werner
• Erich Oliveira
• Jacob Hayes
• Pedro Martins

It was the best of releases...

Changelog

0.13.19

Released on December 8, 2020.

Enhancements

• Use explicit exception chaining - #3306
• Support Bitbucket as storage option - #3711
• Surface pod failures and container errors in jobs deployed with the Kubernetes Agent - 3747
• Support timeout option in GCS tasks - #3732
• Added storage option for AWS CodeCommit - #3733
• Add the image used for a flow-run to the flow run environment as prefect.context.image - #3746
• Add UniversalRun run-config that works with all agents - #3750
• Support flows that have no run-config or environment - #3750
• Allow Docker storage environment vars to be used in commands - #3755
• Add service_account_name and image_pull_secrets options to KubernetesRun and KubernetesAgent - #3778
• Add a new Client function delete_project - #3728

Task Library

• Add task to fetch data from Dremio - #3734
• Add RunGreatExpectationsValidation task - #3753
• Add the option to post markdown artifacts from the RunGreatExpectationsValidation task - #3753

Fixes

• Fix state attempting to read result from absent upstream result - #3618
• Replace deprecated download_as_string method with download_as_bytes method - #3741
• Fix default image whenever working on a non-tagged commit - #3748
• Fix type-casting for task timeout defaults loaded from config - #3761
• Fix the ref default on GitHub storage - #3764
• Fix rare cancellation bug when running with external Dask cluster - #3770

Deprecations

• Deprecated the RunGreatExpectationsCheckpoint task in favor of RunGreatExpectationsValidation - #3766

Contributors

• BluePoof
• Faris ALSaleem
• Jonathan Owen
• Klemen Strojan
• Phillip Choi
• Sam Bail
• Takayuki Hirayama

It's beginning to look a lot like 0.13.18

Changelog

0.13.18

Released on November 30, 2020.

Enhancements

• Display formatted graphql errors on client request failure - #3632
• Refactor Core Client API calls for performance - #3730

Task Library

• Refactor execute query code for PostgresExecute, PostgresExecuteMany, and PostgresFetch tasks - #3714
• Fix PicklingError in BigQueryLoadFile and BigQueryLoadGoogleCloudStorage - #3724
• Allow custom exporter for ExecuteNotebook task - #3725
• Properly forward location parameter in bigquery tasks - #3726
• Support passing helper_script to ShellTask/DBTShellTask at runtime - #3729

Fixes

• Fix bug with docker storage throwing exception while trying to display output - #3717

Contributors

• Amanda Wee
• Panagiotis Simakis
• Swier Heeres
• Takayuki Hirayama

Removing defects from Prefect to great effect

Changelog

0.13.17

Released on November 24, 2020.

Features

• Improved support for Tasks returning multiple results - #3697

Enhancements

• Allow chaining of Task imperative dependency calls - #3696
• Add task_definition_arn to ECSRun run-config - #3681
• Rerun resource_manager tasks when restarting flows from failed - #3689
• Raise nice warning if user passes Task instance to Task constructor, rather than when calling the Task (or using Task.map/Task.set_dependencies) - #3691
• Always use tenant slug in output of Client get_cloud_url function - #3692

Task Library

• Add enhancement to StartFlowRun task to create link artifact for started flow run - #3692
• Add a new postgres task PostgresExecuteMany - #3703
• Add debug logging for Docker tasks PullImage and BuildImage - #3672
• ShellTask returns output on failure - #3649
• ShellTask allows streaming of output independently of the number of lines returned - #3649

Fixes

• Make serialized_hash handle unordered task sets correctly - #3682
• Docker storage build error logs were not always displayed - #3693
• Fix automatic quoting of Docker storage environment variable values - #3694
• Use exist_ok flag in os.makedirs to avoid race condition in local storage class - #3679

Contributors

• Faris ALSaleem
• R Max Espinoza
• Takayuki Hirayama

Remember when Python 2 was a thing?

Changelog

Released on November 17, 2020.

Enhancements

• Experimental support for Python 3.9 - #3411

Fixes

• Fixes Flow.replace freezing reference tasks - #3655
• Fixed bug where flow.serialized_hash() could return inconsistent values across new python instances - #3654

Contributors

• Ben Fogelson

Why dig up artifacts when you can create them?

Changelog

0.13.15

Released on November 11, 2020.

Features

• Add API for storing task run artifacts in the backend - #3581

Enhancements

• Allow for setting Client headers before loading tenant when running with Prefect Server - #3515
• Checkpoint all iterations of Looped tasks - #3619
• Add ref option to GitHub storage for specifying branches other than master - #3638
• Added ExecuteNotebook task for running Jupyter notebooks - #3599
• Pass day_or croniter argument to CronClock and CronSchedule - #3612
• Client.create_project and prefect create project will skip creating the project if the project already exists - #3630
• Update deployments extension to AppsV1Api - #3637
• PrefectSecret and EnvVarSecret tasks no longer require secret names be provided at flow creation time - #3641

Fixes

• Fix issue with retrying mapped pipelines on dask - #3519
• Task arguments take precedence when generating task_run_name - #3605
• Fix breaking change in flow registration with old server versions - #3642
• Task arguments take precedence when generating templated targets and locations - #3627

Breaking Changes

• Environment variable config values now parse without requiring escaping backslashes - #3603

Contributors

• Amanda Wee
• Avi Aminov
• Brad McElroy
• Emilien Garreau
• Joël Luijmes
• Panagiotis Simakis

Now with twice as much Fargate

Changelog

0.13.14

Released on November 5, 2020.

Features

• flow.register accepts an idempotency key to prevent excessive flow versions from being created - #3590
• Added flow.serialized_hash() for easy generation of hash keys from the serialized flow - #3590

Enhancements

• Add option to select cursor_type for MySQLFetch task - #3574
• Add new ECSAgent and ECSRun run config - #3585
• Display exception information on prefect create project failure - #3589
• prefect diagnostics no longer displays keys that have values matching the default config - #3593
• Allow use of multiple image pull secrets in KubernetesAgent, DaskKubernetesEnvironment - #3596
• Added FROM to explicitly chain exceptions in src/prefect/tasks/twitter - #3602
• Add UTC offset to default logging.datefmt; logging timestamp converter now follows Python default behavior - #3607
• Improve error message when API responds with 400 status code - #3615

Deprecations

• Deprecate prefect agent start <kind> in favor of prefect agent <kind> start - #3610
• Deprecate prefect agent install <kind> in favor of prefect agent <kind> install - #3610

Contributors

• Billy McMonagle
• James Lamb
• Juan Calderon-Perez
• Michael Marinaccio

Please, call me `BlobServiceClient` - `BlockBlobService` was my father's name

Changelog

0.13.13

Released on October 27, 2020.

Enhancements

• Don't stop execution if the task runner fails to load a cached result - #3378
• Add option to specify networkMode for tasks created by the Fargate Agent - #3546
• Allows to schedule flow runs at an arbitrary time with StartFlowRun - #3573

Fixes

• Use BlobServiceClient instead of BlockBlobService to connect to azure blob in azure tasks - #3562
• Tasks with log_stdout=True work with non-utf8 output - #3563

Contributors

• Alessandro Lollo
• Kfir Stri
• Lukáš Novotný
• Natalie Smith
• Raphael Riel

There's so much in here, I don't have a name

Changelog

0.13.12

Released on October 20, 2020.

Enhancements

• Agents now submit flow runs in order of scheduled start times - #3165
• Updating k8s tutorial docs to include instructions on how to provide access to S3 from kubernetes deployments on AWS - #3200
• Adds option to specify default values for GetItem and GetAttr tasks - #3489
• Allow disabling default storage labels for the LocalAgent - #3503
• Improve overall functionality of docs search, full list of changes here - #3504
• Add LocalRun implementation for run_config based flows - #3527
• Add DockerRun implementation for run_config based flows - #3537
• Raise a better error message when trying to register a flow with a schedule using custom filter functions - #3450
• RenameFlowRunTask: use default flow_run_id value from context - #3548
• Raise a better error message when trying to register a flow with parameters with JSON-incompatible defaults - #3549

Task Library

• Extended GCSUpload task to allow uploading of bytes/gzip data - #3507
• Allow setting runtime webook_secret on SlackTask and kwarg for webhook_secret retrieved from PrefectSecret task - #3522

Fixes

• Fix get flow-runs and describe flow-runs CLI commands querying of removed duration field - #3517
• Fix multiprocess based timeout handler on linux - #3526
• Fix API doc generation incorrectly compiling mocked imports - #3504
• Fix multiprocessing scheduler failure while running tasks with timeouts - #3511
• Update Fargate task definition validation - #3514
• Fix bug in k8s where the resource-manager would sometimes silently crash on errors - #3521
• Add labels from flow.storage for run_config based flows - #3527
• Fix LocalAgent PYTHONPATH construction on Windows - #3551

Deprecations

• FlowRunTask, RenameFlowRunTask, and CancelFlowRunTask have been renamed to StartFlowRun, RenameFlowRun, and CancelFlowRun respectively - #3539

Contributors

• Michal Baumgartner
• Panagiotis Simakis
• Raphael Riel
• Spencer Ellinor
• andywaugh

More external contributors than Prefect has engineers

Changelog

0.13.11

Released on October 14, 2020.

Features

• Allow for schedules that emit custom Flow Run labels - #3483

Enhancements

• Use explicit exception chaining - #3306
• S3List filtering using the LastModified value - #3460
• Add Gitlab storage - #3461
• Extend module storage capabilities - #3463
• Support adding additional flow labels in prefect register flow - #3465
• Strict Type for default value of a Parameter - #3466
• Enable automatic script upload for file-based storage when using S3 and GCS - #3482
• Allow for passing labels to client.create_flow_run - #3483
• Display flow group ID in registration output URL instead of flow ID to avoid redirect in UI - #3500
• Add informative error log when local storage fails to load flow - #3475

Task Library

• Add cancel flow run task - #3484
• Add new BatchSubmit task for submitting jobs to AWS batch - #3366
• Add new AWSClientWait task for waiting on long-running AWS jobs - #3366
• Add GetAttr task - #3481

Fixes

• Fix default profile directory creation behavior - #3037
• Fix DaskKubernetesEnvironment overwriting log attributes for custom specs - #3231
• Fix default behavior for dbt_kwargs in the dbt task to provide an empty string - #3280
• Fix containerDefinitions environment validation - #3452
• Raise a better error when calling flow.register() from within a Flow context - #3467
• Fix task cancellation on Python 3.8 to properly interrupt long blocking calls - #3474

Contributors

• Aaron Richter
• Alessandro Lollo
• Bruno Casarotti
• Mariia Kerimova
• Max Del Giudice
• Michal Baumgartner
• Panagiotis Simakis
• Raphael Riel
• Shalika Singhal
• Zach McQuiston
• heyitskevin