Releases: PrefectHQ/prefect
Releases · PrefectHQ/prefect
Zero One Four! It's a big one!
Changelog
0.14.0
Released on December 16, 2020.
Features
- New flow run configuration system based on
RunConfig
types, see here for more info
Enhancements
- Kubernetes Agent now reports events for pending pods created by prefect jobs - #3783
- Using
--rbac
for Kubernetes Agent install command now includes theevents
resource - #3783 - Add orchestration-based dependencies to the
prefecthq/prefect
Docker image - #3804 - Add a slimmed down
prefecthq/prefect:core
Docker image that only contains base dependencies - #3804 - Docker storage now installs all orchestration-based dependencies when using default image - #3804
- Add warning on flow registration if
flow.executor
is set but the flow is using the legacyflow.environment
configuration system - #3808 - Echoing prefect config file as JSON to be able to parse it with jq in the terminal - #3818
- Produce artifact for RunGreatExpectationsValidation even if validation fails - #3829
execute flow-run
command now sends flow run log in the case of an error - #3832- Changed name of logs raised by the Kubernetes Agent if they stem from infrastructure events - #3832
- Add
tini
to the official Prefect docker images - #3839 - Remove task run level heartbeats for performance - #3842
Task Library
- Fix mising
job_id
inDatabricksRunNow
task initialization - #3793
Fixes
- Fix Azure result byte decoding of blob data - #3846
- Prefect kubernetes agent no longer relies on existence of any fields in configured Kubernetes Job Template - #3805
- Accept old envvar style configuration for Kubernetes agent for
--service-account-name
/--image-pull-secrets
options - #3814 - Pass
as_user=False
when usingclient.get_cloud_url
inStartFlowRun
- #3850 - Fix AWS boto3 utility passing duplicate kwargs to client initialization - #3857
Deprecations
- Storage classes have been moved from
prefect.environments.storage
toprefect.storage
, the old import paths have been deprecated accordingly - #3796 - Executor classes have been moved from
prefect.engine.executors
toprefect.executors
, the old import paths have been deprecated accordingly - #3798 - Deprecated use of
storage_labels
boolean kwarg on local agent - #3800 - Deprecated use of
--storage-labels
option from agentstart
CLI command - #3800 - Deprecates all
Environment
classes - users should transition to settingflow.run_config
instead offlow.environment
- #3811 - Deprecate the Fargate Agent in favor of the ECS Agent - #3812
Breaking Changes
- Using in-cluster installs of the Kubernetes Agent now requires RBAC for the
events
resource - #3783 - Removed setting of default labels on storage objects and the local agent - #3800
- Remove deprecated
RemoteEnvironment
/RemoteDaskEnvironment
- #3802 - Remove deprecated
executor_kwargs
argument toKubernetesJobEnvironment
/FargateTaskEnvironment
- #3802 - Remove deprecated
prefect run cloud
/prefect run server
CLI commands - #3803 - Remove deprecated
prefect execute cloud-flow
CLI command - #3803 - Stop building the
prefecthq/prefect:all_extras
image and switch flow deployment default to usingprefecthq/prefect:{core_version}
- #3804 - Flows now use
RunConfig
based deployments by default - legacyEnvironment
based deployments are now opt-in only - #3806 - Remove deprecated
prefect.contrib
module - #3813 - Remove all references to result handlers and safe results - #3838
- Remove option to enable deprecated Kubernetes resource manager in agent install CLI command - #3840
Contributors
It was the best of releases...
Changelog
0.13.19
Released on December 8, 2020.
Enhancements
- Use explicit exception chaining - #3306
- Support Bitbucket as storage option - #3711
- Surface pod failures and container errors in jobs deployed with the Kubernetes Agent - 3747
- Support timeout option in GCS tasks - #3732
- Added storage option for AWS CodeCommit - #3733
- Add the image used for a flow-run to the flow run environment as
prefect.context.image
- #3746 - Add
UniversalRun
run-config that works with all agents - #3750 - Support flows that have no run-config or environment - #3750
- Allow Docker storage environment vars to be used in commands - #3755
- Add
service_account_name
andimage_pull_secrets
options toKubernetesRun
andKubernetesAgent
- #3778 - Add a new Client function
delete_project
- #3728
Task Library
- Add task to fetch data from Dremio - #3734
- Add
RunGreatExpectationsValidation
task - #3753 - Add the option to post markdown artifacts from the
RunGreatExpectationsValidation
task - #3753
Fixes
- Fix state attempting to read result from absent upstream result - #3618
- Replace deprecated download_as_string method with download_as_bytes method - #3741
- Fix default image whenever working on a non-tagged commit - #3748
- Fix type-casting for task timeout defaults loaded from config - #3761
- Fix the
ref
default on GitHub storage - #3764 - Fix rare cancellation bug when running with external Dask cluster - #3770
Deprecations
- Deprecated the
RunGreatExpectationsCheckpoint
task in favor ofRunGreatExpectationsValidation
- #3766
Contributors
It's beginning to look a lot like 0.13.18
Changelog
0.13.18
Released on November 30, 2020.
Enhancements
- Display formatted graphql errors on client request failure - #3632
- Refactor Core Client API calls for performance - #3730
Task Library
- Refactor execute query code for
PostgresExecute
,PostgresExecuteMany
, andPostgresFetch
tasks - #3714 - Fix
PicklingError
inBigQueryLoadFile
andBigQueryLoadGoogleCloudStorage
- #3724 - Allow custom exporter for
ExecuteNotebook
task - #3725 - Properly forward
location
parameter in bigquery tasks - #3726 - Support passing
helper_script
toShellTask
/DBTShellTask
at runtime - #3729
Fixes
- Fix bug with docker storage throwing exception while trying to display output - #3717
Contributors
Removing defects from Prefect to great effect
Changelog
0.13.17
Released on November 24, 2020.
Features
- Improved support for Tasks returning multiple results - #3697
Enhancements
- Allow chaining of
Task
imperative dependency calls - #3696 - Add
task_definition_arn
toECSRun
run-config - #3681 - Rerun
resource_manager
tasks when restarting flows from failed - #3689 - Raise nice warning if user passes
Task
instance toTask
constructor, rather than when calling theTask
(or usingTask.map
/Task.set_dependencies
) - #3691 - Always use tenant slug in output of Client
get_cloud_url
function - #3692
Task Library
- Add enhancement to
StartFlowRun
task to create link artifact for started flow run - #3692 - Add a new postgres task
PostgresExecuteMany
- #3703 - Add debug logging for Docker tasks
PullImage
andBuildImage
- #3672 ShellTask
returns output on failure - #3649ShellTask
allows streaming of output independently of the number of lines returned - #3649
Fixes
- Make
serialized_hash
handle unordered task sets correctly - #3682 - Docker storage build error logs were not always displayed - #3693
- Fix automatic quoting of Docker storage environment variable values - #3694
- Use
exist_ok
flag inos.makedirs
to avoid race condition in local storage class - #3679
Contributors
Remember when Python 2 was a thing?
Why dig up artifacts when you can create them?
Changelog
0.13.15
Released on November 11, 2020.
Features
- Add API for storing task run artifacts in the backend - #3581
Enhancements
- Allow for setting
Client
headers before loading tenant when running with Prefect Server - #3515 - Checkpoint all iterations of Looped tasks - #3619
- Add
ref
option to GitHub storage for specifying branches other than master - #3638 - Added
ExecuteNotebook
task for running Jupyter notebooks - #3599 - Pass
day_or
croniter argument to CronClock and CronSchedule - #3612 Client.create_project
andprefect create project
will skip creating the project if the project already exists - #3630- Update deployments extension to AppsV1Api - #3637
PrefectSecret
andEnvVarSecret
tasks no longer require secret names be provided at flow creation time - #3641
Fixes
- Fix issue with retrying mapped pipelines on dask - #3519
- Task arguments take precedence when generating
task_run_name
- #3605 - Fix breaking change in flow registration with old server versions - #3642
- Task arguments take precedence when generating templated targets and locations - #3627
Breaking Changes
- Environment variable config values now parse without requiring escaping backslashes - #3603
Contributors
Now with twice as much Fargate
Changelog
0.13.14
Released on November 5, 2020.
Features
flow.register
accepts an idempotency key to prevent excessive flow versions from being created - #3590- Added
flow.serialized_hash()
for easy generation of hash keys from the serialized flow - #3590
Enhancements
- Add option to select
cursor_type
for MySQLFetch task - #3574 - Add new
ECSAgent
andECSRun
run config - #3585 - Display exception information on
prefect create project
failure - #3589 prefect diagnostics
no longer displays keys that have values matching the default config - #3593- Allow use of multiple image pull secrets in
KubernetesAgent
,DaskKubernetesEnvironment
- #3596 - Added FROM to explicitly chain exceptions in src/prefect/tasks/twitter - #3602
- Add UTC offset to default logging.datefmt; logging timestamp converter now follows Python default behavior - #3607
- Improve error message when API responds with 400 status code - #3615
Deprecations
- Deprecate
prefect agent start <kind>
in favor ofprefect agent <kind> start
- #3610 - Deprecate
prefect agent install <kind>
in favor ofprefect agent <kind> install
- #3610
Contributors
Please, call me `BlobServiceClient` - `BlockBlobService` was my father's name
Changelog
0.13.13
Released on October 27, 2020.
Enhancements
- Don't stop execution if the task runner fails to load a cached result - #3378
- Add option to specify
networkMode
for tasks created by the Fargate Agent - #3546 - Allows to schedule flow runs at an arbitrary time with StartFlowRun - #3573
Fixes
- Use
BlobServiceClient
instead ofBlockBlobService
to connect to azure blob in azure tasks - #3562 - Tasks with
log_stdout=True
work with non-utf8 output - #3563
Contributors
There's so much in here, I don't have a name
Changelog
0.13.12
Released on October 20, 2020.
Enhancements
- Agents now submit flow runs in order of scheduled start times - #3165
- Updating k8s tutorial docs to include instructions on how to provide access to S3 from kubernetes deployments on AWS - #3200
- Adds option to specify default values for GetItem and GetAttr tasks - #3489
- Allow disabling default storage labels for the
LocalAgent
- #3503 - Improve overall functionality of docs search, full list of changes here - #3504
- Add
LocalRun
implementation forrun_config
based flows - #3527 - Add
DockerRun
implementation forrun_config
based flows - #3537 - Raise a better error message when trying to register a flow with a schedule using custom filter functions - #3450
RenameFlowRunTask
: use defaultflow_run_id
value from context - #3548- Raise a better error message when trying to register a flow with parameters with JSON-incompatible defaults - #3549
Task Library
- Extended
GCSUpload
task to allow uploading of bytes/gzip data - #3507 - Allow setting runtime
webook_secret
onSlackTask
and kwarg forwebhook_secret
retrieved fromPrefectSecret
task - #3522
Fixes
- Fix
get flow-runs
anddescribe flow-runs
CLI commands querying of removedduration
field - #3517 - Fix multiprocess based timeout handler on linux - #3526
- Fix API doc generation incorrectly compiling mocked imports - #3504
- Fix multiprocessing scheduler failure while running tasks with timeouts - #3511
- Update Fargate task definition validation - #3514
- Fix bug in k8s where the resource-manager would sometimes silently crash on errors - #3521
- Add labels from
flow.storage
forrun_config
based flows - #3527 - Fix LocalAgent PYTHONPATH construction on Windows - #3551
Deprecations
FlowRunTask
,RenameFlowRunTask
, andCancelFlowRunTask
have been renamed toStartFlowRun
,RenameFlowRun
, andCancelFlowRun
respectively - #3539
Contributors
More external contributors than Prefect has engineers
Changelog
0.13.11
Released on October 14, 2020.
Features
- Allow for schedules that emit custom Flow Run labels - #3483
Enhancements
- Use explicit exception chaining - #3306
- S3List filtering using the LastModified value - #3460
- Add Gitlab storage - #3461
- Extend module storage capabilities - #3463
- Support adding additional flow labels in
prefect register flow
- #3465 - Strict Type for default value of a Parameter - #3466
- Enable automatic script upload for file-based storage when using S3 and GCS - #3482
- Allow for passing labels to
client.create_flow_run
- #3483 - Display flow group ID in registration output URL instead of flow ID to avoid redirect in UI - #3500
- Add informative error log when local storage fails to load flow - #3475
Task Library
- Add cancel flow run task - #3484
- Add new
BatchSubmit
task for submitting jobs to AWS batch - #3366 - Add new
AWSClientWait
task for waiting on long-running AWS jobs - #3366 - Add GetAttr task - #3481
Fixes
- Fix default profile directory creation behavior - #3037
- Fix
DaskKubernetesEnvironment
overwriting log attributes for custom specs - #3231 - Fix default behavior for
dbt_kwargs
in the dbt task to provide an empty string - #3280 - Fix containerDefinitions environment validation - #3452
- Raise a better error when calling
flow.register()
from within aFlow
context - #3467 - Fix task cancellation on Python 3.8 to properly interrupt long blocking calls - #3474