Skip to content
master
Switch branches/tags
Code

Latest commit

* Placeholder commit for `run-flow-api` feature branch

* Run Flow API: Add objects for interacting with flows, flow runs, and task runs (#4426)

* Sketch run with backend flow method

* Add `with_backend` and `with_agent` to `flow.run`

* Rough draft of FlowRunResult and TaskRunResult

* Move from tasks -> run_results

* Rehaul implemention, dropping with_agent

* Fix log level for failure?

* Add exception for null flow run ids, log final state

* Add docstring; feed mypy; remove `with_cloud` from `flow.run()`

* Remove duplicate file

* Fix query for static tasks

* Return `FlowRunResult` from `cloud_run`

* Implement TaskRunResult queries, FlowRunResult.task_rn_ids, FlowRunResult.get_all()

* Add `get_mapped`; consolidate `TaskRunResult` instantiation

* Implement result loading; some fixes as well

* Implement mapped task result hydration; replace get_mapped with iter_mapped

* Add handling for passed tasks without slugs

* Add `run_flow` task

* Move `run_results` to `prefect.api`

* Refactor `Flow/TaskRunResult` into `Flow/TaskRun`

* Additional refactoring of variable names; add update() methods

* Refactor from prefect.api -> prefect.backend to avoid colission with the API plugin

* Add `execute_flow_run` implementation

* Unpack task run data to initalize new TaskRun

* Add `prefect.backend.flow.FlowMetadata`

* Replace `flow.cloud_run` internals with `execute_flow_run`

* Cleanup, mostly logging

* Allow flow metadata instantiation from flow name; add docstring to execute_flow_run; skip static task run loading for flow run state check

* Add `FlowMetadata.from_flow_obj` to lookup by serialized flow

* Require `flow_id` to be passed

* Add some TODO notes

* Import required types

* Add kwarg passing to flow run execution

* Cleanup jsonb variable passing; add docstrings

* Add docstrings to `FlowRun`

* Cleanup reprs and add repr to `FlowMetadata`

* Add some docstrings to `TaskRun`

* Add rough `create_flow_run`

* Fix task caching; would never return cached task

* Add changes/pr4426.yaml

* Cleanup some docs mistakes

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Add repr for EnumValue

* Update get-like methods to `get_task_run`

* Rename cached task vars/methods for clarity

* Do not query for already cached tasks in `get_all_task_runs`

* Remove extraneous `

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Feed mypy; refactor queries in `FlowMetadata` and `TaskRun` to be clearer

* Feed flake8

* Rename backend objects to "Views"

* Rename `update` to `get_latest`

* Fix import

* Rename `TaskRun` to `TaskRunView` in flow_run.py

* Add tests for `TaskRunView`

* Fix repr; simplify empty case; copy data to avoid mutation during `from_task_run_data`

* Add tests for FlowRunView -- excluding get_task_run for now

* Add eq operator to TaskRunView

* Move query to static function in FlowRunView to match TaskRunView

* Add eq operator to EnumValue

Needed for mock assertions to work

* Return instead of raise NotImplemented as fallback to reverse operator

* Add test for mapped task results

* Fix "FlowRunViewner" -> "FlowRunner" bad replace

* Fix bug where final state would be unpopulated

* Fix eq types and add explicit hash to EnumValue

* Add fix for double written results

* Remove `backend.create_flow_run`

Will wait until it is moved out of the client

* Fix `from_flow_name` query

`updated_at` does not exist

* Add `FlowView.from_flow_group_id`

* Add context, parameters, labels to `FlowRunView`

* Add `updated_at` to `FlowRunView`

* Add `from_flow_run_data` for consistency with other views

* Add test coverage for params, context, labels, datetime

* Update `FlowRunView` tests for cherry-picked commits

* Happy mypy

* Fix mutated test data

* Make `from_flow_obj` compatible with `prefect register`

* Add py3.6 fix

* Pass the flow object during creation from flow

* Fixup 3.6 compat change

* Fixup version comparison

* Happy mypy

* Drop keyboard interrupt handling

* Add API docs pages

* Remove `from_flow_obj`

* Make `query_for_flows` private

* Make `query_for_flow_run` private

* Remove public view of `cached_task_runs`

* Make `query_for_task_runs` private

* Drop `TaskRunView.get_latest`

* Add additional notes to `FlowRunView.get_latest()`

* Add tests for `FlowView`

* Deserialize the flow run config; copy input data before mutation

* Make `from_task_run_data` private; add docstrings

* Make from_flow_run_data private; update docstring

* Make from_flow_data private; update docstring

* Update "Attributes" docstring to "Args/Properties"

* Fix bad references

* Fixup args formatting, adding leading -

* Remove unused import

* Add missing args to `FlowRunView` docstring

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Move mapped task interation from `FlowRunView` to `TaskRunView` (#4530)

* Fix missing newline in repr

* Allow map index to be passed to `TaskRunView.from_task_slug`

* Move mapped task iteration into `TaskRunView`

* Add some comments

* Rename `TaskRunView.result` to `TaskRunView.get_result()` (#4542)

* Run Flow API: Rehaul `prefect run` CLI (#4463)

* Sketch run with backend flow method

* Add `with_backend` and `with_agent` to `flow.run`

* Rough draft of FlowRunResult and TaskRunResult

* Move from tasks -> run_results

* Rehaul implemention, dropping with_agent

* Fix log level for failure?

* Add exception for null flow run ids, log final state

* Add docstring; feed mypy; remove `with_cloud` from `flow.run()`

* Remove duplicate file

* Fix query for static tasks

* Return `FlowRunResult` from `cloud_run`

* Implement TaskRunResult queries, FlowRunResult.task_rn_ids, FlowRunResult.get_all()

* Add `get_mapped`; consolidate `TaskRunResult` instantiation

* Implement result loading; some fixes as well

* Implement mapped task result hydration; replace get_mapped with iter_mapped

* Add handling for passed tasks without slugs

* Add `run_flow` task

* Move `run_results` to `prefect.api`

* Refactor `Flow/TaskRunResult` into `Flow/TaskRun`

* Additional refactoring of variable names; add update() methods

* Refactor from prefect.api -> prefect.backend to avoid colission with the API plugin

* Add `execute_flow_run` implementation

* Unpack task run data to initalize new TaskRun

* Add `prefect.backend.flow.FlowMetadata`

* Replace `flow.cloud_run` internals with `execute_flow_run`

* Cleanup, mostly logging

* Allow flow metadata instantiation from flow name; add docstring to execute_flow_run; skip static task run loading for flow run state check

* Add `FlowMetadata.from_flow_obj` to lookup by serialized flow

* Require `flow_id` to be passed

* Add some TODO notes

* Import required types

* Add kwarg passing to flow run execution

* Cleanup jsonb variable passing; add docstrings

* Add docstrings to `FlowRun`

* Cleanup reprs and add repr to `FlowMetadata`

* Add some docstrings to `TaskRun`

* Add rough `create_flow_run`

* Fix task caching; would never return cached task

* Add changes/pr4426.yaml

* Cleanup some docs mistakes

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Add repr for EnumValue

* Update get-like methods to `get_task_run`

* Rename cached task vars/methods for clarity

* Do not query for already cached tasks in `get_all_task_runs`

* Remove extraneous `

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Feed mypy; refactor queries in `FlowMetadata` and `TaskRun` to be clearer

* Feed flake8

* Rename backend objects to "Views"

* Rename `update` to `get_latest`

* Fix import

* Rename `TaskRun` to `TaskRunView` in flow_run.py

* Add tests for `TaskRunView`

* Fix repr; simplify empty case; copy data to avoid mutation during `from_task_run_data`

* Add tests for FlowRunView -- excluding get_task_run for now

* Add eq operator to TaskRunView

* Move query to static function in FlowRunView to match TaskRunView

* Add eq operator to EnumValue

Needed for mock assertions to work

* Return instead of raise NotImplemented as fallback to reverse operator

* Add test for mapped task results

* Fix "FlowRunViewner" -> "FlowRunner" bad replace

* Fix bug where final state would be unpopulated

* Fix eq types and add explicit hash to EnumValue

* Add fix for double written results

* WIP

* Remove `backend.create_flow_run`

Will wait until it is moved out of the client

* Pass the flow object during creation from flow

* First pass implementation for `prefect run`

* Add `watch_flow_run`

* Allow `query_for_flow` to grab the first of many results when order_by is passed

There's no reason to pass order_by to this function if it'll always error on multiple results. This behavior arguably just makes sense.

* Fix `from_flow_name` query

`updated_at` does not exist

* Add `FlowView.from_flow_group_id`

* Add `watch_flow_run`

* Fix minor bugs and add more output

* Add flow group id note

* Add flow group labels to `FlowView`

* Fix typo

* Add flow group labels to runs

This appears to be unecessary and I need to sort out what is going on in the backend, probably going to revert

* Fix no-agent label, fix some echos

* Fix default labels by passing `None`

* Add context, parameters, labels to `FlowRunView`

* Beautify output; display run info instead of passed options

* Fix `prefect run flow`

* Add `updated_at` to `FlowRunView`

* Add `from_flow_run_data` for consistency with other views

* Allow lookup by just "name"

* Allow an `outputter` to be passed to `watch_flow_run`

* Add log display to `watch_flow_run`

* Display all states; rollup with logs; show logs by default

* Rename outputter

* Implement a basic poll interval backoff

* Sort logs by timestamp before level

* Rename `--wait` to `--watch`

* Set the logging level on local runs as well

* Add changes/pr4463.yaml

* Drop error traceback on `get_flow_view` failure

* Move "Done" printout to after pulling for clarity; skip query when quiet

* Make `from_flow_obj` compatible with `prefect register`

* Set poll_interval incase there are no message son first poll

* Optimize imports

* Add deprecated comment

* Add CLI examples for `prefect run`

* Collapse `flow-group-id` and `flow-id` options

* Rehaul context/parameter parsing with error handling and new format

* Add Server line reference

* Cleanup `--watch` docs and change `--no-backend` to `--local`

* Display a message if the flow run is already done

* Improve `--local` execution

- Add `try_error_done` utility
- Renames `--path` to `--file/-f`
- Allows local runs to load flows from backend storage
- Renames `--no-agent` to `--execute/-e`
- Adds TerminalError handling/usage

* Sort imports

* Add labels to agent warning; improve cancel ctrl-c message

* Update examples

* Fix `flow` -> `flow_view`

* Add `RunConfig` to `FlowRunView`

* Set log level and environment variables during agentless execution

* WIP

* Add test coverage for FlowView/FlowRunView additions

* Remove agentless and local execution flags
- path/module flows always run local
- cloud flows cannot be run local

* Move back from `--file` to `--path`

* Update examples to match drop of local/execute

* Fix submodule imports in `get_object`

* Fix type-checking

* Sort imports

* Lint after execute/local flag removal

* Remove whitespace on blank line

* Update docs reference

* Cleanup and move old `prefect run flow` tests

* Do not display the traceback for click errors

* Create copies of the `build_register` loader funcs

These do not set flow storage like the `build_register` ones do and raise errors that can be consistently handled by `try_error_done` instead of directly echoing tracebacks

* Raise `TerminalError` consistently

* Display the traceback on retrieval for unhandled exceptions

* Fix flow configuration newlines

* Cleanup imports

* Write tests for `prefect run` local flow runs

* Add check for the correct exception before silencing

* Add docstrings to new `FlowView` and `FlowRunView` objects

* Add missing `:` and `-`

* Privatize `TimestampedState`

Required to pass State subclass tests and makes sense anyway

* Remove unused import

* Skip test on windows

Likely an issue with the path check in the exception string seems unworthy of complicating since the full traceback is just shown

* Add log formatting to local runs

* Add handler for empty null formatter

* Remove unused import

* Raise TerminalError instead of ClickException

* Error when `--id` and `--name` are passed

* Wrap parameter file loading exceptions

* Add param/context tests

* Cleanup terminal error printing

* Wrap the location name for readability

* Wrap the flow group id in quotes when missing

* Use the already deserialized run confiig

* Add equality dunder to base `RunConfig`

* Add tests for cloud runs

* Cleanup unused items

* Switch to Py 3.6/7 syntax for call_args assertion

* Add debug print for appveyor assertion failure

* Try a repr instead of a str

* Cleanup

* Fixup no labels display

* Clarify log level flag default

* Shorten examaple line to fit more shells [ci skip]

* Add comment to test [ci skip]

* Use monkeypatch to set the sys path

* Make `quiet_echo` robust to mutation

* Explain flow runner exceptions in echo

* Update comment explaining skipped `Done`

* Handle keyboard interrupts during creation

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Update agent warning to include query for compatible agents (#4554)

* Sketch run with backend flow method

* Add `with_backend` and `with_agent` to `flow.run`

* Rough draft of FlowRunResult and TaskRunResult

* Move from tasks -> run_results

* Rehaul implemention, dropping with_agent

* Fix log level for failure?

* Add exception for null flow run ids, log final state

* Add docstring; feed mypy; remove `with_cloud` from `flow.run()`

* Remove duplicate file

* Fix query for static tasks

* Return `FlowRunResult` from `cloud_run`

* Implement TaskRunResult queries, FlowRunResult.task_rn_ids, FlowRunResult.get_all()

* Add `get_mapped`; consolidate `TaskRunResult` instantiation

* Implement result loading; some fixes as well

* Implement mapped task result hydration; replace get_mapped with iter_mapped

* Add handling for passed tasks without slugs

* Add `run_flow` task

* Move `run_results` to `prefect.api`

* Refactor `Flow/TaskRunResult` into `Flow/TaskRun`

* Additional refactoring of variable names; add update() methods

* Refactor from prefect.api -> prefect.backend to avoid colission with the API plugin

* Add `execute_flow_run` implementation

* Unpack task run data to initalize new TaskRun

* Add `prefect.backend.flow.FlowMetadata`

* Replace `flow.cloud_run` internals with `execute_flow_run`

* Cleanup, mostly logging

* Allow flow metadata instantiation from flow name; add docstring to execute_flow_run; skip static task run loading for flow run state check

* Add `FlowMetadata.from_flow_obj` to lookup by serialized flow

* Require `flow_id` to be passed

* Add some TODO notes

* Import required types

* Add kwarg passing to flow run execution

* Cleanup jsonb variable passing; add docstrings

* Add docstrings to `FlowRun`

* Cleanup reprs and add repr to `FlowMetadata`

* Add some docstrings to `TaskRun`

* Add rough `create_flow_run`

* Fix task caching; would never return cached task

* Add changes/pr4426.yaml

* Cleanup some docs mistakes

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Add repr for EnumValue

* Update get-like methods to `get_task_run`

* Rename cached task vars/methods for clarity

* Do not query for already cached tasks in `get_all_task_runs`

* Remove extraneous `

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Feed mypy; refactor queries in `FlowMetadata` and `TaskRun` to be clearer

* Feed flake8

* Rename backend objects to "Views"

* Rename `update` to `get_latest`

* Fix import

* Rename `TaskRun` to `TaskRunView` in flow_run.py

* Add tests for `TaskRunView`

* Fix repr; simplify empty case; copy data to avoid mutation during `from_task_run_data`

* Add tests for FlowRunView -- excluding get_task_run for now

* Add eq operator to TaskRunView

* Move query to static function in FlowRunView to match TaskRunView

* Add eq operator to EnumValue

Needed for mock assertions to work

* Return instead of raise NotImplemented as fallback to reverse operator

* Add test for mapped task results

* Fix "FlowRunViewner" -> "FlowRunner" bad replace

* Fix bug where final state would be unpopulated

* Fix eq types and add explicit hash to EnumValue

* Add fix for double written results

* WIP

* Remove `backend.create_flow_run`

Will wait until it is moved out of the client

* Pass the flow object during creation from flow

* First pass implementation for `prefect run`

* Add `watch_flow_run`

* Allow `query_for_flow` to grab the first of many results when order_by is passed

There's no reason to pass order_by to this function if it'll always error on multiple results. This behavior arguably just makes sense.

* Fix `from_flow_name` query

`updated_at` does not exist

* Add `FlowView.from_flow_group_id`

* Add `watch_flow_run`

* Fix minor bugs and add more output

* Add flow group id note

* Add flow group labels to `FlowView`

* Fix typo

* Add flow group labels to runs

This appears to be unecessary and I need to sort out what is going on in the backend, probably going to revert

* Fix no-agent label, fix some echos

* Fix default labels by passing `None`

* Add context, parameters, labels to `FlowRunView`

* Beautify output; display run info instead of passed options

* Fix `prefect run flow`

* Add `updated_at` to `FlowRunView`

* Add `from_flow_run_data` for consistency with other views

* Allow lookup by just "name"

* Allow an `outputter` to be passed to `watch_flow_run`

* Add log display to `watch_flow_run`

* Display all states; rollup with logs; show logs by default

* Rename outputter

* Implement a basic poll interval backoff

* Sort logs by timestamp before level

* Rename `--wait` to `--watch`

* Set the logging level on local runs as well

* Add changes/pr4463.yaml

* Drop error traceback on `get_flow_view` failure

* Move "Done" printout to after pulling for clarity; skip query when quiet

* Make `from_flow_obj` compatible with `prefect register`

* Set poll_interval incase there are no message son first poll

* Optimize imports

* Add deprecated comment

* Add CLI examples for `prefect run`

* Collapse `flow-group-id` and `flow-id` options

* Rehaul context/parameter parsing with error handling and new format

* Add Server line reference

* Cleanup `--watch` docs and change `--no-backend` to `--local`

* Display a message if the flow run is already done

* Improve `--local` execution

- Add `try_error_done` utility
- Renames `--path` to `--file/-f`
- Allows local runs to load flows from backend storage
- Renames `--no-agent` to `--execute/-e`
- Adds TerminalError handling/usage

* Sort imports

* Add labels to agent warning; improve cancel ctrl-c message

* Update examples

* Fix `flow` -> `flow_view`

* Add `RunConfig` to `FlowRunView`

* Set log level and environment variables during agentless execution

* WIP

* Add test coverage for FlowView/FlowRunView additions

* Remove agentless and local execution flags
- path/module flows always run local
- cloud flows cannot be run local

* Move back from `--file` to `--path`

* Update examples to match drop of local/execute

* Fix submodule imports in `get_object`

* Fix type-checking

* Sort imports

* Lint after execute/local flag removal

* Remove whitespace on blank line

* Update docs reference

* Cleanup and move old `prefect run flow` tests

* Do not display the traceback for click errors

* Create copies of the `build_register` loader funcs

These do not set flow storage like the `build_register` ones do and raise errors that can be consistently handled by `try_error_done` instead of directly echoing tracebacks

* Raise `TerminalError` consistently

* Display the traceback on retrieval for unhandled exceptions

* Fix flow configuration newlines

* Cleanup imports

* Write tests for `prefect run` local flow runs

* Add check for the correct exception before silencing

* Add docstrings to new `FlowView` and `FlowRunView` objects

* Add missing `:` and `-`

* Privatize `TimestampedState`

Required to pass State subclass tests and makes sense anyway

* Remove unused import

* Skip test on windows

Likely an issue with the path check in the exception string seems unworthy of complicating since the full traceback is just shown

* Add log formatting to local runs

* Add handler for empty null formatter

* Remove unused import

* Raise TerminalError instead of ClickException

* Error when `--id` and `--name` are passed

* Wrap parameter file loading exceptions

* Add param/context tests

* Cleanup terminal error printing

* Wrap the location name for readability

* Wrap the flow group id in quotes when missing

* Use the already deserialized run confiig

* Add equality dunder to base `RunConfig`

* Add tests for cloud runs

* Cleanup unused items

* Switch to Py 3.6/7 syntax for call_args assertion

* Add debug print for appveyor assertion failure

* Try a repr instead of a str

* Cleanup

* Fixup no labels display

* Update agent warning to include query for compatible agents

* Remove unused import

* Add tests; cover more cases verbosely

* Fixup first agent warning timing; round to nearest 5 seconds for display

First warning would not show until 30 seconds rather than 5

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Run Flow API: Update flow run documentation with new dedicated section (#4492)

* WIP

* Remove outdated concurrency slots note

* Write overview and outline of new flow-run docs

* Add scheduling docs

* Move task concurrency limits

* Update feature tier note [ci skip]

* Add changes/pr4492.yaml [ci skip]

* Add more outline; change `Prefect Client` sections to `Prefect library`

* Add more outline; add next steps; shorten some lines

* Move everything out of the flow run concept page

* Fix png links

* remove concept from config.js

* flow_run -> flow-runs, "Task Runs" -> "Inspecting task runs"

* Limit sidebar depth

* Upgrade tab package

* Use tabs in flow run creation settings

* Continue to struggle with overview sidebar depth

* Cleanup task concurrency limits and try tabs in one section

* Continue to struggle with sidebar depth?

* Add more tabs to the task concurrency doc

* Add --watch and flow run name to flow run creation docs

* Add flow-concurrency UI images

* Write flow concurrency limit docs

* Drop agentless execution

* Add `prefect run --help` note

* Put an example UUID in example commands

* WIP

* Add local run blurb

* Fixup overview links

* Write `FlowRunView` docs

* Add graphql flow run query examples

* Add state setting doc to sidebar

* Add task run graphql queries

* Write task runs doc

* Update use of `get_result()` in docs

Co-authored-by: Zach Angell <zachary.james.angell@gmail.com>

* Drop task object slug inference from `get_task_run` (#4579)

* `prefect run`: Add automatic quoting for JSON inputs (#4578)

* Add automatic quoting for JSON strings

* Update example to use unquoted string

* Add test for forced string param

* Add type annotation [ci skip]

* Run Flow API: Add flow run tasks to simplify flow run result passing (#4563)

* Sketch run with backend flow method

* Add `with_backend` and `with_agent` to `flow.run`

* Rough draft of FlowRunResult and TaskRunResult

* Move from tasks -> run_results

* Rehaul implemention, dropping with_agent

* Fix log level for failure?

* Add exception for null flow run ids, log final state

* Add docstring; feed mypy; remove `with_cloud` from `flow.run()`

* Remove duplicate file

* Fix query for static tasks

* Return `FlowRunResult` from `cloud_run`

* Implement TaskRunResult queries, FlowRunResult.task_rn_ids, FlowRunResult.get_all()

* Add `get_mapped`; consolidate `TaskRunResult` instantiation

* Implement result loading; some fixes as well

* Implement mapped task result hydration; replace get_mapped with iter_mapped

* Add handling for passed tasks without slugs

* Add `run_flow` task

* Move `run_results` to `prefect.api`

* Refactor `Flow/TaskRunResult` into `Flow/TaskRun`

* Additional refactoring of variable names; add update() methods

* Refactor from prefect.api -> prefect.backend to avoid colission with the API plugin

* Add `execute_flow_run` implementation

* Unpack task run data to initalize new TaskRun

* Add `prefect.backend.flow.FlowMetadata`

* Replace `flow.cloud_run` internals with `execute_flow_run`

* Cleanup, mostly logging

* Allow flow metadata instantiation from flow name; add docstring to execute_flow_run; skip static task run loading for flow run state check

* Add `FlowMetadata.from_flow_obj` to lookup by serialized flow

* Require `flow_id` to be passed

* Add some TODO notes

* Import required types

* Add kwarg passing to flow run execution

* Cleanup jsonb variable passing; add docstrings

* Add docstrings to `FlowRun`

* Cleanup reprs and add repr to `FlowMetadata`

* Add some docstrings to `TaskRun`

* Add rough `create_flow_run`

* Fix task caching; would never return cached task

* Add changes/pr4426.yaml

* Cleanup some docs mistakes

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Add repr for EnumValue

* Update get-like methods to `get_task_run`

* Rename cached task vars/methods for clarity

* Do not query for already cached tasks in `get_all_task_runs`

* Remove extraneous `

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Feed mypy; refactor queries in `FlowMetadata` and `TaskRun` to be clearer

* Feed flake8

* Rename backend objects to "Views"

* Rename `update` to `get_latest`

* Fix import

* Rename `TaskRun` to `TaskRunView` in flow_run.py

* Add tests for `TaskRunView`

* Fix repr; simplify empty case; copy data to avoid mutation during `from_task_run_data`

* Add tests for FlowRunView -- excluding get_task_run for now

* Add eq operator to TaskRunView

* Move query to static function in FlowRunView to match TaskRunView

* Add eq operator to EnumValue

Needed for mock assertions to work

* Return instead of raise NotImplemented as fallback to reverse operator

* Add test for mapped task results

* Fix "FlowRunViewner" -> "FlowRunner" bad replace

* Fix bug where final state would be unpopulated

* Fix eq types and add explicit hash to EnumValue

* Add fix for double written results

* WIP

* Remove `backend.create_flow_run`

Will wait until it is moved out of the client

* Pass the flow object during creation from flow

* First pass implementation for `prefect run`

* Add `watch_flow_run`

* Allow `query_for_flow` to grab the first of many results when order_by is passed

There's no reason to pass order_by to this function if it'll always error on multiple results. This behavior arguably just makes sense.

* Fix `from_flow_name` query

`updated_at` does not exist

* Add `FlowView.from_flow_group_id`

* Add `watch_flow_run`

* Fix minor bugs and add more output

* Add flow group id note

* Add flow group labels to `FlowView`

* Fix typo

* Add flow group labels to runs

This appears to be unecessary and I need to sort out what is going on in the backend, probably going to revert

* Fix no-agent label, fix some echos

* Fix default labels by passing `None`

* Add context, parameters, labels to `FlowRunView`

* Beautify output; display run info instead of passed options

* Fix `prefect run flow`

* Add `updated_at` to `FlowRunView`

* Add `from_flow_run_data` for consistency with other views

* Allow lookup by just "name"

* Allow an `outputter` to be passed to `watch_flow_run`

* Add log display to `watch_flow_run`

* Display all states; rollup with logs; show logs by default

* Rename outputter

* Implement a basic poll interval backoff

* Sort logs by timestamp before level

* Rename `--wait` to `--watch`

* Set the logging level on local runs as well

* Add changes/pr4463.yaml

* Drop error traceback on `get_flow_view` failure

* Move "Done" printout to after pulling for clarity; skip query when quiet

* Make `from_flow_obj` compatible with `prefect register`

* Set poll_interval incase there are no message son first poll

* Optimize imports

* Add deprecated comment

* Add CLI examples for `prefect run`

* Collapse `flow-group-id` and `flow-id` options

* Rehaul context/parameter parsing with error handling and new format

* Add Server line reference

* Cleanup `--watch` docs and change `--no-backend` to `--local`

* Display a message if the flow run is already done

* Improve `--local` execution

- Add `try_error_done` utility
- Renames `--path` to `--file/-f`
- Allows local runs to load flows from backend storage
- Renames `--no-agent` to `--execute/-e`
- Adds TerminalError handling/usage

* Sort imports

* Add labels to agent warning; improve cancel ctrl-c message

* Update examples

* Fix `flow` -> `flow_view`

* Add `RunConfig` to `FlowRunView`

* Set log level and environment variables during agentless execution

* WIP

* Add test coverage for FlowView/FlowRunView additions

* Remove agentless and local execution flags
- path/module flows always run local
- cloud flows cannot be run local

* Move back from `--file` to `--path`

* Update examples to match drop of local/execute

* Fix submodule imports in `get_object`

* Fix type-checking

* Sort imports

* Lint after execute/local flag removal

* Remove whitespace on blank line

* Update docs reference

* Cleanup and move old `prefect run flow` tests

* Do not display the traceback for click errors

* Create copies of the `build_register` loader funcs

These do not set flow storage like the `build_register` ones do and raise errors that can be consistently handled by `try_error_done` instead of directly echoing tracebacks

* Raise `TerminalError` consistently

* Display the traceback on retrieval for unhandled exceptions

* Fix flow configuration newlines

* Cleanup imports

* Write tests for `prefect run` local flow runs

* Add check for the correct exception before silencing

* Add docstrings to new `FlowView` and `FlowRunView` objects

* Add missing `:` and `-`

* Privatize `TimestampedState`

Required to pass State subclass tests and makes sense anyway

* Remove unused import

* Skip test on windows

Likely an issue with the path check in the exception string seems unworthy of complicating since the full traceback is just shown

* Add log formatting to local runs

* Add handler for empty null formatter

* Remove unused import

* Raise TerminalError instead of ClickException

* Error when `--id` and `--name` are passed

* Wrap parameter file loading exceptions

* Add param/context tests

* Cleanup terminal error printing

* Wrap the location name for readability

* Wrap the flow group id in quotes when missing

* Use the already deserialized run confiig

* Add equality dunder to base `RunConfig`

* Add tests for cloud runs

* Cleanup unused items

* Switch to Py 3.6/7 syntax for call_args assertion

* Add debug print for appveyor assertion failure

* Try a repr instead of a str

* Cleanup

* Fixup no labels display

* Clarify log level flag default

* Shorten examaple line to fit more shells [ci skip]

* Add comment to test [ci skip]

* Use monkeypatch to set the sys path

* Make `quiet_echo` robust to mutation

* Explain flow runner exceptions in echo

* Update `watch_flow_run` output to take `FlowRunLog`

* Improve `watch_flow_run` finished flow run output

* Expose map index in `FlowRunView.get_task_run`

* Handle null states from backend

* Refuse to get task run results if not finished

* Improve edge-case handling during mapped task run result retrieval

* Add `from_id` generic id handling to `FlowView`

* WIP: CreateFlowRun task

* FIxup broken indentation for child task run result

* Switch to functional task for flow run creation

* Fixup `from_id`

* Add scheduled start time, run config, docstring; cleanup

* Add changes/pr4563.yaml

* Update example to include basic run

* Add docstrings

* Fix agent warning output

* Add map index docstring

* Update test

* Drop task object slug inference from `get_task_run`

* Cleanup slug/index based caching

* Cleanup missed merge conflicts

* Run Flow API: Update `watch_flow_run` to be an iterator (#4585)

* Update `watch_flow_run` to yield logs instead of taking a callable

* Cleanup

* Update docstring

* Do not load static tasks by default; not a common use-case

* Add timeout to `watch_flow_run` and tests

* Cleanup unused imports

* Add `stream_states` toggle to `watch_flow_run`

* Improve `time_ago` calculation

* Labels is `None` by default; not an empty list like in the CLI

* Handle edge-case run name display

* Remove `get_flow_run_state`

Not sure how this would play with returning State types from tasks yet

* Add new tasks to tasks.prefect init

* Add tests for `create_flow_run` and `wait_for_flow_run`

* Add tests for `get_task_run_result`

* Remove unused import

* Add `create_flow_run` to flow run creation docs

* Add `get_task_run_result` docs

* Add new tasks to outline.toml [ci skip]

* Drop debug print [ci skip]

* Add types to return docstring

* Move tasks to "classes" in autodoc?

* Add name attr access to `FunctionTask` for docs

* Fixup docstring

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>

* Remove moved concepts from sidebar

* Add blurb about creating flow run to docs [ci skip]

* Fix language about time of creation [ci skip]

* Add documentation for `TaskRunView.iter_mapped`

* Remove aggregate queries

* Fix link to UI task run page

* Attempt to fix task concurrency from library tabs [ci skip]

* Update usage of "Prefect library" to clarify "Python"

* Run Flow API: Cleanup view object interfaces (#4615)

* Remove `result` from `TaskRunView` properties

* Move `FlowRunView.task_run_ids` to `get_task_run_ids()` method

* Update `FlowRunView.flow` to `FlowRunView.get_flow_metadata()`

* Add experimental flags

* Add labels to `FlowRunView` repr

* Fix metadata call

* Add missing "-" to `watch_flow_run` docstring args

* Clarify that these functions are tasks

* Add check for `get_task_run_result` referencing current flow run

* Add check for custom result types which we cannot support yet

* Add test for error on missing result data

* Update test to use repr match for win32 compat

* Skip match for win32 -- not worth dealing with file name handling

* Remove the match entirely...

* Fix static analysis

* Prevent test from looping

* Bump agent initial wait to 15s

* Use `query_for_flows` instead of `query_for_flow` when looking up by flow group id

This fixes a bug where if there were multiple flows in the group it would error
instead of retrieving the most recent flow

* Fixup timing in test

* Cleanup changes/ entries [ci skip]

Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com>
Co-authored-by: Zach Angell <zachary.james.angell@gmail.com>
6e91701

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Hello, world! 👋

We've rebuilt data engineering for the data science era.

Prefect is a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows, and Prefect takes care of the rest.

Read the docs; get the code; ask us anything!

Welcome to Workflows

Prefect's Pythonic API should feel familiar for newcomers. Mark functions as tasks and call them on each other to build up a flow.

from prefect import task, Flow, Parameter


@task(log_stdout=True)
def say_hello(name):
    print("Hello, {}!".format(name))


with Flow("My First Flow") as flow:
    name = Parameter('name')
    say_hello(name)


flow.run(name='world') # "Hello, world!"
flow.run(name='Marvin') # "Hello, Marvin!"

For more detail, please see the Core docs

UI and Server

In addition to the Prefect Cloud platform, Prefect includes an open-source backend for orchestrating and managing flows, consisting primarily of Prefect Server and Prefect UI. This local server stores flow metadata in a Postgres database and exposes a GraphQL API.

Before running the server for the first time, run prefect backend server to configure Prefect for local orchestration. Please note the server requires Docker and Docker Compose to be running.

To start the server, UI, and all required infrastructure, run:

prefect server start

Once all components are running, you can view the UI by visiting http://localhost:8080.

Please note that executing flows from the server requires at least one Prefect Agent to be running: prefect agent local start.

Finally, to register any flow with the server, call flow.register(). For more detail, please see the orchestration docs.

"...Prefect?"

From the Latin praefectus, meaning "one who is in charge", a prefect is an official who oversees a domain and makes sure that the rules are followed. Similarly, Prefect is responsible for making sure that workflows execute properly.

It also happens to be the name of a roving researcher for that wholly remarkable book, The Hitchhiker's Guide to the Galaxy.

Integrations

Thanks to Prefect's growing task library and deep ecosystem integrations, building data applications is easier than ever.

Something missing? Open a feature request or contribute a PR! Prefect was designed to make adding new functionality extremely easy, whether you build on top of the open-source package or maintain an internal task library for your team.

Task Library

Airtable

Asana

AWS

Azure

Azure ML

Databricks

DBT

Docker

Dremio

Dropbox

Email

Fivetran

GitHub

Google Cloud

Google Sheets

Great Expectations

Jira

Jupyter

Kubernetes

Monday

MySQL

PostgreSQL

Python

Pushbullet

Redis

RSS

Shell

Slack

Snowflake

SpaCy

SQLite

SQL Server

Trello

Twitter

Deployment & Execution

Azure

AWS

Dask

Docker

Google Cloud

Kubernetes

Universal Deploy

Resources

Prefect provides a variety of resources to help guide you to a successful outcome.

We are committed to ensuring a positive environment, and all interactions are governed by our Code of Conduct.

Documentation

Prefect's documentation -- including concepts, tutorials, and a full API reference -- is always available at docs.prefect.io.

Instructions for contributing to documentation can be found in the development guide.

Slack Community

Join our Slack to chat about Prefect, ask questions, and share tips.

Blog

Visit the Prefect Blog for updates and insights from the Prefect team.

Support

Prefect offers a variety of community and premium support options for users of both Prefect Core and Prefect Cloud.

Contributing

Read about Prefect's community or dive in to the development guides for information about contributions, documentation, code style, and testing.

Installation

Requirements

Prefect requires Python 3.6+. If you're new to Python, we recommend installing the Anaconda distribution.

Latest Release

To install Prefect, run:

pip install prefect

or, if you prefer to use conda:

conda install -c conda-forge prefect

or pipenv:

pipenv install --pre prefect

Bleeding Edge

For development or just to try out the latest features, you may want to install Prefect directly from source.

Please note that the master branch of Prefect is not guaranteed to be compatible with Prefect Cloud or the local server.

git clone https://github.com/PrefectHQ/prefect.git
pip install ./prefect

License

Prefect Core is licensed under the Apache Software License Version 2.0. Please note that Prefect Core includes utilities for running Prefect Server and the Prefect UI, which are themselves licensed under the Prefect Community License.