Introducing CLI Commands for Shell Flows #11998

gabcoyne · 2024-02-15T19:49:15Z

Overview

This update introduces two new CLI commands, prefect command watch and prefect command serve, allowing users to monitor and serve shell commands as Prefect flows effortlessly. Additionally, we're adding a ModuleStorage class to enable module-based flow storage and enable command serving. This feature simplifies integrating shell commands with Prefect, enhancing automation and monitoring capabilities.

Example

Imagine scheduling a daily data backup with a simple shell command. With prefect command watch, you can now easily incorporate this into your Prefect workflows, monitoring its execution as part of your data pipeline.

Checklist

This pull request includes tests that ensure functionality works as expected.
This pull request is labeled with feature to highlight the new additions.

For documentation changes:

Adjustments to the documentation are included to guide users on how to utilize these new commands.

For new functions or classes in the Python SDK:

Helpful docstrings have been added to the new classes and functions for better understanding.
A stub page in the Python SDK docs and an entry in mkdocs.yml navigation have been prepared to introduce ModuleStorage.

Happy engineering!

netlify · 2024-02-15T19:49:35Z

✅ Deploy Preview for prefect-docs-preview ready!

Name	Link
🔨 Latest commit	`a1c25d4`
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-docs-preview/deploys/66034ed98e43a30008f4bff6
😎 Deploy Preview	https://deploy-preview-11998--prefect-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

src/prefect/cli/command.py

src/prefect/runner/storage.py

github-actions · 2024-03-05T15:01:12Z

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas Frederik Hoeck <44194839+thomasfrederikhoeck@users.noreply.github.com> Co-authored-by: Thomas Frederik Hoeck <tfh@norden.com> Co-authored-by: nate nowack <thrast36@gmail.com> Co-authored-by: nate nowack <nate@prefect.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: marvin-robot <marvin-robot@users.noreply.github.com> Co-authored-by: Chris Guidry <chris.g@prefect.io> Co-authored-by: Chris Pickett <chris.pickett@prefect.io> Co-authored-by: Andrew Brookins <andrew.b@prefect.io> Co-authored-by: urimandujano <uriel@prefect.io> Co-authored-by: Collin <collin@prefect.io> Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com> Co-authored-by: Alexander Streed <desertaxle@users.noreply.github.com> Co-authored-by: Edward Park <ed.sh.park@gmail.com> Co-authored-by: Serina Grill <42048900+serinamarie@users.noreply.github.com> Co-authored-by: Jeff Hale <discdiver@users.noreply.github.com> Co-authored-by: jakekaplan <40362401+jakekaplan@users.noreply.github.com> Co-authored-by: Zanie Blue <zanie@prefect.io> Co-authored-by: Kevin Grismore <146098880+kevingrismore@users.noreply.github.com> Co-authored-by: Hamza Mogni <hamzamogni5@gmail.com> Co-authored-by: Bill Palombi <bill@prefect.io> Co-authored-by: Brandon Reid <brandonreidak@gmail.com> Co-authored-by: Elad Moshe <eladm26@gmail.com> Co-authored-by: Jessica Smith <8505845+NodeJSmith@users.noreply.github.com> Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com> Co-authored-by: Alexander Streed <alex.s@prefect.io> Co-authored-by: Sean <64788907+seano-vs@users.noreply.github.com>

…ure/script-watch

zzstoatzz

nice!

what do you think about adding a small bit of docs on this?

src/prefect/cli/shell.py

tests/cli/test_shell.py

src/prefect/cli/shell.py

Co-authored-by: Nicholas Brown <znicholasbrown@gmail.com>

Co-authored-by: Serina Grill <42048900+serinamarie@users.noreply.github.com>

…refect into feature/script-watch

…ure/script-watch

desertaxle

It'd be great to see one test ensuring this CLI integrates with the runner correctly. It should be sufficient to mock out Runner and check that Runner.start is called with the expected arguments.

src/prefect/cli/shell.py

…ure/script-watch

desertaxle

LGTM!

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas Frederik Hoeck <44194839+thomasfrederikhoeck@users.noreply.github.com> Co-authored-by: Thomas Frederik Hoeck <tfh@norden.com> Co-authored-by: nate nowack <thrast36@gmail.com> Co-authored-by: nate nowack <nate@prefect.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: marvin-robot <marvin-robot@users.noreply.github.com> Co-authored-by: Chris Guidry <chris.g@prefect.io> Co-authored-by: Chris Pickett <chris.pickett@prefect.io> Co-authored-by: Andrew Brookins <andrew.b@prefect.io> Co-authored-by: urimandujano <uriel@prefect.io> Co-authored-by: Collin <collin@prefect.io> Co-authored-by: Zach Angell <42625717+zangell44@users.noreply.github.com> Co-authored-by: Alexander Streed <desertaxle@users.noreply.github.com> Co-authored-by: Edward Park <ed.sh.park@gmail.com> Co-authored-by: Serina Grill <42048900+serinamarie@users.noreply.github.com> Co-authored-by: Jeff Hale <discdiver@users.noreply.github.com> Co-authored-by: jakekaplan <40362401+jakekaplan@users.noreply.github.com> Co-authored-by: Zanie Blue <zanie@prefect.io> Co-authored-by: Kevin Grismore <146098880+kevingrismore@users.noreply.github.com> Co-authored-by: Hamza Mogni <hamzamogni5@gmail.com> Co-authored-by: Bill Palombi <bill@prefect.io> Co-authored-by: Brandon Reid <brandonreidak@gmail.com> Co-authored-by: Elad Moshe <eladm26@gmail.com> Co-authored-by: Jessica Smith <8505845+NodeJSmith@users.noreply.github.com> Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com> Co-authored-by: Alexander Streed <alex.s@prefect.io> Co-authored-by: Sean <64788907+seano-vs@users.noreply.github.com> Co-authored-by: Nicholas Brown <znicholasbrown@gmail.com>

gabcoyne added 5 commits January 30, 2024 11:09

src/prefect/cli/command.py

06d06c0

WIP comment serve

988e319

just use serve (#11788)

05d07da

add storage class

959c0d4

WIP for command cli

e8067a9

zzstoatzz reviewed Feb 16, 2024

View reviewed changes

src/prefect/cli/command.py Outdated Show resolved Hide resolved

desertaxle reviewed Feb 20, 2024

View reviewed changes

src/prefect/runner/storage.py Outdated Show resolved Hide resolved

desertaxle mentioned this pull request Mar 1, 2024

Add ability to use a module path entrypoint when using .serve or .deploy #12134

Merged

6 tasks

github-actions bot added the status:stale label Mar 5, 2024

github-actions bot removed the status:stale label Mar 7, 2024

gabcoyne changed the title ~~Introducing CLI Commands for Shell Command Flows and ModuleStorage~~ Introducing CLI Commands for Shell Flows Mar 13, 2024

gabcoyne added 9 commits March 13, 2024 12:46

merge from main

6e1ba78

fix merge

22648d7

preliminary tests

f3dbca0

shell serve tests

e1ccf94

Merge branch 'main' of https://github.com/prefecthq/prefect into feat…

c24310b

…ure/script-watch

add shell tests

1cb2ca9

add List from typing

f20996a

update docstrings and descriptions

dcda499

Merge branch 'main' of https://github.com/prefecthq/prefect into feat…

34c8366

…ure/script-watch

gabcoyne marked this pull request as ready for review March 19, 2024 16:04

gabcoyne requested a review from a team as a code owner March 19, 2024 16:04

zzstoatzz approved these changes Mar 19, 2024

View reviewed changes

src/prefect/cli/shell.py Outdated Show resolved Hide resolved

desertaxle requested changes Mar 19, 2024

View reviewed changes

update from comments

f54555a

znicholasbrown reviewed Mar 19, 2024

View reviewed changes

src/prefect/cli/shell.py Outdated Show resolved Hide resolved

serinamarie reviewed Mar 20, 2024

View reviewed changes

src/prefect/cli/shell.py Show resolved Hide resolved

src/prefect/cli/shell.py Outdated Show resolved Hide resolved

src/prefect/cli/shell.py Show resolved Hide resolved

src/prefect/cli/shell.py Outdated Show resolved Hide resolved

serinamarie added the enhancement An improvement of an existing feature label Mar 20, 2024

gabcoyne and others added 16 commits March 20, 2024 09:31

fix serve issue

6c9bb8a

Update src/prefect/cli/shell.py

46c4d40

Co-authored-by: Nicholas Brown <znicholasbrown@gmail.com>

Update src/prefect/cli/shell.py

787f9cb

Co-authored-by: Serina Grill <42048900+serinamarie@users.noreply.github.com>

Update src/prefect/cli/shell.py

1db789a

Co-authored-by: Serina Grill <42048900+serinamarie@users.noreply.github.com>

Merge branch 'feature/script-watch' of https://github.com/prefecthq/p…

62ca63a

…refect into feature/script-watch

update tests fix serve

f23c925

re-order and add tag description

a8b3729

update traceback for exit_with_error

c1338dd

update flow-name

2d04e7b

remove exit with error

2db3015

call typer.exit directly

0099642

add fail state

f414c3f

fix tests

68a885c

Merge branch 'main' of https://github.com/prefecthq/prefect into feat…

146a8f0

…ure/script-watch

add doc

01db31f

Merge branch 'main' of https://github.com/prefecthq/prefect into feat…

7effb70

…ure/script-watch

desertaxle reviewed Mar 21, 2024

View reviewed changes

src/prefect/cli/shell.py Outdated Show resolved Hide resolved

add doc, streaming

7c320b5

gabcoyne requested a review from a team as a code owner March 21, 2024 14:43

gabcoyne added 3 commits March 26, 2024 15:59

Merge branch 'main' of https://github.com/prefecthq/prefect into feat…

5cf6719

…ure/script-watch

add test for runner.start

d4bd350

Merge branch 'main' into feature/script-watch

a1c25d4

gabcoyne enabled auto-merge (squash) March 26, 2024 22:58

desertaxle approved these changes Mar 26, 2024

View reviewed changes

gabcoyne merged commit 8c87076 into main Mar 26, 2024
46 checks passed

gabcoyne deleted the feature/script-watch branch March 26, 2024 23:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing CLI Commands for Shell Flows #11998

Introducing CLI Commands for Shell Flows #11998

gabcoyne commented Feb 15, 2024 •

edited by zzstoatzz

Loading

netlify bot commented Feb 15, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024

zzstoatzz left a comment

desertaxle left a comment

desertaxle left a comment

Introducing CLI Commands for Shell Flows #11998

Introducing CLI Commands for Shell Flows #11998

Conversation

gabcoyne commented Feb 15, 2024 • edited by zzstoatzz Loading

Overview

Example

Checklist

netlify bot commented Feb 15, 2024 • edited Loading

✅ Deploy Preview for prefect-docs-preview ready!

github-actions bot commented Mar 5, 2024

zzstoatzz left a comment

Choose a reason for hiding this comment

desertaxle left a comment

Choose a reason for hiding this comment

desertaxle left a comment

Choose a reason for hiding this comment

gabcoyne commented Feb 15, 2024 •

edited by zzstoatzz

Loading

netlify bot commented Feb 15, 2024 •

edited

Loading