add `prefect.yaml` and cli support for new schedule fields #13318

zzstoatzz · 2024-05-13T17:43:51Z

this PR:

adds a setting PREFECT_EXPERIMENTAL_ENABLE_SCHEDULE_CONCURRENCY to gate CLI prompts for new fields while the worker does not actually enforce these limits
adds prompts to prefect deploy to choose a value for max_active_runs and catchup for each schedule
adds support for configuring and saving these fields for schedule definitions in prefect.yaml

netlify · 2024-05-13T17:44:08Z

✅ Deploy Preview for prefect-docs-preview ready!

Name	Link
🔨 Latest commit	`13643f0`
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-docs-preview/deploys/66437c36ef5b9600086361ee
😎 Deploy Preview	https://deploy-preview-13318--prefect-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

…thub.com/prefecthq/prefect into new-schedule-fields-yaml-and-cli-support

bunchesofdonald

I had a few minor nits, but looks great overall!

bunchesofdonald · 2024-05-14T13:58:23Z

tests/cli/test_deploy.py

+            command=(
+                f"deploy ./flows/hello.py:my_flow -n test-name --pool {work_pool.name}"
+            ),
+            user_input=(


Don't block this PR on this, but a couple weeks ago I added a more semantic way of dealing with this in tests called prompts_and_responses, you can see an example over here: https://github.com/PrefectHQ/prefect/blob/main/tests/cli/test_deploy.py#L4130C13-L4140

bunchesofdonald · 2024-05-14T14:09:17Z

src/prefect/cli/_prompts.py

+        default=None,
+    )
+    catchup = confirm(
+        "Catch up on missed runs?",


Maybe include the fact that we're talking about 'late' runs here?

Suggested change

"Catch up on missed runs?",

"Catch up on late flow runs?",

bunchesofdonald · 2024-05-14T14:10:04Z

src/prefect/cli/_prompts.py

@@ -238,6 +242,21 @@ def process_response(self, value: str) -> str:
            raise InvalidResponse(self.validate_error_message)


+def prompt_for_max_active_runs_and_catchup(console):


I'd split this into two prompts, one for max active and one for catchup.

bunchesofdonald · 2024-05-14T14:12:04Z

src/prefect/cli/_prompts.py

@@ -376,7 +405,7 @@ def prompt_schedules(console) -> List[MinimalDeploymentSchedule]:
 async def prompt_select_work_pool(
    console: Console,
    prompt: str = "Which work pool would you like to deploy this flow to?",
-    client: PrefectClient = None,
+    client: Optional[PrefectClient] = None,


Another thing to not block this PR on, but I added a client_injector that makes the typing on this kind of stuff better:

prefect/src/prefect/client/utilities.py

Lines 66 to 74 in 8454a1c

def client_injector(

func: Callable[Concatenate["PrefectClient", P], Awaitable[R]],

) -> Callable[P, Awaitable[R]]:

@wraps(func)

async def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:

client, _ = get_or_create_client()

return await func(client, *args, **kwargs)

return wrapper

It's used the same way as inject_client except that it always expects the client will be the first argument and non-optional:

prefect/src/prefect/input/actions.py

Lines 54 to 57 in 8454a1c

@sync_compatible

@client_injector

async def create_flow_run_input(

client: "PrefectClient",

bunchesofdonald · 2024-05-14T14:12:24Z

src/prefect/client/orchestration.py

-        version: str = None,
-        schedule: SCHEDULE_TYPES = None,
-        schedules: List[DeploymentScheduleCreate] = None,
+        version: Optional[str] = None,


abrookins

Agreed I would split up the prompts, but I have no blocking comments. Looks good!

…thub.com/prefecthq/prefect into new-schedule-fields-yaml-and-cli-support

…#13318)

add prefect.yaml and cli support for new schedule fields

c46c1f7

zzstoatzz marked this pull request as ready for review May 13, 2024 17:43

zzstoatzz requested review from zangell44 and a team as code owners May 13, 2024 17:43

zzstoatzz marked this pull request as draft May 13, 2024 17:44

zzstoatzz added 10 commits May 13, 2024 12:44

Merge branch 'main' into new-schedule-fields-yaml-and-cli-support

76b4235

delete straggler

c50cb4d

Merge branch 'new-schedule-fields-yaml-and-cli-support' of https://gi…

531c71c

…thub.com/prefecthq/prefect into new-schedule-fields-yaml-and-cli-support

fix tests

e283e42

fix tests

1dd26d1

add comment to retry ci

6c42eb8

Merge branch 'main' into new-schedule-fields-yaml-and-cli-support

0b81f8f

Merge branch 'main' into new-schedule-fields-yaml-and-cli-support

dc25518

Merge branch 'main' into new-schedule-fields-yaml-and-cli-support

a1570f7

Merge branch 'main' into new-schedule-fields-yaml-and-cli-support

bbf3b6b

zzstoatzz marked this pull request as ready for review May 13, 2024 22:41

Merge branch 'main' into new-schedule-fields-yaml-and-cli-support

f4aea03

bunchesofdonald approved these changes May 14, 2024

View reviewed changes

abrookins approved these changes May 14, 2024

View reviewed changes

zzstoatzz added 2 commits May 14, 2024 09:58

address review comments

d7712d7

Merge branch 'new-schedule-fields-yaml-and-cli-support' of https://gi…

13643f0

…thub.com/prefecthq/prefect into new-schedule-fields-yaml-and-cli-support

zzstoatzz merged commit 20792ec into main May 14, 2024
54 of 59 checks passed

zzstoatzz deleted the new-schedule-fields-yaml-and-cli-support branch May 14, 2024 15:14

serinamarie pushed a commit to bsignoret/prefect that referenced this pull request May 16, 2024

add prefect.yaml and cli support for new schedule fields (PrefectHQ…

9df3fd9

…#13318)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add `prefect.yaml` and cli support for new schedule fields #13318

add `prefect.yaml` and cli support for new schedule fields #13318

zzstoatzz commented May 13, 2024 •

edited

Loading

netlify bot commented May 13, 2024 •

edited

Loading

bunchesofdonald left a comment

bunchesofdonald May 14, 2024

bunchesofdonald May 14, 2024

bunchesofdonald May 14, 2024

bunchesofdonald May 14, 2024

bunchesofdonald May 14, 2024

abrookins left a comment

		@@ -238,6 +242,21 @@ def process_response(self, value: str) -> str:
		raise InvalidResponse(self.validate_error_message)


		def prompt_for_max_active_runs_and_catchup(console):

	def client_injector(
	func: Callable[Concatenate["PrefectClient", P], Awaitable[R]],
	) -> Callable[P, Awaitable[R]]:
	@wraps(func)
	async def wrapper(args: P.args, *kwargs: P.kwargs) -> R:
	client, _ = get_or_create_client()
	return await func(client, args, *kwargs)

	return wrapper

	@sync_compatible
	@client_injector
	async def create_flow_run_input(
	client: "PrefectClient",

add prefect.yaml and cli support for new schedule fields #13318

add prefect.yaml and cli support for new schedule fields #13318

Conversation

zzstoatzz commented May 13, 2024 • edited Loading

netlify bot commented May 13, 2024 • edited Loading

✅ Deploy Preview for prefect-docs-preview ready!

bunchesofdonald left a comment

Choose a reason for hiding this comment

bunchesofdonald May 14, 2024

Choose a reason for hiding this comment

bunchesofdonald May 14, 2024

Choose a reason for hiding this comment

bunchesofdonald May 14, 2024

Choose a reason for hiding this comment

bunchesofdonald May 14, 2024

Choose a reason for hiding this comment

bunchesofdonald May 14, 2024

Choose a reason for hiding this comment

abrookins left a comment

Choose a reason for hiding this comment

add `prefect.yaml` and cli support for new schedule fields #13318

add `prefect.yaml` and cli support for new schedule fields #13318

zzstoatzz commented May 13, 2024 •

edited

Loading

netlify bot commented May 13, 2024 •

edited

Loading