Use native VertexAI scheduler capability instead of old GCP official workaround #2310

francoisserra · 2024-01-17T15:50:57Z

Describe changes

I implemented the switch to native VertexAI pipelines scheduling capability instead of the old way that was relying on creating a cloud function along with a cloud scheduler.
It somewhat simplifies the ZenML VertexAI orchestrator code and also the comprehension (activation/deletion/...) of the scheduled pipelines since now it fully embraces the native VertexAI philosophy.

Note that I deprecated the no longer used settings fields related to "cloud functions" and "cloud scheduler" service accounts (and updated the doc).

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

coderabbitai · 2024-01-17T15:51:20Z

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit-tests for this file.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit tests for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository from git and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit tests.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

htahir1

Wow love this change and it seems very thorough. Thank you @francoisserra for getting rid of that ugly scheduling code! 🚀 I just have a couple of comments. In general I also think we should remove "google-cloud-scheduler>=2.7.3",
"google-cloud-functions>=1.8.3", as dependencies in zenml/integations/gcp/__init__/py

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

strickvl

Looks good to me at this point. We'll test this on our end as well.

docs/book/stacks-and-components/component-guide/orchestrators/vertex.md

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

safoinme

Looks good, Great PR!

…vertex.md Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

htahir1

Nice one! the only thing left IMO is just talking about timezones! :-) Rest looks great!

docs/book/stacks-and-components/component-guide/orchestrators/vertex.md

htahir1

Awesome. @francoisserra thank you so much for this! People are gonna love it! 🚀

htahir1 · 2024-01-29T20:52:43Z

@coderabbitai review

socket-security · 2024-02-05T12:50:47Z

Removed dependencies detected. Learn more about Socket for GitHub ↗︎

🚮 Removed packages: pypi/google-api-python-client@1.12.11, pypi/google-api-python-client@1.12.11

View full report↗︎

strickvl

Two tiny nits, but otherwise looks good to go. I'll merge these in from my end.

docs/book/stacks-and-components/component-guide/orchestrators/vertex.md

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

strickvl

Thank you for this!

…workaround (zenml-io#2310) * Use native VertexAI pipelines scheduling facility * deprecate VertexOrchestrator no longer used fields * updating docs * remove dead code * style: 🎨 linting * Update docs/book/stacks-and-components/component-guide/orchestrators/vertex.md Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com> * Update docs/book/stacks-and-components/component-guide/orchestrators/vertex.md Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com> * Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com> * Remove no longer used libraries * feat: add support for start_time and end_time in the schedule * Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com> * docs: Add details about schedule parameters * Apply suggestions from code review --------- Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com> Co-authored-by: Hamza Tahir <hamza@zenml.io> Co-authored-by: Barış Can Durak <36421093+bcdurak@users.noreply.github.com>

francoisserra added 3 commits January 17, 2024 15:30

Use native VertexAI pipelines scheduling facility

52c5aa8

deprecate VertexOrchestrator no longer used fields

b8eda2e

updating docs

413a9e7

strickvl added the enhancement New feature or request label Jan 17, 2024

remove dead code

9726a3e

francoisserra force-pushed the feat/gcp_vertexai_scheduler branch from f42ed0f to 9726a3e Compare January 17, 2024 16:11

francoisserra marked this pull request as ready for review January 17, 2024 16:18

francoisserra added 2 commits January 18, 2024 10:31

Merge branch 'develop' into feat/gcp_vertexai_scheduler

00a2829

style: 🎨 linting

dcd86f4

htahir1 requested review from bcdurak and avishniakov and removed request for bcdurak January 19, 2024 20:07

htahir1 assigned strickvl Jan 19, 2024

htahir1 requested changes Jan 19, 2024

View reviewed changes

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Show resolved Hide resolved

avishniakov requested review from strickvl and safoinme and removed request for avishniakov January 22, 2024 07:51

strickvl reviewed Jan 22, 2024

View reviewed changes

Merge branch 'develop' into feat/gcp_vertexai_scheduler

ee09e38

safoinme requested changes Jan 23, 2024

View reviewed changes

safoinme self-requested a review January 23, 2024 15:10

francoisserra and others added 8 commits January 24, 2024 10:12

Update docs/book/stacks-and-components/component-guide/orchestrators/…

2782f30

…vertex.md Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

Update docs/book/stacks-and-components/component-guide/orchestrators/…

a8fc8e9

…vertex.md Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

93e9fa4

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

Merge branch 'develop' into feat/gcp_vertexai_scheduler

afd4655

Remove no longer used libraries

c934761

feat: add support for start_time and end_time in the schedule

7b432ce

Merge branch 'develop' into feat/gcp_vertexai_scheduler

cf167af

Merge branch 'develop' into feat/gcp_vertexai_scheduler

eab683b

francoisserra requested review from htahir1 and strickvl January 25, 2024 07:45

Merge branch 'develop' into feat/gcp_vertexai_scheduler

b74a5c1

strickvl requested changes Jan 25, 2024

View reviewed changes

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Outdated Show resolved Hide resolved

francoisserra and others added 2 commits January 25, 2024 09:34

Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py

360d7a5

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

Merge branch 'develop' into feat/gcp_vertexai_scheduler

2ec8d9b

htahir1 requested changes Jan 25, 2024

View reviewed changes

docs/book/stacks-and-components/component-guide/orchestrators/vertex.md Show resolved Hide resolved

francoisserra added 2 commits January 26, 2024 10:30

docs: Add details about schedule parameters

11ef15d

Merge branch 'develop' into feat/gcp_vertexai_scheduler

7cbf8e7

htahir1 approved these changes Jan 26, 2024

View reviewed changes

htahir1 and others added 2 commits January 29, 2024 21:53

Merge branch 'develop' into feat/gcp_vertexai_scheduler

a57102c

Merge branch 'develop' into feat/gcp_vertexai_scheduler

c4b87fb

strickvl marked this pull request as draft January 31, 2024 15:17

strickvl marked this pull request as ready for review January 31, 2024 15:17

strickvl added the run-slow-ci label Feb 5, 2024

Merge branch 'develop' into feat/gcp_vertexai_scheduler

bd19f4d

strickvl requested changes Feb 5, 2024

View reviewed changes

docs/book/stacks-and-components/component-guide/orchestrators/vertex.md Outdated Show resolved Hide resolved

src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Outdated Show resolved Hide resolved

Apply suggestions from code review

cc0bb7f

safoinme approved these changes Feb 5, 2024

View reviewed changes

strickvl approved these changes Feb 5, 2024

View reviewed changes

Merge branch 'develop' into feat/gcp_vertexai_scheduler

43731fc

bcdurak merged commit 44110b5 into zenml-io:develop Feb 6, 2024
53 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use native VertexAI scheduler capability instead of old GCP official workaround #2310

Use native VertexAI scheduler capability instead of old GCP official workaround #2310

francoisserra commented Jan 17, 2024

coderabbitai bot commented Jan 17, 2024 •

edited

Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

CodeRabbit Discord Community

htahir1 left a comment

strickvl left a comment

safoinme left a comment

htahir1 left a comment

htahir1 left a comment

htahir1 commented Jan 29, 2024

socket-security bot commented Feb 5, 2024 •

edited

Loading

strickvl left a comment

strickvl left a comment

Use native VertexAI scheduler capability instead of old GCP official workaround #2310

Use native VertexAI scheduler capability instead of old GCP official workaround #2310

Conversation

francoisserra commented Jan 17, 2024

Describe changes

Pre-requisites

Types of changes

coderabbitai bot commented Jan 17, 2024 • edited Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

CodeRabbit Discord Community

htahir1 left a comment

Choose a reason for hiding this comment

strickvl left a comment

Choose a reason for hiding this comment

safoinme left a comment

Choose a reason for hiding this comment

htahir1 left a comment

Choose a reason for hiding this comment

htahir1 left a comment

Choose a reason for hiding this comment

htahir1 commented Jan 29, 2024

socket-security bot commented Feb 5, 2024 • edited Loading

strickvl left a comment

Choose a reason for hiding this comment

strickvl left a comment

Choose a reason for hiding this comment

coderabbitai bot commented Jan 17, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

socket-security bot commented Feb 5, 2024 •

edited

Loading