Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add first version of gke-starter values file #5026

Merged

Conversation

cjidboon94
Copy link
Contributor

Tracking issue

#5016

Closes #5016

Why are the changes needed?

There are examples of how to deploy flyte minimally on AWS/EKS described in Single Cluster Simple Cloud Deployment but not for GCP/GKE.
This values file makes it easier for new people to deploy.

What changes were proposed in this pull request?

A minimal value file with default settings filled in for a minimal deployment on GKE/GCP using the flyte-binary chart

How was this patch tested?

I used it to deploy on GKE.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly. (not sure if applicable)
  • All commits are signed-off.

Related PRs

Docs link

Copy link

welcome bot commented Mar 8, 2024

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. documentation Improvements or additions to documentation enhancement New feature or request labels Mar 8, 2024
Copy link

codecov bot commented Mar 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.99%. Comparing base (5fc57fa) to head (f57b636).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5026      +/-   ##
==========================================
- Coverage   58.99%   58.99%   -0.01%     
==========================================
  Files         645      645              
  Lines       55648    55648              
==========================================
- Hits        32831    32830       -1     
- Misses      20222    20223       +1     
  Partials     2595     2595              
Flag Coverage Δ
unittests 58.99% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@davidmirror-ops davidmirror-ops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. Thank you!

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 13, 2024
@davidmirror-ops
Copy link
Contributor

For the failing Helm test, can you try running make helm from the base folder in your fork and then push the changes?

@cjidboon94
Copy link
Contributor Author

For the failing Helm test, can you try running make helm from the base folder in your fork and then push the changes?

When I try to run make helm, I also get this (same as output of failing helm test):

bash script/generate_helm.sh
+ echo 'Generating Helm'
Generating Helm
+ HELM_SKIP_INSTALL=false
+ '[' false '!=' true ']'
+ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
+ bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11679  100 11679    0     0   140k      0 --:--:-- --:--:-- --:--:--  140k
Helm v3.14.3 is already latest
+ helm version
version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.21.7"}
+ DEPLOYMENT_CORE='eks gcp'
+++ dirname script/generate_helm.sh
++ cd script
++ pwd
+ DIR=<root>/projects/flyte/script
+ HELM_CAPABILITIES='-a rbac.authorization.k8s.io/v1 -a networking.k8s.io/v1/Ingress -a apiextensions.k8s.io/v1/CustomResourceDefinition'
+ helm dep update <root>/projects/flyte/script/../charts/flyte-deps/
Getting updates for unmanaged Helm repositories...
...Unable to get an update from the "https://googlecloudplatform.github.io/spark-on-k8s-operator" chart repository:
        failed to fetch https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml : 404 Not Found
...Successfully got an update from the "https://kubernetes.github.io/dashboard/" chart repository
...Successfully got an update from the "https://helm.dask.org" chart repository
...Successfully got an update from the "https://charts.bitnami.com/bitnami" chart repository
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "flyteorg" chart repository
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈
Error: no cached repository for helm-manager-06b85d82fd48330fadb64937df35abfe0b44014d605c8ab4244eb6e6fcc29780 found. (try 'helm repo update'): open <root>/.cache/helm/repository/helm-manager-06b85d82fd48330fadb64937df35abfe0b44014d605c8ab4244eb6e6fcc29780-index.yaml: no such file or directory
make: *** [Makefile:39: helm] Error 1

And creates no changes that I can commit

@eapolinario
Copy link
Contributor

@cjidboon94 , can you merge master (to bring in d5784e5) and re-run?

@cjidboon94
Copy link
Contributor Author

I see new tests failing after having bumped up the chart. Should I have done that? Or should I revert/undo the latest commit?

@davidmirror-ops
Copy link
Contributor

@cjidboon94 Please revert :)

@lowc1012
Copy link
Contributor

@cjidboon94 @davidmirror-ops
I'll create a PR to fix helm check CI errors, plz stay tuned

@lowc1012
Copy link
Contributor

@davidmirror-ops @pingsutw
Could you help review this PR?

@cjidboon94
I think after my PR is merged, you can revert ddfe840 and rebase master branch to pass CI

@cjidboon94 cjidboon94 force-pushed the feature/5016-gke-starter-values branch from 2f4cd0d to 65b069d Compare March 20, 2024 11:35
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
…templateUri

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
This reverts commit ddfe840.

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
…tance GPUs in GKE

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
@cjidboon94 cjidboon94 force-pushed the feature/5016-gke-starter-values branch from 65b069d to 1224e69 Compare March 21, 2024 20:55
Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
@cjidboon94
Copy link
Contributor Author

@davidmirror-ops Since #5072 is merged, I have rebased myself onto the new master and the tests aren't failing anymore. I think this is ready to merge now?

Copy link
Contributor

@davidmirror-ops davidmirror-ops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@davidmirror-ops davidmirror-ops merged commit b6f35ad into flyteorg:master Mar 22, 2024
52 of 53 checks passed
Copy link

welcome bot commented Mar 22, 2024

Congrats on merging your first pull request! 🎉

yubofredwang pushed a commit to yubofredwang/flyte that referenced this pull request Mar 26, 2024
* add first version of gke-starter values file

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* remove AWS metadata env var, add service account annotations, update templateUri

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* let user fill in GCP project ID in templateUri

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* add link to GCP workload identity configuration

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* run make helm

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* bump flyte-binary chart version to v0.1.11

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* Revert "bump flyte-binary chart version to v0.1.11"

This reverts commit ddfe840.

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* add default configuration to allow scheduling on gpu nodes in GKE

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* add default gpu-partition-size label to allow scheduling on multi-instance GPUs in GKE

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* run make helm

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

* fix linting errors

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>

---------

Signed-off-by: Cornelis Boon <cjidboon94@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core feature] Add a gke-starter.yaml file for minimal deployment of flyte-binary on GKE clusters
4 participants