Add script for component image release #571
Conversation
|
/assign @hongye-sun @gaoning777 |
.release.cloudbuild.yaml
Outdated
| @@ -1,4 +1,4 @@ | |||
| # Copyright 2018 Google LLC | |||
| # Copyright 2018 Google LLC | |||
There was a problem hiding this comment.
Why adding a tab/space?typo?
| @@ -116,229 +116,6 @@ steps: | |||
| id: 'copyPythonSDKToLatest' | |||
| waitFor: ['copyPythonSDKLocal'] | |||
|
|
|||
| # Pull and retag the images for the Dataflow-based pipeline components | |||
There was a problem hiding this comment.
Why not do all the job in the release yaml in the release script?
There was a problem hiding this comment.
the system images doesn't have the double release problem.
| @@ -65,7 +65,7 @@ spec: | |||
| spec: | |||
| containers: | |||
| - name: deploy | |||
| image: gcr.io/ml-pipeline/bootstrapper:0.1.4 #TODO-release: update the release tag for the next release | |||
There was a problem hiding this comment.
maybe use hash for the bootstrapper image?
There was a problem hiding this comment.
this change is specifically for sample images. the bootstrapper is deprecating. this file should be removed in a separate PR
|
/assign @Ark-kun |
components/release.sh
Outdated
| ${FROM_GCR_PREFIX}${image}:${COMMIT_SHA} ${TARGET_IMAGE} | ||
|
|
||
| # Update the code | ||
| find "${PARENT_PATH}/../samples" -type f -exec sed -i -e "s|${TARGET_IMAGE_BASE}:\([a-zA-Z0-9_.-]\)\+|${TARGET_IMAGE}|g" {} \; |
There was a problem hiding this comment.
find -exec won't fail on errors. We need to replace this with find | while read file; do ...; done
See my comment here: https://github.com/kubeflow/pipelines/blob/master/backend/Dockerfile#L27
|
/test kubeflow-pipeline-e2e-test |
|
/test kubeflow-pipeline-e2e-test |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: IronPan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
1 similar comment
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: IronPan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm |
|
Can we maybe change the release script name to release_component.sh? To my understanding, the script releases the component images and updates the samples to reflect the component changes. |
| "ml-pipeline-dataflow-tfma" | ||
| "ml-pipeline-kubeflow-deployer" | ||
| "ml-pipeline-kubeflow-tf-trainer" | ||
| "ml-pipeline-kubeflow-tf" |
There was a problem hiding this comment.
could you help to add one more image here called: ml-pipeline-kubeflow-tf-trainer-gpu?
* Define a script to close obsolete PRs to update an application. * In the event there are multiple PRs open to update a Kubeflow application we want to close the older PRs; so there is a single open PR updating the application to the newest code. Related to kubeflow#571 * Define a script to close obsolete PRs to update an application. * In the event there are multiple PRs open to update a Kubeflow application we want to close the older PRs; so there is a single open PR updating the application to the newest code. Related to kubeflow#571 * Setup a def namespace for use with apps-cd. * Update update_kf_apps.py to close old PRs on each sync. * Bake the source code into the docker image rather than using a wrapper script to sync the code from git. * Sync'ing the code from git became to difficult to reason about once we start splitting the source code across multiple repositories * We now depend on github/kubeflow/code-intelligence for utilities for working with GitHub Apps * Using a docker image also ensures we don't get broken suddenly when new changes are in place * In the future we could use github actions to automate updating the deployment on postsubmits * Turn app-pipeline.template.yaml into a ConfigMap * This allows better versioning * We can rely on kustomize to create a configmap with a hash based on the contents * kustomize will then reference the config map using its hash. As a result a rolling update is triggered whenever the hash contents changes. * This makes it easier to handle rollous and updates. Define a dev instance of the update KF apps infrastructure to facilitate development * Use profiles in skaffold. * update_kf_apps.py in dev uses a config map now to ubtain app-pipeline.template.yaml rather than fetching it from git * This makes it much easier to test changes in the dev instance Fix a bunch of bugs preventing update_kf_apps.py from working * Update requirements.txt with a bunch of missing packages. * Fix some imports in update_kf_apps.py * Need to set resource requests for the build pods otherwise builds get CPU starved and take forever. Miscellaneous * Create a tool to copy secrets between namespaces from GCS * Fix lint. * Due to kubeflow#460 we need to disable pylint.
* Add namespace selector for pod mutating webhook * Enable namespace label match based on Env Var * Rename env var name * Document this feature * fix typo error
move kubeclient test case to e2e toolchain pipelines
This resolves the double release issue in the component image release process
To create a new release of the component image, run the release.sh. See more details in the release.sh comment on how to run the script
This change is