Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-5879 Admission Webhook 1.22 Compatibility #6459

Merged

Conversation

aaron-arellano
Copy link
Contributor

Updates the admission webhook to be compatible with kubernetes v1.22 per issue #6428.

@google-cla
Copy link

google-cla bot commented Apr 29, 2022

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

For more information, open the CLA check for this pull request.

@aws-kf-ci-bot
Copy link

Hi @aaron-arellano. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@aaron-arellano
Copy link
Contributor Author

/assign @kimwnasptd

Copy link
Contributor

@alexlatchford alexlatchford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely not my area of expertise but looks great to me @aaron-arellano, thanks for tackling this!

components/admission-webhook/main.go Show resolved Hide resolved
Copy link
Member

@Arhell Arhell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test

@kimwnasptd
Copy link
Member

Thanks for the contribution @aaron-arellano and thank you for your time reviewing this @alexlatchford!

I'll also take a look on Tuesday to wrap the 1.22 effort up

@aaron-arellano
Copy link
Contributor Author

@kimwnasptd any updates on getting this review? Thanks!

@aaron-arellano
Copy link
Contributor Author

@kimwnasptd will this be getting a review soon? We are looking to merge the changes on our end ASAP once the contribution gets approved here. Thanks

Copy link
Member

@kimwnasptd kimwnasptd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the only questions I had. I'll also compile and apply these to a K8s 1.22 cluster to verify.

@aaron-arellano one last request, could you also update the Makefile to:

  1. Use the TAG ?= $(shell git describe --tags --always)
  2. Rename build-gcr to docker-build
  3. Add one more rule docker-push that just pushes the image

### admissionregistration.k8s.io/v1 default failurePolicy
In adopting `admissionregistration.k8s.io/v1` for the `MutatingWebhookConfiguration` we accept the default value
for `failurePolicy` to be `Fail` per [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy). Upon testing this default feature it was discovered if the `AdmissionWebhook`'s `mutating-webhook-configuration` failed to mutate a pod then the pod would fail to start, its associated `Deployment` or `StatefulSet` would continually attempt to create the pod until the process that created the pod was terminated. This continuous failure to mutate the target pod may block other target pods from mutation
until the failing process ends.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This continuous failure to mutate the target pod may block other target pods from mutation
until the failing process ends.

Could you clarify a little bit more what you mean here? By other Pods you are not referring to other unrelated Pods in the cluster right?

IIUC, as you described, the Deployment/StatefulSet will be retrying to create the Pod which will keep failing. Are there other side effects to other pods?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep. will clarify. Pretty much any pod that is targeted by the admission webhook for mutation is effected and will fail to start if there are configuration collisions or some other issue. Already running pods are not effected at all, again only pods being created after the policy change. Will add this to Readme for clarification.

env:
items:
type: object
x-kubernetes-preserve-unknown-fields: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you need to make x-kubernetes-preserve-unknown-fields: true for these fields or did you intentionally want to block pruning here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if I did not add x-kubernetes-preserve-unknown-fields: true I noticed the fields would get pruned in our pod-defaults, we would end up having empty pod-defaults though they were not prior to the upgrade. It seems the pruning mechanism recognizes string, boolean and integer no problem but did not recognize array or object. Once I added these tags the pod-defaults were generated as normal.

@aaron-arellano
Copy link
Contributor Author

@kimwnasptd should I also update the image url in the makefile to public.ecr.aws/j1r0q0g6/? I saw that was updated in the notebook controller v1.22 code change.

@aaron-arellano
Copy link
Contributor Author

@kimwnasptd ping for review.

@kimwnasptd
Copy link
Member

Just tested this locally, as well as in a K8s 1.22 cluster.

Thanks for the work @aaron-arellano!

/lgtm
/approve

@google-oss-prow google-oss-prow bot added the lgtm label May 13, 2022
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexlatchford, kimwnasptd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 8be0d98 into kubeflow:master May 13, 2022
alembiewski pushed a commit to mesosphere/kubeflow that referenced this pull request Jun 16, 2022
* init

* update dependancies

* refine

* update webhook

* update readme and makefile

(cherry picked from commit 8be0d98)
apo-ger pushed a commit to arrikto/kubeflow that referenced this pull request Jan 26, 2023
* init

* update dependancies

* refine

* update webhook

* update readme and makefile

Cherry-picked-from: 8be0d98
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
chuckbelisle pushed a commit to StatCan/kubeflow that referenced this pull request Jul 11, 2023
* release: Images for the 1.5.0 tag (#6398)

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* added env for app secure cookies (#6395)

* build(deps): bump github.com/gogo/protobuf from 1.1.1 to 1.3.2 in /components/tensorboard-controller (#6424)

Bumps [github.com/gogo/protobuf](https://github.com/gogo/protobuf) from 1.1.1 to 1.3.2.
- [Release notes](https://github.com/gogo/protobuf/releases)
- [Commits](https://github.com/gogo/protobuf/compare/v1.1.1...v1.3.2)

---
updated-dependencies:
- dependency-name: github.com/gogo/protobuf
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump notebook from 6.4.0 to 6.4.10 in /components/example-notebook-servers/jupyter (#6431)

Bumps [notebook](http://jupyter.org) from 6.4.0 to 6.4.10.

---
updated-dependencies:
- dependency-name: notebook
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump minimist from 1.2.5 to 1.2.6 in /components/crud-web-apps/tensorboards/frontend (#6417)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump minimist from 1.2.5 to 1.2.6 in /components/crud-web-apps/jupyter/frontend (#6429)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump follow-redirects from 1.14.1 to 1.14.8 in /components/crud-web-apps/tensorboards/frontend (#6357)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.1 to 1.14.8.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.1...v1.14.8)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump ajv from 6.10.2 to 6.12.6 in /components/crud-web-apps/volumes/frontend (#6358)

Bumps [ajv](https://github.com/ajv-validator/ajv) from 6.10.2 to 6.12.6.
- [Release notes](https://github.com/ajv-validator/ajv/releases)
- [Commits](https://github.com/ajv-validator/ajv/compare/v6.10.2...v6.12.6)

---
updated-dependencies:
- dependency-name: ajv
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump ajv from 6.10.2 to 6.12.6 in /components/crud-web-apps/jupyter/frontend (#6359)

Bumps [ajv](https://github.com/ajv-validator/ajv) from 6.10.2 to 6.12.6.
- [Release notes](https://github.com/ajv-validator/ajv/releases)
- [Commits](https://github.com/ajv-validator/ajv/compare/v6.10.2...v6.12.6)

---
updated-dependencies:
- dependency-name: ajv
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump lodash-es from 4.17.11 to 4.17.21 in /components/crud-web-apps/tensorboards/frontend (#6402)

Bumps [lodash-es](https://github.com/lodash/lodash) from 4.17.11 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.11...4.17.21)

---
updated-dependencies:
- dependency-name: lodash-es
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump minimist from 1.2.5 to 1.2.6 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6416)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump minimist from 1.2.5 to 1.2.6 in /components/crud-web-apps/volumes/frontend (#6423)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump undefsafe from 2.0.2 to 2.0.5 in /components/centraldashboard (#6342)

Bumps [undefsafe](https://github.com/remy/undefsafe) from 2.0.2 to 2.0.5.
- [Release notes](https://github.com/remy/undefsafe/releases)
- [Commits](https://github.com/remy/undefsafe/compare/v2.0.2...v2.0.5)

---
updated-dependencies:
- dependency-name: undefsafe
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump url-parse from 1.5.1 to 1.5.10 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6378)

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.5.1 to 1.5.10.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](https://github.com/unshiftio/url-parse/compare/1.5.1...1.5.10)

---
updated-dependencies:
- dependency-name: url-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump url-parse from 1.5.1 to 1.5.10 in /components/crud-web-apps/tensorboards/frontend (#6379)

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.5.1 to 1.5.10.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](https://github.com/unshiftio/url-parse/compare/1.5.1...1.5.10)

---
updated-dependencies:
- dependency-name: url-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps-dev): bump karma from 6.3.4 to 6.3.16 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6386)

Bumps [karma](https://github.com/karma-runner/karma) from 6.3.4 to 6.3.16.
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v6.3.4...v6.3.16)

---
updated-dependencies:
- dependency-name: karma
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps-dev): bump karma from 6.3.4 to 6.3.16 in /components/crud-web-apps/tensorboards/frontend (#6387)

Bumps [karma](https://github.com/karma-runner/karma) from 6.3.4 to 6.3.16.
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v6.3.4...v6.3.16)

---
updated-dependencies:
- dependency-name: karma
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump moment from 2.24.0 to 2.29.2 in /components/centraldashboard (#6432)

Bumps [moment](https://github.com/moment/moment) from 2.24.0 to 2.29.2.
- [Release notes](https://github.com/moment/moment/releases)
- [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/moment/moment/compare/2.24.0...2.29.2)

---
updated-dependencies:
- dependency-name: moment
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump url-parse from 1.5.1 to 1.5.10 in /components/crud-web-apps/volumes/frontend (#6380)

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.5.1 to 1.5.10.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](https://github.com/unshiftio/url-parse/compare/1.5.1...1.5.10)

---
updated-dependencies:
- dependency-name: url-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump url-parse from 1.4.7 to 1.5.10 in /components/centraldashboard (#6377)

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.4.7 to 1.5.10.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](https://github.com/unshiftio/url-parse/compare/1.4.7...1.5.10)

---
updated-dependencies:
- dependency-name: url-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps-dev): bump karma from 6.3.4 to 6.3.16 in /components/crud-web-apps/volumes/frontend (#6388)

Bumps [karma](https://github.com/karma-runner/karma) from 6.3.4 to 6.3.16.
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v6.3.4...v6.3.16)

---
updated-dependencies:
- dependency-name: karma
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps-dev): bump karma from 6.3.4 to 6.3.16 in /components/crud-web-apps/jupyter/frontend (#6389)

Bumps [karma](https://github.com/karma-runner/karma) from 6.3.4 to 6.3.16.
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v6.3.4...v6.3.16)

---
updated-dependencies:
- dependency-name: karma
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump follow-redirects from 1.14.1 to 1.14.9 in /components/crud-web-apps/jupyter/frontend (#6441)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.1 to 1.14.9.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.1...v1.14.9)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump nanoid from 3.1.23 to 3.3.2 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6437)

Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.23 to 3.3.2.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.1.23...3.3.2)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump follow-redirects from 1.14.1 to 1.14.9 in /components/crud-web-apps/volumes/frontend (#6439)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.1 to 1.14.9.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.1...v1.14.9)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump shelljs from 0.8.3 to 0.8.5 in /components/centraldashboard (#6284)

Bumps [shelljs](https://github.com/shelljs/shelljs) from 0.8.3 to 0.8.5.
- [Release notes](https://github.com/shelljs/shelljs/releases)
- [Changelog](https://github.com/shelljs/shelljs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/shelljs/shelljs/compare/v0.8.3...v0.8.5)

---
updated-dependencies:
- dependency-name: shelljs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump shelljs from 0.8.4 to 0.8.5 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6285)

Bumps [shelljs](https://github.com/shelljs/shelljs) from 0.8.4 to 0.8.5.
- [Release notes](https://github.com/shelljs/shelljs/releases)
- [Changelog](https://github.com/shelljs/shelljs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/shelljs/shelljs/compare/v0.8.4...v0.8.5)

---
updated-dependencies:
- dependency-name: shelljs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump nanoid from 3.1.23 to 3.3.2 in /components/crud-web-apps/volumes/frontend (#6442)

Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.23 to 3.3.2.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.1.23...3.3.2)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump nanoid from 3.1.23 to 3.3.2 in /components/crud-web-apps/tensorboards/frontend (#6443)

Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.23 to 3.3.2.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.1.23...3.3.2)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump tensorflow from 2.5.0 to 2.5.3 in /components/example-notebook-servers/jupyter-tensorflow (#6445)

Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.5.0 to 2.5.3.
- [Release notes](https://github.com/tensorflow/tensorflow/releases)
- [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md)
- [Commits](https://github.com/tensorflow/tensorflow/compare/v2.5.0...v2.5.3)

---
updated-dependencies:
- dependency-name: tensorflow
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump url-parse from 1.5.1 to 1.5.10 in /components/crud-web-apps/jupyter/frontend (#6381)

Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.5.1 to 1.5.10.
- [Release notes](https://github.com/unshiftio/url-parse/releases)
- [Commits](https://github.com/unshiftio/url-parse/compare/1.5.1...1.5.10)

---
updated-dependencies:
- dependency-name: url-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump follow-redirects from 1.14.1 to 1.14.9 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6438)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.1 to 1.14.9.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.1...v1.14.9)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump follow-redirects from 1.9.0 to 1.14.9 in /components/centraldashboard (#6440)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.9.0 to 1.14.9.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.9.0...v1.14.9)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump async from 2.6.3 to 2.6.4 in /components/crud-web-apps/tensorboards/frontend (#6446)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump async from 2.6.3 to 2.6.4 in /components/crud-web-apps/common/frontend/kubeflow-common-lib (#6447)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump async from 2.6.3 to 2.6.4 in /components/crud-web-apps/jupyter/frontend (#6448)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump shelljs from 0.8.4 to 0.8.5 in /components/crud-web-apps/tensorboards/frontend (#6286)

Bumps [shelljs](https://github.com/shelljs/shelljs) from 0.8.4 to 0.8.5.
- [Release notes](https://github.com/shelljs/shelljs/releases)
- [Changelog](https://github.com/shelljs/shelljs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/shelljs/shelljs/compare/v0.8.4...v0.8.5)

---
updated-dependencies:
- dependency-name: shelljs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump shelljs from 0.8.4 to 0.8.5 in /components/crud-web-apps/volumes/frontend (#6287)

Bumps [shelljs](https://github.com/shelljs/shelljs) from 0.8.4 to 0.8.5.
- [Release notes](https://github.com/shelljs/shelljs/releases)
- [Changelog](https://github.com/shelljs/shelljs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/shelljs/shelljs/compare/v0.8.4...v0.8.5)

---
updated-dependencies:
- dependency-name: shelljs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump node-fetch from 2.6.0 to 2.6.7 in /components/centraldashboard (#6307)

Bumps [node-fetch](https://github.com/node-fetch/node-fetch) from 2.6.0 to 2.6.7.
- [Release notes](https://github.com/node-fetch/node-fetch/releases)
- [Changelog](https://github.com/node-fetch/node-fetch/blob/main/docs/CHANGELOG.md)
- [Commits](https://github.com/node-fetch/node-fetch/compare/v2.6.0...v2.6.7)

---
updated-dependencies:
- dependency-name: node-fetch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump nanoid from 3.1.23 to 3.3.2 in /components/crud-web-apps/jupyter/frontend (#6444)

Bumps [nanoid](https://github.com/ai/nanoid) from 3.1.23 to 3.3.2.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.1.23...3.3.2)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump async from 2.6.3 to 2.6.4 in /components/crud-web-apps/volumes/frontend (#6449)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix kubernetes python client to v22.6.0 (#6422)

Co-authored-by: Benjamin Tan <benjamin.tan@tech.jago.com>

* Support K8s 1.22 in notebook controller (#6374)

Fix https://github.com/kubeflow/kubeflow/issues/6366

Migrating to Kubebuilder v3 leads to the following changes:
- Add .dockerignore file.
- Upgrade Go version from v1.15 to v1.17.
- Adapt Makefile.
- Add image (build + push) target to makefile.
- Upgrade EnvTest to use K8s v1.22.
- Update PROJECT template.
- Migrate CRD apiVersion from v1beta to v1.
- Add livenessProbe and readinessProbe to controller manager.
- Upgrade controller-runtime from v0.2.0 to v0.11.0.

Other changes:
- Build image using public.ecr.aws registry instead of gcr.io.
- Update README.md documentation.
- Update 3rd party licences.
- Fix notebook.spec description.
- Add 3 sample notebooks (v1, v1alpha1 and v1beta1).

Signed-off-by: Samuel Veloso <svelosol@redhat.com>

* Fix vendortooltip typo (#6426)

* Cherry-pick latest commits to the 1.5-branch (#6354)

* Fix(manifests): Upgrade rbac.authorization.k8s.io from v1beta1 to v1 (#6261)

* proposal: Extend Notebook Controller to expose idleness for Jupyter (#6295)

* proposal: Extend Notebook Controller to expose idleness for Jupyter (#6270)

Provide a design doc as a proposal for extending Notebook Controller to
expose idleness for Jupyter. Our proposal is in markdown format and follows
the guidelines of the kubeflow/components/proposal/README.md guide.

You can view the #6270 issue in the following link:
https://github.com/kubeflow/kubeflow/issues/6270

Signed-off-by: Athanasios Markou <athamark@arrikto.com>

* review: change the title of the proposal

Change the title of the proposal to only include the
proposed new feature. The new title of the proposal
will now be "Expose Idleness Information for Jupyter
Notebooks".

* review: rename the proposal markdown file

We want to give a more specific name to the markdown
which contains the proposal. Since this proposal
emphasizes on a feature regarding the Jupyter Notebooks,
the new name will be:

20220121-jupyter-notebook-idleness.md

* Synchronize jupyter-web-application role with clusterrole (#6241)

* Update role.yaml

* Update role.yaml

* Update cluster-role.yaml

* Kubeflow Roadmap update - with 1.5 details (#6266)

* Kubeflow Roadmap update - with 1.5 details

These proposed changes include: identifying that 1.4.1 has been delivered, provides themes for 1.5 and provides details of major features in 1.5 by working group.   This is an initial proposal that needs review by the working group leads.

* correct formatting in KFP features

Moved KFP features under KFP Control Flow doc

* updating KFP section

updating KFP references with updates from KFP team

* Updated the 1.5 release date to March

updated the 1.5 release date to March

* Update ROADMAP.md

change Hyperparameter leader election to Katib leader election

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update ROADMAP.md

improve description and details of feature for metrics collector

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* Update Katib description for Early stopping in 1.5

updating with Andrey's suggestion (but without the world proper).   * Validation for Early Stopping algorithm settings helps users to proper reduce model overfitting

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* notebooks: Extend Notebook Controller to expose idleness for Jupyter (#6297)

* notebooks: Update image's tag in make

Modify Makefile to update properly the TAG
based on the git TAG.

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Expose last-activity

Extend the notebook-controller to:
* cull idle Notebook Servers based on their new `last-activity`
  annotation
* expose the last activity of each Notebook Server as an annotation
  on the metadata of the corresponding CR object

Modify notebook_controller.go to:
* update the Last Activity of each Notebook Server that has a
  Running pod
* delete the Last Activity Annotation for every Notebook Server
  that does not have a Running pod

Extend culler.go to:
* perform culling based on the new `last-activity` annotation and
  not based on the `/api/status` endpoint.
* update the last activity of a Notebook Server, based on the
  kernels' execution states.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Athanasios Markou <athamark@arrikto.com>

* notebooks: Introduce a DEV env var

We introduce a DEV ENV var to allow admins
develop and test on their local machine their
custom Notebook Controller.
We provide information and instructions inside
the components/notebook-controller/README.md.

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Add unit tests for last-activity

* Introduce new tests for allKernelsAreIdle()
* Extend the tests for NotebookIsIdle() and for
  NotebookNeedsCulling().

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: UpdateNotebookLastActivityAnnotation()

Ensure that UpdateNotebookLastActivityAnnotation() does not return
"true". This function should not return any value.

Signed-off-by: Athanasios Markou <athamark@arrikto.com>

* jwa: Rework the Storage API of the web app (#6321)

* wa(back): Add helper for deserializing JSON obj

In some cases we might need to construct Python k8s lib objects from the
JSONs that are provided by clients. I.e. the UI will be sending a PVC
object in json format, so the backend will need to create the
corresponding client.V1PersistentVolumeClaim object and submit it.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* wa(back): Serialization helper

Add helper function for converting a k8s-client object into a dict that
can be sent as an HTTP response.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* wa(back): Add dry run to Notebooks and PVCs

The backend will need to be able to create objects with dry-run, in
order to ensure they are valid. The backend will need to check that both
the Notebook and the PVCs can be created beforehand.

This way we avoid the scenario where we create PVCs but the Notebook
fails to be created, and the PVCs are never garbage collected.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* wa(back): Update kubernetes to 0.17

In order to support dry-run we must use the 0.17 version of the Python
k8s client.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* wa(back): Extend api module to patch pvcs

The backend will need to be able to PATCH PVCs in order to set the
ownerReference to the Notebook that mounts the PVCs.

Ref: arrikto/dev/issues/386#issuecomment-856700392

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* jwa(back): Work with new Volumes API

The backend API should not add any more layers of abstractions on top of
the K8s API. The backend should expect the client/UI to be sending the
entire PVC spec of a new PVC.

Refs: arrikto/dev/issues/386

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* jwa(back): Add unittests for new volumes API

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* jwa(back): Extend the PVC info returned

We want to show both the access mode and size of the existing PVCs, when
a user clicks on the dropdown to select which PVC to mount.

The backend will need to provide this information to the frontend. We
don't want to send the K8s list of PVCs since this will result in a lot
of unnecessary data to be sent.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Ilias Katsakioris <elikatsis@arrikto.com>

* jwa(front): Add proxy config for Rok

When developing the Rok flavor locally we will need to be able to open
the Rok chooser. This can be done by using Angular/webpack proxy to
bring the exposed rok service and the app under the same domain.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* jwa(front): Remove card from form

The form of the app should not be a big card, but a normal form.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* jwa(front): Install AceModule for yaml editing

Install AceModule to allow users to edit yamls of objects.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* wa(front): Change the styling of form sections

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* jwa(front): Create common volume components

Component for:
* New PVC and configuring its spec
* Attaching an existing PVC in a Notebook

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* jwa(front): Update Rok form for new Volume API

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* jwa(front): Mark inputs as dirty when restoring Lab

When the UI autofills the form with values from a JupyterLab snapshot
then it should mark the touched fields as dirty. This way if a field has
errors the UI will make that input red.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Tasos Alexiou <tasos@arrikto.com>

* jwa: Update ConfigMap in manifests

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* jwa(front): Fix format errors

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* profiles: Update the permissions for notebook idleness (#6335)

Extend the Profiles Controller to give permissions to Notebooks
controller for making GET requests to notebook's /api/kernels endpoint.

Refs https://github.com/kubeflow/kubeflow/blob/master/components/proposals/20220121-jupyter-notebook-idleness.md

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Graceful handling of events (#6338)

* notebooks: Handle events gracefully

The controller is not exiting the reconciliation loop after it has
re-emitted a Pod/STS Event as a Notebook Event. This results in the
controller to later on try and GET a Notebook with the name of the Event
that triggered the reconciliation loop.

The controller should exit the reconciliation function once it has
emitted the event.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Don't reconcile on deleted events

We don't want to trigger the reconciliation function when an event gets
deleted.

If a Notebook would be deleted then the underlying events would
be deleted as well, which results in the reconcile function to get
triggered and try to GET Events and Notebooks with the name of the
deleted event.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Fix endless restarts (#6341)

* notebooks: Update notebook if timestamp changed

We don't want to be updating the spec of the notebook if the timestamp
hasn't changed, since this will lead to constant updates and
reconciliation loops.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Use a deep-copy of the notebook spec

The controller should use a deep-copy of the notebook spec when
calculating the spec for the StatefulSet. If not then we could
update the notebook object without wanting it, since the spec could have
been changed when calculating the STS spec.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Add prefix env var only if missing

The controller should be setting OR updating the NB_PREFIX env var.
Previously it would always blindly append it to the spec, which could
result in double entries for the same env var.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* releasing: Update tags for v1.5.0-rc.1 (#6343)

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* fix error comparison between pointer and pointer in "CopyStatefulSetFields" (#6195)

error comparison between pointer and pointer in "CopyStatefulSetFields"

* [bugfix]: update unsupported links to latest pages (#6192)

* centraldashboard: Add KServe overlay (#6383)

Configure the dashboard to use the KServe app instead of the KFServing
0.6.1 one.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Don't reconcile on Events deletion (#6391)

The controller should not trigger the reconcile loop when an Event is
deleted. Previously the controller would run the reconciliation loop on
any event deletion.

This commit updates it to not run the loop for ANY event.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* releasing: Create v1.5.0-rc.2 images (#6394)

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* fixed typo

* fixed typo

* changed remove rc2

Co-authored-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Co-authored-by: mofanke <54242816+mofanke@users.noreply.github.com>
Co-authored-by: Jaeyeon Kim <anencore94@gmail.com>

* notebooks: notebook controller config RESTClient QPS and Burst in com… (#6453)

* AIP-5879 Admission Webhook 1.22 Compatibility (#6459)

* init

* update dependancies

* refine

* update webhook

* update readme and makefile

* KF conformance test driver (#6478)

* (Draft) KF conformance test framework

* Add env var to support debug output.

* Adressed code review comments.

* Moved tests from 1.0 to 1.5 folder.

* 1st update of Kubeflow Roadmap for Kubeflow 1.6 (#6456)

* 1st update of Kubeflow Roadmap for Kubeflow 1.6

@annajung @kimwnasptd @DomFleischmann I have created this PR as a quick way to update the Kubeflow Roadmap to include KF 1.6.   My view is that we can update this in a simple way and then add more details if needed.     Please review, improve as you see needed.   Thanks.   Jsoh

* accept anna's updates to ROADMAP.md

Co-authored-by: Anna <antheaj@vmware.com>

* Added Themes section

added Themes section and moved K8s 1.22 support to themes, removed x.y.z version details from each working group roadmap

* fixed formatting, removed version  on Notebook

fixed formatting, removed version placeholder on Notebook, removed 1.22 support statement at end of 1.6 sections (as we have added it to the Themes section)

* Added link to KFP CLI design doc

Added link to KFP CLI design doc

Co-authored-by: Anna <antheaj@vmware.com>

* Add GH actions to push images to a DockerHub registry after a merge (#6510)

* Add a github action to build & push JWA docker image after a merge. (#6502)

* Add a github action to build & push TWA docker images after a merge. (#6502)

* Add a github action to build & push VWA docker images after a merge. (#6502)

* Add a github action to build & push Tensorboard Controller docker image after a merge. (#6502)

* Add a github action to build & push Profile Controller docker image after a merge. (#6502)

* Add a github action to build & push PodDefaults docker image after a merge. (#6502)

* Add a github action to build & push Notebook Server docker images after a merge. (#6502)

* Add a github action to build & push Notebook Controller docker image after a merge. (#6502)

* Add a github action to build & push KFAM docker image after a merge. (#6502)

* Add a github action to build & push Central Dashboard docker image after a merge. (#6502)

* Update CentralDashboard GH action after review comments.

* Add a path for the common code in JWA Gh action after review comment.

* Update KFAM GH action after review comments.

* Update JWA GH action after review comments.

* Update Notebook Controller GH action after review comments.

* Update Notebook Server GH action after review comments.

* Update PodDefaults GH action after review comments.

* Update Profile Controller GH action after review comments.

* Update Tensorboard Controller GH action after review comments.

* Update TWA GH action after review comments.

* Update VWA GH action after review comments.

* Update credentials in all GH actions.

* Pin alpine repository version (#6494)

With the `latest-stable` version of alpine package repository, an attempt to build central dashboard image fails with:
```
 => ERROR [build 3/7] RUN apk add --no-cache bash@stable chromium@stable nss@stable     freetype@stable     harfbuzz@stable     ttf-freefont@stable     libstdc++@stable                                        4.4s
------
 > [build 3/7] RUN apk add --no-cache bash@stable chromium@stable nss@stable     freetype@stable     harfbuzz@stable     ttf-freefont@stable     libstdc++@stable:
#7 0.940 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
#7 1.239 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
#7 1.803 fetch http://nl.alpinelinux.org/alpine/latest-stable/community/x86_64/APKINDEX.tar.gz
#7 2.701 fetch http://nl.alpinelinux.org/alpine/latest-stable/main/x86_64/APKINDEX.tar.gz
#7 3.630 ERROR: unable to select packages:
#7 4.065   so:libmbedcrypto.so.7 (no such package):
#7 4.065     required by: librist-0.2.6-r1[so:libmbedcrypto.so.7]
------
executor failed running [/bin/sh -c apk add --no-cache bash@stable chromium@stable nss@stable     freetype@stable     harfbuzz@stable     ttf-freefont@stable     libstdc++@stable]: exit code: 4
make: *** [Makefile:42: build] Error 1
```

* feat(Jupyter): Show group one or two only if images exists (#6482)

* Upgrade API version of `Tensorboard` CRD to `v1` (#6406)

* Migrate tensorboard-controller to Kubebuilder v3

* Fix paths inside Docker context

* Remove test dependency from docker-build

* Switch to kustomize 3.2.0, fix image tag

* Fix namePrefix

* Rename deployments, remove namespaces

* Add runAsUser

* Make tensorboard image and istio gateway configurable

* Add GH actions for building manifests at any pull request (#6524)

* Add GH actions for building manifests at any pull request.

* Change names of GH actions and specify the manifests path.

* Fix manifests path in Profile Controller GH action.

* update jupyterlab to 3.4.3 (#6527)

* notebooks: Allow notebook controller to patch events (#6523)

* Fix typos in GH actions that build and push Docker images (#6541)

* Make GH actions build images on PR and push only on merge (#6543)

* Make GH actions build images on PR and push only on merge.

* Fix typo from rebase.

* Add GH actions for applying manifests at any pull request (#6530)

* Add GH actions for applying manifests at any PR.

* Add scripts for all the KinD GH actions.

* Give permission to run the scripts.

* Modify GH actions to use scripts for installations.

* Address review comments.

* Change GH actions based on review comments.

* Add newlines at the end of the script files.

* jwa: Expose a last-activity column (#6544)

We want to extend JWA to have a column for showing the `Last activity`
of a Notebook CR.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Reword image name in TWA Gh action (#6547)

* Fix VWA path in Gh action for building image (#6549)

* Add GH action for publishing all images when VERSION is updated (#6537)

* Add GH action for publishing all images when version is updated.

* Fix typos in GH action for building all images.

* Fix typo in tensorboards image.

* Update install command for code server extension (#6554)

Signed-off-by: Anna Jung (VMware) <antheaj@vmware.com>

* Introduce a mechanism to build all Kubeflow images (#6555)

* build: Update components makefiles for building

We'll create a top-level Makefile under components/ dir
that has the following rules:

* build-all:
  To build all images locally

* push-all:
  * We can use a specific REGISTRY and retag the images
  * Push all the images

This top-level Makefile will run the sub-Makefiles that every
component has for building and pushing the images.

We modified every sub-Makefile as follows:
* We don't use a registry in images by default
* Removed unused rules and vars
* Use the --dirty flag of git describe in TAG

        --dirty[=<mark>]
               Describe the working tree. It means describe HEAD and
               appends <mark> (-dirty by default) if the working tree
               is dirty.

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Create makefiles for notebook servers

The common starting point of building the notebook-server
images are the following 4 *base* images:
- base
- codeserver
- jupyter
- rstudio

All other server images need to build on top of them. We'll
dynamically pass the base images in every Dockerfile by
using an ARG IMG. We can set the value of this ARG during
docker build with the --build-arg CLI argument.

This way we build both the base images with a tag locally,
and then we pass that image as arg via the Makefile and build the rest

So we modified our building procedure as follows:
1. Build the base image since everything starts from there

2. Pass the base image as an ARG in the Dockerfiles of
jupyter, codeserver, rstudio images and build on top

3. Pass the base images in all other server images and build
on top

For that we will:

1. Create a Makefile for each of the notebook servers, in each folder
   a. Each makefile will be responsible for building the bases and use args for passing them on

2. Use the central Makefile to call each Makefile from above

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* cherry-pick: Notebook server upstream fixes

Relevant upstream PR: https://github.com/kubeflow/kubeflow/pull/6466/files

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Fix Makefiles

- Remove build-gcp and build-gcr rules as we don't use them anywhere in
the project
- Fix code conficts

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Fix Dockerfiles for notbook-server images

We remove the previous logic of using already built images as bases.
The users must use only the Makefiles to pass the appropriate BASE_IMG
and build the images correctly.

Thus, we have Makefiles everywhere that:

- Can build any base image
- If an image requires another notebook base, then we first build that one using its makefile,
  and then use it as docker ARG for building the next one

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Fix notebook-controller Makefile

Removed a misplaced "|" char that breaks the Makefile

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Update GH action workflows

* Update workflow for notebook-server images:
  - Add a step for building all images by using the
  central-Makefile under components/example-notebook-servers/ dir.
  - Add a step for pushing all images by using the
  central-Makefile under components/example-notebook-servers/ dir.

* Update workflow for all Kubeflow images:
  - Add a step for building & pushing all images by using the
  top-level Makefile under components/ dir.

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Remove completely ECR references from images

Replace everywhere the "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers"
prefix with "kubeflownotebookswg"

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* build: Fix GH actions for Kubeflow components

Fix GH actions to use the updated make rules
when building the Kubeflow component images.

Remove the "docker.io" prefix when building with
GH action workflows

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* proposals: Show objects from all namespaces (#6531)

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Add support for `command` and `args` in `PodDefault` (#6542)

* Add support for setting custom "command" and "args" for Notebook servers (#20)

* Support command and args in PodDefault

* Improve logging, refactoring

* Add unit test for setCommandAndArgs, fix broken tests

* Cleanup

* Support K8s 1.22 in profile controller (#6491)

* profile-controller: Delete old kubebuilder v1 project

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Initialize Kubebuilder v3 project

go module name:
github.com/kubeflow/kubeflow/components/profile-controller

project domain:
kubeflow.org

Changes:

* Go Version got upgraded from v1.15 to v1.17
* With Kubebuilder V3 our project dependencies got
updated to a more up-to-date version.
* controller-runtime library got updated from v0.2.0 to v0.12.1.

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: create placeholder API types and controller logic files

We created the Profile API types
v1: kubebuilder create api --version v1 kind Profile
v1beta1: kubebuilder create api --version v1beta1 Profile

We copied over the API definitions from the old project
to the new one

(Note: Currently KFAM uses the v1beta1 version of Profile
to provide fine-grain user-namespace level access control)

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: copy controllers logic from old project to the new one

* Copied the files under controller/ dir from the old to our new project

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Code modifications to fix profile-controller

With Kubebuilder V3 the dependencies of our project got updated
to a more up-to-date version. Newer versions of certain packages
resulted to broken code parts that we had to fix. More specifically:

* In this project we are using a namespace-labels-data ConfigMap
to set certain labels on every Profile namespace. Every change in
this ConfigMap produces an event and for every event we construct
a list of reconcile.Requests that contains the name and namespace
of every Profile in our system.

Thus we introduced a mapEventToRequest MapFunc (profile_controller.go)
that maps an event to reconcile requests for all Profiles. This change
was necessary because controller-runtime library got updated from v0.6.3
to v0.12.1

* Kubebuilder V3 introduced for the profile-controller:
1) a livenessprobe (/healthz) on port :8081
2) a readinessProbe (/readyz) on port :8081

We have a conflict here as KFAM Service uses the :8081 port as well.
Therefore, we switched the port to :9876 for the above Probes.

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Fix manifests, Makefile, Dockerfile

Makefile:
* Add image (build + push) target
* Add test-overlays target
* Build image using public.ecr.aws registry instead of gcr.io.

Dockerfile:
* fix Dockerfile to build our controller correclty

Manifests
* Add base/ and overlays/ manifests
* Fix the manifests to be in sync with upstream

other changes:
Update README.md file

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Add test files

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* profile-controller: Fix Makefile

* Set the correct IMG:TAG
* Remove test dependency

Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>

* Fix typo in notebook_controller.go (#6577)

* Use controller-gen for PodDefaults (#6578)

* poddefaults: Use controller-gen for generation

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* poddefaults: Generate deepcopy files

Use the `make generate` make rule to automatically generate the deepcopy
files.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* poddefaults: Generate CRD

Use the controller-gen to generate the CRD, instead of manually updating
it.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* crud-web-apps: Migrate from tslint to eslint (#6464)

* build: Migrate tslint to eslint

* crud webapps - migrate tslint to eslint

* fix npm versions

* fix npm deps

* Reload certificates in admission-webhook (#6581)

Reload server certificates if they are changed on disk. Use certwatcher
from controller-runtime that already implements this functionality.

Closes #5560.

Signed-off-by: Manolis Androulidakis <manolis@arrikto.com>

* Create separate workflows for each notebook server (#6571)

* Add separate GH actions for all notebook servers.

* Keep only the workflows for the images that appear in JWA's ConfigMap.

* Ensure workflows trigger when the base image files change.

* crud-web-apps: Apply ng lint (#6580)

* Fix ng lint (#6586)

* fix lint errors

* fix lint

* Update package-lock.json

* Update package-lock.json

* Add workflow for frontend lint

* Fix legacy of Python version information in README files. (#6561)

* Update few packages and images to fix critical CVEs in centraldashboard component (#6559)

Signed-off-by: MessKon <messiskon@gmail.com>

* Bump Jupyter web app to Python 3.8. (#6562)

* Fix(Jupyter): Fix the ImageGroup object path (#6596)

* notebooks: Fix notebook endless restarts (#6337) (#6603)

* Cast nvidia.com/gpu value as string respective of K8s Resource Spec (#6051)

* Fix #6616: Typo on New Notebook page (#6620)

* JWA(front): Fix image group one and two not showing (#6625)

Fix image group one and two not showing in JWA's New notebook form, even
when the avaialable config provides value for them.

Signed-off-by: Orfeas Kourkakis <orfeas@arrikto.com>

Signed-off-by: Orfeas Kourkakis <orfeas@arrikto.com>

* fix Profile Plugins functionality (#6624)

* fix profile plugin loading

* add proper sample

* add GH action for unit tests

* rename GH job name

* Fix #6528: Mirroring Pod conditions to Notebook (#6619)

* Fix #6528: Mirroring Pod conditions to Notebook

* Added missing fields which are part of PodConditions into NotebookConditions

* Added suggested changes

* Fix #6056: Update Notebook status properly (#6628)

* Fix #6056: Update Notebook status properly

Signed-off-by: Apostolos Gerakaris apoger@arrikto.com

* Added suggested code changes

Signed-off-by: Apostolos Gerakaris apoger@arrikto.com

* notebook-controller: Add unit tests

*Introduce basic unit tests for "createNotebookStatus" function
*Add GH action for unit tests

Signed-off-by: Apostolos Gerakaris apoger@arrikto.com

* Fix PodCoditionsMirroringToNotebook & Unit-tests

We encountered an error during testing. It seems that
the pod.status.conditions.condition.LastProbeTime remains
always null and so the controller ends up applying a Notebook
CR instance with null condition values.

Relevant Issues:
*https://github.com/kubernetes/kubernetes/issues/109958
*https://github.com/kubernetes/kubernetes/issues/79402
*https://github.com/kubernetes/kubernetes/issues/14393

Fix: Check if the Pod's condition.LastProbeTime
and condition.LastTransitionTime timestamp fields are null.
If so, initialize them so we dont end up applying
a Notebook instance with null condition values.

Other changes:
*Fix basic unit tests
*Introduced a unit test for the case where Notebook's Pod
 is unschedulable

Signed-off-by: Apostolos Gerakaris apoger@arrikto.com

Signed-off-by: Apostolos Gerakaris apoger@arrikto.com

* Fix invalid kustomization patch introduced for profile-controller manager (#6604)

* update stablebot configs (#6634)

* Extend PodDefaults for support for imagePullSecrets (#6600)

* poddefaults: add imagepullsecrets

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* poddefaults: Generate CRD/deepcopy files

Run make generate and make manifests to generate the correct CRD and
deepcopy files.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* update Kubeflow roadmap to show delivery of v1.6 (#6637)

* update Kubeflow roadmap to show delivery of v1.6

* Update ROADMAP.md

Co-authored-by: Anna <antheaj@vmware.com>

* Update ROADMAP.md

Co-authored-by: Anna <antheaj@vmware.com>

Co-authored-by: Anna <antheaj@vmware.com>

* Fix notebook culling (#6659)

The notebook controller writes the last-activity annotation
before culling the Notebook, however, doesn't remove this
annotation before start. This causes the Notebook to be culled
again before is has a chance to start.

Fix:
* calculate correctly the podFound variable and ensure its value
  its true only if the Pod is actually found. This way the culling
  annotation will be removed when there is no Pod.

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

* centraldashboard: Format code (#6671)

Format centraldashboard's code based on the eslint configuration.

Signed-off-by: Tasos Alexiou <tasos@arrikto.com>

Signed-off-by: Tasos Alexiou <tasos@arrikto.com>

* Docker -> OCI Update form-image.component.html (#6673)

Docker is dead and superseeded by OCI on all common Platforms

* chore(icons): the model icon is grey instead of black (#6656)

* build(deps): bump jose from 2.0.5 to 2.0.6 in /components/crud-web-apps/jupyter/frontend (#6654)

Bumps [jose](https://github.com/panva/jose) from 2.0.5 to 2.0.6.
- [Release notes](https://github.com/panva/jose/releases)
- [Changelog](https://github.com/panva/jose/blob/v2.0.6/CHANGELOG.md)
- [Commits](https://github.com/panva/jose/compare/v2.0.5...v2.0.6)

---
updated-dependencies:
- dependency-name: jose
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump jszip from 3.6.0 to 3.10.1 in /components/crud-web-apps/volumes/frontend (#6640)

Bumps [jszip](https://github.com/Stuk/jszip) from 3.6.0 to 3.10.1.
- [Release notes](https://github.com/Stuk/jszip/releases)
- [Changelog](https://github.com/Stuk/jszip/blob/main/CHANGES.md)
- [Commits](https://github.com/Stuk/jszip/compare/v3.6.0...v3.10.1)

---
updated-dependencies:
- dependency-name: jszip
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* kwa(front): Fixes needed for Katib unit tests (#6676)

* Fixes needed for Katib unit tests to run successfully.

Signed-off-by: Elena Zioga <elena@arrikto.com>

Signed-off-by: Elena Zioga <elena@arrikto.com>

* centraldashboard: Show objects from all namespaces (#6674)

* centraldashboard: Introduce onAllNamespacesSelected

Introduce the onAllNamespacesSelected method in centraldashboard's
library. It is going to notify other UIs that a user selects the
"All namespace" option in centraldashboard.

Signed-off-by: Tasos Alexiou <tasos@arrikto.com>

* centraldashboard: Add support for the "All namespace" option

Introduce the "All namespace" option in centraldashboard's namespace
selector. When a user selects this option the centraldashboard is
using its library in order to modify the UIs inside the iframe.

Signed-off-by: Tasos Alexiou <tasos@arrikto.com>

* fixup! centraldashboard: Add support for the "All namespace" option

Signed-off-by: Tasos Alexiou <tasos@arrikto.com>

* Add quotes to notebook_template.yaml namespace field. (#6655)

* build(deps): bump jszip from 3.6.0 to 3.10.1 in /components/crud-web-apps/jupyter/frontend (#6641)

Bumps [jszip](https://github.com/Stuk/jszip) from 3.6.0 to 3.10.1.
- [Release notes](https://github.com/Stuk/jszip/releases)
- [Changelog](https://github.com/Stuk/jszip/blob/main/CHANGES.md)
- [Commits](https://github.com/Stuk/jszip/compare/v3.6.0...v3.10.1)

---
updated-dependencies:
- dependency-name: jszip
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump jose from 2.0.5 to 2.0.6 in /components/crud-web-apps/tensorboards/frontend (#6652)

Bumps [jose](https://github.com/panva/jose) from 2.0.5 to 2.0.6.
- [Release notes](https://github.com/panva/jose/releases)
- [Changelog](https://github.com/panva/jose/blob/v2.0.6/CHANGELOG.md)
- [Commits](https://github.com/panva/jose/compare/v2.0.5...v2.0.6)

---
updated-dependencies:
- dependency-name: jose
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump jose from 2.0.5 to 2.0.6 in /components/crud-web-apps/volumes/frontend (#6653)

Bumps [jose](https://github.com/panva/jose) from 2.0.5 to 2.0.6.
- [Release notes](https://github.com/panva/jose/releases)
- [Changelog](https://github.com/panva/jose/blob/v2.0.6/CHANGELOG.md)
- [Commits](https://github.com/panva/jose/compare/v2.0.5...v2.0.6)

---
updated-dependencies:
- dependency-name: jose
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix(volume name): update formCtrl.get() (#6681)

The Angular code for `<app-form-data-volumes>` was missing `.value`.

* Remove unused python imports. (#6595)

* [vwa-datetime] Showing dates in main page uniformly in VWA (#6683)

* vwa(back): Modify age column to be a timestamp

* Modify the backend to send a timestamp as is without doing any formatting.

Signed-off-by: Elena Zioga <elena@arrikto.com>

* vwa(front): Use DateTimeValue class in age column

* Use DateTimeValue class in TableConfig for age column instead of PropertyValue one.

Signed-off-by: Elena Zioga <elena@arrikto.com>

Signed-off-by: Elena Zioga <elena@arrikto.com>

* Make admission-webhook port number configurable (#6692)

Previously the webhook listened on a fixed port, 4443, which can clash
with other services when the webhook is run on the host network in
Kubernetes, which is required when using some CNI implementations,
notably Calico on EKS [1].

Enable configuration of the webhook listen port via the program flags.

[1] https://projectcalico.docs.tigera.io/getting-started/kubernetes/managed-public-cloud/eks#install-eks-with-calico-networking

Signed-off-by: Steve Larkin <steve.larkin@gmail.com>

Signed-off-by: Steve Larkin <steve.larkin@gmail.com>

* build(deps): bump socket.io-parser and karma in /components/centraldashboard (#6691)

Bumps [socket.io-parser](https://github.com/socketio/socket.io-parser) to 4.2.1 and updates ancestor dependency [karma](https://github.com/karma-runner/karma). These dependencies need to be updated together.


Updates `socket.io-parser` from 3.2.0 to 4.2.1
- [Release notes](https://github.com/socketio/socket.io-parser/releases)
- [Changelog](https://github.com/socketio/socket.io-parser/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io-parser/compare/3.2.0...4.2.1)

Updates `karma` from 4.3.0 to 6.4.1
- [Release notes](https://github.com/karma-runner/karma/releases)
- [Changelog](https://github.com/karma-runner/karma/blob/master/CHANGELOG.md)
- [Commits](https://github.com/karma-runner/karma/compare/v4.3.0...v6.4.1)

---
updated-dependencies:
- dependency-name: socket.io-parser
  dependency-type: indirect
- dependency-name: karma
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [wa-age-column-improvements] Improvements in the web app's age column (#6694)

* web-apps(front): Rename Age header in WAs

* Rename Age header to Created at.

Signed-off-by: Elena Zioga <elena@arrikto.com>

* web-apps(front): Right-align date columns in WAs

* Make date columns to be right-aligned.

Signed-off-by: Elena Zioga <elena@arrikto.com>

Signed-off-by: Elena Zioga <elena@arrikto.com>

* [wa-upgrade-k8s-client] Upgrade @kubernetes/client-node in wa frontends (#6696)

* jwa(front): Upgrade @kubernetes/client-node module

* Upgrade @kubernetes/client-node module, so that to include
  quantityToScalar function directly from kubernetes js module.
* Set allowSyntheticDefaultImports flag to true to allow 'import x
  from y' when a module doesn't have a default export.

Signed-off-by: Elena Zioga <elena@arrik…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants