Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local deployment on KIND needs update #318

Closed
BrandonYifanLiu opened this issue Mar 18, 2022 · 8 comments
Closed

Local deployment on KIND needs update #318

BrandonYifanLiu opened this issue Mar 18, 2022 · 8 comments
Assignees
Labels
bug Something isn't working documentation Improvements or additions to documentation question Further information is requested

Comments

@BrandonYifanLiu
Copy link

Describe the bug

When deploying MLX by following instructions here, hitting some errors in below format.

unable to recognize "STDIN": no matches for kind "XXXXX" in version "xxxx/xxxx"

To Reproduce

With following CLI versions.

brandonliu@Brandons-MacBook-Pro ~ % kind --version          
kind version 0.12.0
brandonliu@Brandons-MacBook-Pro ~ % kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-03-06T21:39:59Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/arm64"}
brandonliu@Brandons-MacBook-Pro ~ % kustomize version
{Version:kustomize/v4.5.2 GitCommit:9091919699baf1c5a5bf71b32ca73a993e98088b BuildDate:2022-02-09T23:19:28Z GoOs:darwin GoArch:arm64}

Steps to reproduce the behavior:

  1. Go to this link and follow instructions.
  2. Running command while ! kustomize build mlx-single-kind | \ kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done.
  3. See error.
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "EnvoyFilter" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "VirtualService" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
unable to recognize "STDIN": no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
unable to recognize "STDIN": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

Expected behavior

The command should return without error

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

  • OS: MacOS Monterey V12.3
  • MLX Version [e.g. 22]

Additional context

Add any other context about the problem here.

@BrandonYifanLiu
Copy link
Author

Checked k8s doc, it should relate to API deprecation. Link here https://kubernetes.io/docs/reference/using-api/deprecation-guide/

@ckadner
Copy link
Member

ckadner commented Mar 18, 2022

Hi @BrandonYifanLiu thanks for reporting this. Did the deployment succeed eventually?

@Tomcli I think I have seen (some of) these error/warning messages before but the deployment seemed to succeed eventually

You had mentioned to

# run the below command two times if the CRDs take too long to provision

Is this connected?

@ckadner ckadner added bug Something isn't working documentation Improvements or additions to documentation question Further information is requested labels Mar 18, 2022
@BrandonYifanLiu
Copy link
Author

As a workaround discussed today, I will downgrade the version to make deployment succeed locally and document the process.

@Tomcli
Copy link
Member

Tomcli commented Mar 23, 2022

Hi @BrandonYifanLiu thanks for reporting this. Did the deployment succeed eventually?

@Tomcli I think I have seen (some of) these error/warning messages before but the deployment seemed to succeed eventually

You had mentioned to

# run the below command two times if the CRDs take too long to provision

Is this connected?

Yes, the CRDs take some time to deploy, so we might need to run it twice if we deploy MLX on a new cluster.

@ckadner
Copy link
Member

ckadner commented Mar 28, 2022

As a workaround discussed today, I will downgrade the version to make deployment succeed locally and document the process.

Hi @BrandonYifanLiu -- could you create a PR to update the docs with the Kubernetes versions we need to use and how to do it with KIND?

@ckadner
Copy link
Member

ckadner commented May 18, 2022

@rafvasq -- I think this is the issue you encountered today? Could you verify the Kubernetes version you were using? i.e. Brandon had 1.23.4.

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", > GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-03-06T21:39:59Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/arm64"}

Supported (required) Kubernetes should probably be no newer than 1.21.

Action items:

  1. Update KIND deployment doc:
  • mention Kubernetes high version boundary in Create KIND Cluster paragraph
  • describe and link to KIND docs how to create clusters for older Kubernetes versions
  • update the kind create cluster --name mlx code snippet to specify the K8s version v1.21 (kindest/node:v1.21.12)
    • kind create cluster ... --image kindest/node:v1.21.12
  • add Troubleshooting section with error message and pointer to use K8s version stated in prereqs
  1. Update Kubernetes deployment doc:
  • mention Kubernetes high version boundary in prereqs
  • add Troubleshooting section with error message and pointer to use K8s version stated in prereqs
  1. Update MLX deployment specs for Kubernetes 1.22 and 1.23

@ckadner ckadner assigned rafvasq and unassigned Tomcli May 18, 2022
kiranp2396 pushed a commit to kiranp2396/mlx that referenced this issue May 20, 2022
kiranp2396 pushed a commit to kiranp2396/mlx that referenced this issue May 23, 2022
@rafvasq
Copy link
Contributor

rafvasq commented May 25, 2022

@ckadner, I've tested and continue to run into this issue with kubectl versions 1.20 and 1.21. Continuing to test but my latest attempt used the following:

  • kind version

    • 0.14.0
  • kubectl version

    • Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:27:39Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"darwin/amd64"}
  • kustomize version

    • Version: {KustomizeVersion:3.2.0 GitCommit:a3103f1e62ddb5b696daa3fd359bb6f2e8333b49 BuildDate:2019-09-18T16:26:36Z GoOs:darwin GoArch:amd64}

@ckadner
Copy link
Member

ckadner commented May 25, 2022

@rafvasq -- I just merged the PR from @kiranp2396 so the kind create cluster command should work as expected now.

mlx-bot-app bot pushed a commit that referenced this issue Oct 28, 2022
* Add dev doc links to CONTRIBUTING.md (#312)

Resolves a task from #304

Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>

* Fixes the references to latest kfctl release (#311)

Resolves a task from #304

Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>

* Do not verify links in project dependencies (#320)

Resolves #319

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Add navigation and description table of docs (#314)

* Create readme for docs
* Add description table to readme
* Add links to table
* update relative links
* Change documentation to document

Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>

* Add script to update the docs table  (#317)


Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>

* Modified the way sed version is set (#315)

Instead of checking the operating system or shell emulator,
test which version of `sed` is actually installed in the local
environment.

Resolves #301

Signed-off-by: Krishna Kumar <krishnakumar@ibm.com>

* Bump waitress from 2.0.0 to 2.1.1 in /api/server (#321)

Bumps [waitress](https://github.com/Pylons/waitress) from 2.0.0 to 2.1.1.
- [Release notes](https://github.com/Pylons/waitress/releases)
- [Changelog](https://github.com/Pylons/waitress/blob/master/CHANGES.txt)
- [Commits](Pylons/waitress@v2.0.0...v2.1.1)

---
updated-dependencies:
- dependency-name: waitress
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump jupyter-server from 1.13.4 to 1.15.4 in /api/server (#324)

Bumps [jupyter-server](https://github.com/jupyter/jupyter_server) from 1.13.4 to 1.15.4.
- [Release notes](https://github.com/jupyter/jupyter_server/releases)
- [Changelog](https://github.com/jupyter-server/jupyter_server/blob/main/CHANGELOG.md)
- [Commits](jupyter-server/jupyter_server@v1.13.4...v1.15.4)

---
updated-dependencies:
- dependency-name: jupyter-server
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump minimist from 1.2.5 to 1.2.6 in /dashboard/origin-mlx (#326)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump notebook from 6.4.8 to 6.4.10 in /api/server (#327)

Bumps [notebook](http://jupyter.org) from 6.4.8 to 6.4.10.

---
updated-dependencies:
- dependency-name: notebook
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update API developer docs (#325)

* Add codegen workflow diagram
* Describe codegen workflow
* Describe API server package modules

Closes #325

Signed-off-by: Pavan Pss <psspavan96@gmail.com>

Co-authored-by: Christian Kadner <ckadner@us.ibm.com>

* Update UI developer docs (#323)

Closes #323

Signed-off-by: RRM123 <rithikmamidi@gmail.com>

* Correct description for make update_doc_table (#329)

Signed-off-by: ezinneanne <onwukaezinneanne@gmail.com>

* Update Kubernetes high version in deployment docs (#318) (#332)

* Add Troubleshooting section
* Limit K8s version for KIND cluster using `--image` flag

Resolves #318

Signed-off-by: Kiran-Patel <kiran.patel@ibm.com>
Signed-off-by: Kiran Patel 104844270+kiranp2396@users.noreply.github.com
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

Co-authored-by: Kiran-Patel <kiran.patel@ibm.com>
Co-authored-by: Christian Kadner <ckadner@us.ibm.com>

* fix errors in mlx-ui startup (#338)

* Run mlx-ui as non-root user (#339)

Closes #337

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Update MLX setup instructions for KF 1.5 (#346)

* Update MLX setup instructions for KF 1.5

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* fix selected assets show up in different menus (#342)

Signed-off-by: Jiaxuan-Yang <jiaxuan.yang@ibm.com>

* Consolidate readme (#356)

Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>

Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>
Co-authored-by: Rafael Vasquez <rafael.vasquez@ryerson.com>

* Document MLX Models Workshop (#352)

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

* Add GitHub action to verify doc links (#357)

Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com>

Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com>
Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
Signed-off-by: Krishna Kumar <krishnakumar@ibm.com>
Signed-off-by: RRM123 <rithikmamidi@gmail.com>
Signed-off-by: ezinneanne <onwukaezinneanne@gmail.com>
Signed-off-by: Jiaxuan-Yang <jiaxuan.yang@ibm.com>
Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com>
Co-authored-by: Rafael Vasquez <rafael.vasquez@ryerson.ca>
Co-authored-by: Krishna Kumar Ramachandran <krishnakumar@gatech.edu>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Pavan Pss <psspavan96@gmail.com>
Co-authored-by: RRM123 <40575227+rrm123@users.noreply.github.com>
Co-authored-by: Ezinne Anne Emilia <onwukaezinneanne@gmail.com>
Co-authored-by: Kiran Patel <104844270+kiranp2396@users.noreply.github.com>
Co-authored-by: Kiran-Patel <kiran.patel@ibm.com>
Co-authored-by: jbusche <101845348+jbusche@users.noreply.github.com>
Co-authored-by: Joanna <Jiaxuan.Yang@ibm.com>
Co-authored-by: Rafael Vasquez <rafael.vasquez@ryerson.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Improvements or additions to documentation question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants