Add troubleshooting guide for ray version mismatch #154
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 tasks
Jeffwan
reviewed
Feb 21, 2022
|
|
||
| ## Solution | ||
|
|
||
| In above cases, you need to check if the client ray version is exactly the same with the images version in the ray cluster's configuration. |
There was a problem hiding this comment.
We probably won't need 100% match clients.
- Python & Ray minor version matching covers most of the case.
- upstream ray image has different python versions support. Try to match with your client version.
There was a problem hiding this comment.
yes, after I check, the version check only goes to minor version, matching minor version is enough and image and python patch version difference is tolerant. I will make some complement before push a amend commit.
There was a problem hiding this comment.
Yes. It only check minor version level. Great guidance!
Jeffwan
reviewed
Feb 23, 2022
| ``` | ||
| Traceback (most recent call last): | ||
| ... | ||
| RuntimeError: Version mismatch: The cluster was started with: |
There was a problem hiding this comment.
There's another case that even you use same ray version. if the python minor version doesn't match, it would fail as well. I will merge this one as "add trobleshooting for ray version mismatch" and please give it a try when you get a chance.
Jeffwan
changed the title
Jeffwan
changed the title
chenk008
pushed a commit
to chenk008/kuberay
that referenced
this pull request
Mar 8, 2022
Co-authored-by: chenyu.jiang <chenyu.jiang@bytedance.com>
chenk008
pushed a commit
that referenced
this pull request
Mar 22, 2022
Merge branch add_pod_lables of git@gitlab.alipay-inc.com:Arc/kuberay.git into master https://code.alipay.com/Arc/kuberay/pull_requests/4?tab=diff Signed-off-by: 壮酱 <zhengchicheng.zcc@antgroup.com> * add operator config * add vendor * folder name change to kuberay-operator (#143) * Add CRD verb permission in helm (#144) * add crd verb permission in helm * fix ray cluster volume * Update helm-chart/kuberay-operator/README.md Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com> Co-authored-by: wuhua.ck <wuhua.ck@alibaba-inc.com> Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com> * Explanation and Best Practice for workers-head Reconnection (#142) * Add Explaination and Best Practice for workers-head Reconnection * Fixed formatting issue * Minor updates * updated to permlink and remove one empty line * minor fix Co-authored-by: Lin Ma <lin.ma1@bytedance.com> * refactor: rename kubray -> kuberay (#145) * docs: fix typo in README.md * refactor: rename kubray -> kuberay * Add nightly docker build workflow (#141) * no --all-tags for nightly build add nightly docker build workflow add nightly docker build workflow remove kubebuilder update to workflow * change docker build tag to repo revision * add more debug info for goimport issue (#151) * add more debug info for bug-150: goimport issue * update goimport failure message * Support enableIngress for RayCluster (#38) * Support enableIngress for RayCluster Add ingress resources in role Copy more configurations from cluster annotation Add ingress example Update ingress version from v1beta1 to v1 * Update to expose dashboard only * Add troubleshooting guide for ray version mismatch (#154) Co-authored-by: chenyu.jiang <chenyu.jiang@bytedance.com> * Enable gofmt and move goimports to linter job (#158) * Enable gofmt and move related work to linter job 1. Introduce gofmt github action 2. Move goimports from build job to lint job 3. Run gofmt -s -w to simplify test codes and fix all lint issues in apiserver and cli projects * Fix lint errors in apiserver * Fix lint errors in cli * Fix goimports error * Setup ci for apiserver (#162) * first release * add vendor * PullRequest: 2 add aci Merge branch aci of git@gitlab.alipay-inc.com:Arc/kuberay.git into master https://code.alipay.com/Arc/kuberay/pull_requests/2 Signed-off-by: 五花 <wuhua.ck@antgroup.com> * add aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * fix aci * test aci * test * test * fix comment * Add KubeRay release guideline (#161) * add flag watchNamespace (#165) Co-authored-by: chen kang <kongchen28@gmail.com> * [Feature]Add subcommand `--version` (#166) * add subcommand --version * fix * lint Co-authored-by: wuhua.ck <wuhua.ck@alibaba-inc.com> * Leader election need lease permission (#169) Co-authored-by: wuhua.ck <wuhua.ck@alibaba-inc.com> * [CLI] check viper error (#172) * check file exist before write * fix Co-authored-by: wuhua.ck <wuhua.ck@alibaba-inc.com> * fix cli typo (#173) Co-authored-by: wuhua.ck <wuhua.ck@alibaba-inc.com> * add vendor * fix * fix
lowang-bh
pushed a commit
to lowang-bh/kuberay
that referenced
this pull request
Sep 24, 2023
Co-authored-by: chenyu.jiang <chenyu.jiang@bytedance.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
A troubleshooting guide for mismatch python version.
Related issue number
#148
Checks