Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"No healthy upstream" when using PVC #11

Open
grobbie opened this issue Feb 17, 2022 · 2 comments
Open

"No healthy upstream" when using PVC #11

grobbie opened this issue Feb 17, 2022 · 2 comments
Labels
bug Something isn't working Kubeflow 1.8 This issue affects the Charmed Kubeflow 1.8 release
Projects
MLOps Solution Issues

Comments

@grobbie
Copy link

grobbie commented Feb 17, 2022

When creating a new tensorboard with a PVC mounted at mnt/tfboard, I get the error "no healthy upstream" when trying to connect to the board.

If I shell into the container, I can see that the tensorBoard is running perfectly fine:

microk8s.kubectl exec -it $(microk8s.kubectl get pods -nrob | grep ^demo- | awk '{ print $1 }' | tail -n 1) -nrob -- bash
root@demo-7498d9d94-zk5gw:/# curl -s localhost:6006/ | head -n 17
<!doctype html><!--
@license
Copyright 2016 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--><meta charset="utf-8">
<title>TensorBoard</title>

image

image
@DomFleischmann DomFleischmann added this to Needs Triage in MLOps Solution Issues Feb 17, 2022
@ca-scribner
Copy link
Contributor

Did this error persist? I have a feeling if you refreshed the page a few moments later it would work.

If I am right, the reason for this is the same as this issue where that the notebook controller follows whether a container on the pod ready, not whether all required containers are ready. This is a common problem with notebooks, where the connect button will go green slightly before the notebook server is actually presenting the page. If you're quick, you'll get this error.

I don't think this is a heavy lift to fix, but it would be a fix upstream

@ca-scribner ca-scribner moved this from Needs Triage to Labeled in MLOps Solution Issues Feb 18, 2022
@ca-scribner ca-scribner added the bug Something isn't working label Feb 18, 2022
@grobbie
Copy link
Author

grobbie commented Feb 18, 2022

the problem has persisted, sorry...

@DnPlas DnPlas added the Kubeflow 1.8 This issue affects the Charmed Kubeflow 1.8 release label May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Kubeflow 1.8 This issue affects the Charmed Kubeflow 1.8 release
Projects
MLOps Solution Issues
Labeled
Milestone
No milestone
Development

No branches or pull requests
3 participants
@grobbie @DnPlas @ca-scribner