Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SeldonDeployment keeps hanging #499

Closed
eramol opened this issue Apr 9, 2019 · 1 comment · Fixed by #500
Closed

SeldonDeployment keeps hanging #499

eramol opened this issue Apr 9, 2019 · 1 comment · Fixed by #500
Assignees
@ukclivecox
Projects
0.2.7
Milestone
0.2.x

Comments

@eramol
Copy link

eramol commented Apr 9, 2019 •
edited

There is a problem with the new 2.7-snapshot version of cluster manager - when a seldondeployment is deployed, it correctly creates a deployment, a replica set and the pods. However then it fails, because it fails to get the namespacedHorizontalPodAutoscalers, it correctly logs the error (KubeCRDHandlerImpl->getOwnedHPAs:324) but then it returns null, which causes a silent NullPointerException in SeldonDeletionHandler:144 which kills the whole task without updating the state of the seldon deployment (whatever it is).

A solution is to catch the nullpointer (heavy handed) or to return an optional instead (imho better)

The symptom is that the seldondeployment forever hangs as Creating
@ukclivecox ukclivecox mentioned this issue Apr 9, 2019
Merged
@ahousley
Copy link
Member

ahousley commented Apr 9, 2019

Thanks @eramol for sharing this fix 👍

@ukclivecox ukclivecox self-assigned this Apr 10, 2019
@ukclivecox ukclivecox added this to the 0.2.x milestone Apr 10, 2019
@ukclivecox ukclivecox added this to In progress in 0.2.7 Apr 10, 2019
@ukclivecox ukclivecox moved this from In progress to Done in 0.2.7 Apr 10, 2019
agrski pushed a commit that referenced this issue Dec 2, 2022
@ukclivecox
Notebook doc updates (#499)
e12f681 
* initial docs update for local and pipeline examples

* more docs updates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ukclivecox ukclivecox

Labels
None yet
Projects
No open projects
0.2.7
  
Done
Milestone
0.2.x
Development

Successfully merging a pull request may close this issue.

Fix HPA Nullpointer
3 participants
@ahousley @eramol @ukclivecox