-
Notifications
You must be signed in to change notification settings - Fork 168
hyperctl run fails consistently with certain images on random k8s nodes #1081
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
What we know so far
|
Update: We are seeing the following log everytime we
In comparison, the following log appears with a working
|
Update: It seems the parent image, However the grandparent, (I'm assuming this is the right Dockerfile for
|
Tried to rebuild an image on top of the working grandparent Will try to binary search and figure out exactly which line was causing this issue. |
In the meantime, checked disk space with
|
Seems like this line is what's causing the
Will keep digging exactly why this line is causing fs corruption UPDATE: It seems this issue has nothing to do with the base image or the path of the volume. Even an image created with Dockerfile like below will have the same error, as long as the host is in the bad state.
|
Update: Seems like the issue is with the default volume location hyperd uses on the host OS:
Would be nice to know where does hyperd mount The disk space usage does look healthy on the host OS though:
|
Being a bug in hyperd, this is not something SD can fix. |
What happened:
On some of the k8s nodes, we are observing
hyperctl run
failing consistently with the following error message when running with a docker image our user built on top of thedocker.io/gradle:jdk8
image:This is causing the workers fail to launch builds using this image.
What you expected to happen:
hyperctl should be able to launch hyperd container with any normal Docker images
How to reproduce it:
sudo hyperctl run -t <IMAGE> date
The text was updated successfully, but these errors were encountered: