New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hubble-cli: Mount /var/run/cilium as a directory #11129
Conversation
I've been hitting an issue with mounting /var/run/cilium/hubble.sock as the Socket type where sometimes /var/run/cilium/hubble.sock gets created as a directory. To reproduce this behavior on minikube: - Deploy Cilium with the Hubble CLI enabled and wait for the hubble-cli to become ready. Verify that hubble.sock is a UNIX domain socket. - Set `enable-hubble` to "false" in cilium-config configmap and restart Cilium. - Now /var/run/cilium/hubble.sock is a directory. It seems like somehow hubble-cli creates the directory when Cilium deletes hubble.sock. Because of this directory Cilium fails to start Hubble server the next time `enable-hubble` gets set to `true`. With this PR /var/run/cilium directory gets mounted as a directory instead. I manually verified this fixes the issue. Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
Please set the appropriate release note label. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the insightful commit description
test-me-please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very weird, but I might have observed the same! Let's get this in and debug outside of CI
@gandro try adding the hubble socket path as a volume mount for cilium with type socket to see if it helps. |
test-me-please |
test-with-kernel |
Another good reason for mounting the directory is avoid issues with socket file actually getting deleted/re-created. |
test-gke |
I've been hitting an issue with mounting /var/run/cilium/hubble.sock as
the Socket type where sometimes /var/run/cilium/hubble.sock gets created
as a directory. To reproduce this behavior on minikube:
to become ready. Verify that hubble.sock is a UNIX domain socket.
enable-hubble
to "false" in cilium-config configmap and restartCilium.
hubble-cli creates the directory when Cilium deletes hubble.sock.
Because of this directory Cilium fails to start Hubble server the next
time
enable-hubble
gets set totrue
. With this PR /var/run/ciliumdirectory gets mounted as a directory instead. I manually verified this
fixes the issue.
Signed-off-by: Michi Mutsuzaki michi@isovalent.com