Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hubble-cli: Mount /var/run/cilium as a directory #11129

Merged
merged 1 commit into from Apr 26, 2020
Merged

Conversation

michi-covalent
Copy link
Contributor

I've been hitting an issue with mounting /var/run/cilium/hubble.sock as
the Socket type where sometimes /var/run/cilium/hubble.sock gets created
as a directory. To reproduce this behavior on minikube:

  • Deploy Cilium with the Hubble CLI enabled and wait for the hubble-cli
    to become ready. Verify that hubble.sock is a UNIX domain socket.
  • Set enable-hubble to "false" in cilium-config configmap and restart
    Cilium.
  • Now /var/run/cilium/hubble.sock is a directory. It seems like somehow
    hubble-cli creates the directory when Cilium deletes hubble.sock.

Because of this directory Cilium fails to start Hubble server the next
time enable-hubble gets set to true. With this PR /var/run/cilium
directory gets mounted as a directory instead. I manually verified this
fixes the issue.

Signed-off-by: Michi Mutsuzaki michi@isovalent.com

I've been hitting an issue with mounting /var/run/cilium/hubble.sock as
the Socket type where sometimes /var/run/cilium/hubble.sock gets created
as a directory. To reproduce this behavior on minikube:

- Deploy Cilium with the Hubble CLI enabled and wait for the hubble-cli
  to become ready. Verify that hubble.sock is a UNIX domain socket.
- Set `enable-hubble` to "false" in cilium-config configmap and restart
  Cilium.
- Now /var/run/cilium/hubble.sock is a directory. It seems like somehow
  hubble-cli creates the directory when Cilium deletes hubble.sock.

Because of this directory Cilium fails to start Hubble server the next
time `enable-hubble` gets set to `true`. With this PR /var/run/cilium
directory gets mounted as a directory instead. I manually verified this
fixes the issue.

Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
@michi-covalent michi-covalent requested review from gandro and a team April 23, 2020 22:55
@maintainer-s-little-helper
Copy link

Please set the appropriate release note label.

@maintainer-s-little-helper maintainer-s-little-helper bot added this to In progress in 1.8.0 Apr 23, 2020
@michi-covalent michi-covalent added the release-note/misc This PR makes changes that have no direct user impact. label Apr 23, 2020
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.003%) to 44.753% when pulling 4443043 on pr/michi/hubblecli into 0316bc2 on master.

Copy link
Member

@aanm aanm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the insightful commit description

@aanm aanm added the sig/hubble Impacts hubble server or relay label Apr 24, 2020
@aanm
Copy link
Member

aanm commented Apr 24, 2020

test-me-please

Copy link
Member

@gandro gandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very weird, but I might have observed the same! Let's get this in and debug outside of CI

@aanm
Copy link
Member

aanm commented Apr 24, 2020

@gandro try adding the hubble socket path as a volume mount for cilium with type socket to see if it helps.

@gandro
Copy link
Member

gandro commented Apr 24, 2020

test-me-please

@michi-covalent
Copy link
Contributor Author

test-with-kernel

@errordeveloper
Copy link
Contributor

Another good reason for mounting the directory is avoid issues with socket file actually getting deleted/re-created.

@gandro
Copy link
Member

gandro commented Apr 25, 2020

test-gke

@pchaigno pchaigno merged commit 86045f7 into master Apr 26, 2020
1.8.0 automation moved this from In progress to Merged Apr 26, 2020
@pchaigno pchaigno deleted the pr/michi/hubblecli branch April 26, 2020 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay
Projects
No open projects
1.8.0
  
Merged
Development

Successfully merging this pull request may close these issues.

None yet

6 participants