cilium: fix gops port <10k #14218
Assignees
Labels
kind/bug
This is a bug in the Cilium logic.
Comments
borkmann
added a commit
that referenced
this issue
Dec 9, 2020
Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: #14218 Reported-by: Lee Hu via Slack Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
borkmann
added a commit
that referenced
this issue
Dec 10, 2020
Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: #14218 Reported-by: Lee Hu via Slack Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
joestringer
pushed a commit
that referenced
this issue
Dec 15, 2020
[ upstream commit 7757d31 ] [ Backporter's notes: Resolved conflict in exit code for gops errors ] Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: #14218 Reported-by: Lee Hu via Slack Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Joe Stringer <joe@cilium.io>
joestringer
pushed a commit
that referenced
this issue
Dec 16, 2020
[ upstream commit 7757d31 ] [ Backporter's notes: Resolved conflict in exit code for gops errors ] Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: #14218 Reported-by: Lee Hu via Slack Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Joe Stringer <joe@cilium.io>
aanm
pushed a commit
to tklauser/cilium
that referenced
this issue
Apr 17, 2021
[ upstream commit 7757d31 ] Manually backported from cilium#14329 to address cilium#13400 for v1.8. Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: cilium#14218 Reported-by: Lee Hu via Slack Co-authored-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>
aanm
pushed a commit
that referenced
this issue
Apr 17, 2021
[ upstream commit 7757d31 ] Manually backported from #14329 to address #13400 for v1.8. Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: #14218 Reported-by: Lee Hu via Slack Co-authored-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>
EricMountain
pushed a commit
to DataDog/cilium
that referenced
this issue
Feb 21, 2022
[ upstream commit 7757d31 ] Manually backported from cilium#14329 to address cilium#13400 for v1.8. Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: cilium#14218 Reported-by: Lee Hu via Slack Co-authored-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Set it to specific port below 10k range as it conflicts with nodeport and allow for user override via cmdline.
Reported-by: Lee Hu
299629d
The text was updated successfully, but these errors were encountered: