-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add subcommands for agent and debug events #537
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Neat! Thanks a lot 🎉
Just one thing that the linter complains about
7091367
to
498bcfb
Compare
498bcfb
to
c7a6d2c
Compare
c7a6d2c
to
68497d4
Compare
here is our current proposal based on an offline discussion we had: add a sub-sub command for hubble observethe existing behavior of getting flows. no change here. hubble observe flowssame as hubble observe hubble observe agent-eventsget agent-events hubble observe debug-eventsget debug-events and then share common flags like |
68497d4
to
bf87be7
Compare
Reworked this PR as noted above, except for the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm 🚀
just out of curiosity how did you get these debug events? i enabled the debug flag and ran
./hubble observe agent-events --server unix:///var/run/cilium/hubble.sock -f
but i didn't see get any debug events.
completionExample is a constant text not changed at runtime. Declare it as const instead of var. Signed-off-by: Tobias Klauser <tobias@cilium.io>
Move timeFormatNameToLayout and the valid format names into package time. This will allow to reuse them for the agent and debug event commands introduced in successive commits. Signed-off-by: Tobias Klauser <tobias@cilium.io>
Add a new `hubble observe agent-events` subcommand used to observe agent events using the Hubble GetAgentEvents API. This works similar to the existing `hubble observe` command for flows and will be extended to support other events, namely debug events. Example: ``` $ hubble observe agent-events TIMESTAMP TYPE DETAILS Apr 26 13:32:03.225 ENDPOINT_CREATED id: 3478 Apr 26 13:32:14.281 IPCACHE_UPSERTED cidr: 10.244.2.221/32, identity: 2806, host ip: 172.18.0.4, encrypt key: 0 Apr 26 13:32:14.805 IPCACHE_UPSERTED cidr: 10.244.0.80/32, identity: 5027, host ip: 172.18.0.3, encrypt key: 0 Apr 26 13:32:14.866 IPCACHE_UPSERTED cidr: 10.244.0.164/32, identity: 5882, host ip: 172.18.0.3, encrypt key: 0 Apr 26 13:32:14.923 IPCACHE_UPSERTED cidr: 10.244.0.174/32, identity: 5027, host ip: 172.18.0.3, encrypt key: 0 Apr 26 13:32:17.620 ENDPOINT_REGENERATE_SUCCESS id: 142, labels: [reserved:host] Apr 26 13:32:17.624 ENDPOINT_REGENERATE_SUCCESS id: 142, labels: [reserved:host] Apr 26 13:32:18.683 ENDPOINT_REGENERATE_SUCCESS id: 3478, labels: [reserved:health] Apr 26 13:32:18.686 ENDPOINT_REGENERATE_SUCCESS id: 3478, labels: [reserved:health] Apr 26 13:32:24.296 SERVICE_UPSERTED id: 2, frontend: 10.96.0.10:53, backends: [10.244.0.174:53], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns Apr 26 13:32:24.297 SERVICE_UPSERTED id: 3, frontend: 10.96.0.10:9153, backends: [10.244.0.174:9153], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns Apr 26 13:32:26.475 SERVICE_UPSERTED id: 4, frontend: 10.96.214.93:80, backends: [10.244.2.221:4245], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: hubble-relay Apr 26 13:32:27.986 SERVICE_UPSERTED id: 2, frontend: 10.96.0.10:53, backends: [10.244.0.174:53,10.244.0.80:53], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns Apr 26 13:32:27.986 SERVICE_UPSERTED id: 3, frontend: 10.96.0.10:9153, backends: [10.244.0.174:9153,10.244.0.80:9153], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns Apr 26 13:33:02.147 IPCACHE_UPSERTED cidr: 172.18.0.2/32, identity: 1, old identity: 1, encrypt key: 0 Apr 26 13:33:02.147 IPCACHE_UPSERTED cidr: 10.244.1.116/32, identity: 1, old identity: 1, encrypt key: 0 Apr 26 13:33:02.148 IPCACHE_UPSERTED cidr: 0.0.0.0/0, identity: 2, old identity: 2, encrypt key: 0 Apr 26 13:34:02.149 IPCACHE_UPSERTED cidr: 172.18.0.2/32, identity: 1, old identity: 1, encrypt key: 0 Apr 26 13:34:02.149 IPCACHE_UPSERTED cidr: 10.244.1.116/32, identity: 1, old identity: 1, encrypt key: 0 Apr 26 13:34:02.150 IPCACHE_UPSERTED cidr: 0.0.0.0/0, identity: 2, old identity: 2, encrypt key: 0 ``` Signed-off-by: Tobias Klauser <tobias@cilium.io>
Add a new `hubble observe debug-events` subcommand used to observe debug events using the Hubble GetDebugEvents API. This works similar to the `hubble observe` command for flows and the `hubble observe agent-events` command for agent events. Example: ``` $ hubble observe debug-events TIMESTAMP FROM TYPE CPU/MARK MESSAGE Apr 26 15:58:41.050 ID: 142 DBG_CT_MATCH 01 0x3cbea25 CT entry found lifetime=16811869, revnat=0 Apr 26 15:58:41.050 ID: 142 DBG_CT_VERDICT 01 0x3cbea25 CT verdict: Reply, revnat=0 Apr 26 15:58:41.050 ID: 142 DBG_CT_LOOKUP4_1 01 0x3cbea25 Conntrack lookup 1/2: src=172.18.0.3:6443 dst=172.18.0.2:44656 Apr 26 15:58:41.050 ID: 142 DBG_CT_LOOKUP4_2 01 0x3cbea25 Conntrack lookup 2/2: nexthdr=6 flags=0 Apr 26 15:58:41.050 ID: 142 DBG_CT_MATCH 01 0x3cbea25 CT entry found lifetime=16811869, revnat=0 Apr 26 15:58:41.050 ID: 142 DBG_CT_VERDICT 01 0x3cbea25 CT verdict: Reply, revnat=0 Apr 26 15:58:41.050 ID: 142 DBG_CT_LOOKUP4_1 01 0x60902282 Conntrack lookup 1/2: src=172.18.0.2:44656 dst=172.18.0.3:6443 Apr 26 15:58:41.050 ID: 142 DBG_CT_LOOKUP4_2 01 0x60902282 Conntrack lookup 2/2: nexthdr=6 flags=1 Apr 26 15:58:41.050 ID: 142 DBG_CT_MATCH 01 0x60902282 CT entry found lifetime=16811869, revnat=0 Apr 26 15:58:41.050 ID: 142 DBG_CT_VERDICT 01 0x60902282 CT verdict: Established, revnat=0 Apr 26 15:58:41.266 ID: 142 DBG_IP_ID_MAP_SUCCEED4 01 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:41.266 ID: 142 DBG_IP_ID_MAP_SUCCEED4 01 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:41.266 ID: 142 DBG_IP_ID_MAP_SUCCEED4 01 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:41.267 ID: 142 DBG_IP_ID_MAP_SUCCEED4 04 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:41.267 ID: 142 DBG_IP_ID_MAP_SUCCEED4 04 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:42.036 ID: 142 DBG_IP_ID_MAP_SUCCEED4 04 0xf142bcaf Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:42.275 ID: 142 DBG_IP_ID_MAP_SUCCEED4 04 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:42.275 ID: 142 DBG_IP_ID_MAP_SUCCEED4 04 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:42.275 ID: 142 DBG_IP_ID_MAP_SUCCEED4 04 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 Apr 26 15:58:42.275 ID: 142 DBG_IP_ID_MAP_SUCCEED4 00 0x9ec04f8c Successfully mapped addr=172.18.0.3 to identity=6 ``` Signed-off-by: Tobias Klauser <tobias@cilium.io>
AFAIK only starting the Cilium agent with debug flag is not enough, I also needed to enable the debug events using
|
bf87be7
to
8dc2df6
Compare
👍 thanks @tklauser i was able to observe debug events as well:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me! Thanks a ton for getting this re-implemented on such short notice! 💯 🚀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Amazing work @tklauser I tested using your instructions and LGTM!
Question: are we planning on adding hubble observe flows
as an alias of hubble observe
for consistency's sake in another PR, or is it too difficult?
Let's try to do that in a follow-up PR if feasible. This one is already big enough as is. I created a tracking issue: #543 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR adds the
hubble observe agent-events
andhubble observe debug-events
sub-commands to retrieve agent and debug events, respectively. These use the newGetAgentEvents
/GetDebugEvents
API introduced in cilium/cilium#15715.Flags such as
--since
,--until
,--follow
or--time-format
are available for the sub-commands as well. Flow-specific flags (e.g.--from-*
,--http-*
or--ip-translation
are restricted tohubble observe
and not available in the sub-commands. Seehubble observe -h
,hubble observe agent-events -h
andhubble observe debug-events -h
.Examples:
See individual commits for details.