Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add subcommands for agent and debug events #537

Merged
merged 4 commits into from
May 3, 2021

Conversation

tklauser
Copy link
Member

@tklauser tklauser commented Apr 26, 2021

This PR adds the hubble observe agent-events and hubble observe debug-events sub-commands to retrieve agent and debug events, respectively. These use the new GetAgentEvents/GetDebugEvents API introduced in cilium/cilium#15715.

Flags such as --since, --until, --follow or --time-format are available for the sub-commands as well. Flow-specific flags (e.g. --from-*, --http-* or --ip-translation are restricted to hubble observe and not available in the sub-commands. See hubble observe -h, hubble observe agent-events -h and hubble observe debug-events -h.

Examples:

$ hubble observe agent-events
TIMESTAMP             TYPE                          DETAILS
Apr 26 13:32:03.225   ENDPOINT_CREATED              id: 3478
Apr 26 13:32:14.281   IPCACHE_UPSERTED              cidr: 10.244.2.221/32, identity: 2806, host ip: 172.18.0.4, encrypt key: 0
Apr 26 13:32:14.805   IPCACHE_UPSERTED              cidr: 10.244.0.80/32, identity: 5027, host ip: 172.18.0.3, encrypt key: 0
Apr 26 13:32:14.866   IPCACHE_UPSERTED              cidr: 10.244.0.164/32, identity: 5882, host ip: 172.18.0.3, encrypt key: 0
Apr 26 13:32:14.923   IPCACHE_UPSERTED              cidr: 10.244.0.174/32, identity: 5027, host ip: 172.18.0.3, encrypt key: 0
Apr 26 13:32:17.620   ENDPOINT_REGENERATE_SUCCESS   id: 142, labels: [reserved:host]
Apr 26 13:32:17.624   ENDPOINT_REGENERATE_SUCCESS   id: 142, labels: [reserved:host]
Apr 26 13:32:18.683   ENDPOINT_REGENERATE_SUCCESS   id: 3478, labels: [reserved:health]
Apr 26 13:32:18.686   ENDPOINT_REGENERATE_SUCCESS   id: 3478, labels: [reserved:health]
Apr 26 13:32:24.296   SERVICE_UPSERTED              id: 2, frontend: 10.96.0.10:53, backends: [10.244.0.174:53], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:32:24.297   SERVICE_UPSERTED              id: 3, frontend: 10.96.0.10:9153, backends: [10.244.0.174:9153], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:32:26.475   SERVICE_UPSERTED              id: 4, frontend: 10.96.214.93:80, backends: [10.244.2.221:4245], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: hubble-relay
Apr 26 13:32:27.986   SERVICE_UPSERTED              id: 2, frontend: 10.96.0.10:53, backends: [10.244.0.174:53,10.244.0.80:53], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:32:27.986   SERVICE_UPSERTED              id: 3, frontend: 10.96.0.10:9153, backends: [10.244.0.174:9153,10.244.0.80:9153], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:33:02.147   IPCACHE_UPSERTED              cidr: 172.18.0.2/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:33:02.147   IPCACHE_UPSERTED              cidr: 10.244.1.116/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:33:02.148   IPCACHE_UPSERTED              cidr: 0.0.0.0/0, identity: 2, old identity: 2, encrypt key: 0
Apr 26 13:34:02.149   IPCACHE_UPSERTED              cidr: 172.18.0.2/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:34:02.149   IPCACHE_UPSERTED              cidr: 10.244.1.116/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:34:02.150   IPCACHE_UPSERTED              cidr: 0.0.0.0/0, identity: 2, old identity: 2, encrypt key: 0
$ hubble observe debug-events
TIMESTAMP             FROM         TYPE                     CPU/MARK        MESSAGE
Apr 26 15:58:41.050   ID: 142      DBG_CT_MATCH             01 0x3cbea25    CT entry found lifetime=16811869, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_VERDICT           01 0x3cbea25    CT verdict: Reply, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_1         01 0x3cbea25    Conntrack lookup 1/2: src=172.18.0.3:6443 dst=172.18.0.2:44656
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_2         01 0x3cbea25    Conntrack lookup 2/2: nexthdr=6 flags=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_MATCH             01 0x3cbea25    CT entry found lifetime=16811869, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_VERDICT           01 0x3cbea25    CT verdict: Reply, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_1         01 0x60902282   Conntrack lookup 1/2: src=172.18.0.2:44656 dst=172.18.0.3:6443
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_2         01 0x60902282   Conntrack lookup 2/2: nexthdr=6 flags=1
Apr 26 15:58:41.050   ID: 142      DBG_CT_MATCH             01 0x60902282   CT entry found lifetime=16811869, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_VERDICT           01 0x60902282   CT verdict: Established, revnat=0
Apr 26 15:58:41.266   ID: 142      DBG_IP_ID_MAP_SUCCEED4   01 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.266   ID: 142      DBG_IP_ID_MAP_SUCCEED4   01 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.266   ID: 142      DBG_IP_ID_MAP_SUCCEED4   01 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.267   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.267   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.036   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0xf142bcaf   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   00 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6

See individual commits for details.

@tklauser tklauser added ⌨️ area/cli Impacts the command line interface of any command in the repository. release-note/major This PR introduces major new functionality to Hubble. labels Apr 26, 2021
Copy link
Member

@gandro gandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat! Thanks a lot 🎉

Just one thing that the linter complains about

pkg/time/time.go Outdated Show resolved Hide resolved
cmd/events/events.go Outdated Show resolved Hide resolved
@tklauser tklauser force-pushed the pr/tklauser/agent-debug-events branch from 7091367 to 498bcfb Compare April 26, 2021 16:23
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 26, 2021
cmd/events/debug.go Outdated Show resolved Hide resolved
@tklauser tklauser force-pushed the pr/tklauser/agent-debug-events branch from 498bcfb to c7a6d2c Compare April 27, 2021 15:42
@tklauser tklauser added the dont-merge/blocked PR/issue blocked by an external factor (be that PR, issue, or else) label Apr 29, 2021
@tklauser tklauser force-pushed the pr/tklauser/agent-debug-events branch from c7a6d2c to 68497d4 Compare April 29, 2021 11:20
@michi-covalent
Copy link
Collaborator

here is our current proposal based on an offline discussion we had:

add a sub-sub command for observe sub-command. so:

hubble observe

the existing behavior of getting flows. no change here.

hubble observe flows

same as hubble observe

hubble observe agent-events

get agent-events

hubble observe debug-events

get debug-events

and then share common flags like --follow, --since, --until at theobserve sub-command level, and add event specific flags at sub-sub-command level.

@rolinh rolinh removed the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 30, 2021
@tklauser tklauser force-pushed the pr/tklauser/agent-debug-events branch from 68497d4 to bf87be7 Compare April 30, 2021 21:53
@tklauser
Copy link
Member Author

here is our current proposal based on an offline discussion we had:

add a sub-sub command for observe sub-command. so:

hubble observe

the existing behavior of getting flows. no change here.

hubble observe flows

same as hubble observe

hubble observe agent-events

get agent-events

hubble observe debug-events

get debug-events

and then share common flags like --follow, --since, --until at theobserve sub-command level, and add event specific flags at sub-sub-command level.

Reworked this PR as noted above, except for the hubble observe flows sub-command which isn't there yet. Flows can still only be observed using hubble observe. Otherwise everything should work as expected, including flags shared between hubble observe and hubble observe {agent,debug}-events.

@tklauser tklauser removed the dont-merge/blocked PR/issue blocked by an external factor (be that PR, issue, or else) label Apr 30, 2021
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 30, 2021
@tklauser tklauser requested a review from rolinh April 30, 2021 22:01
@tklauser tklauser requested a review from kaworu April 30, 2021 22:01
@tklauser tklauser requested a review from gandro April 30, 2021 22:01
Copy link
Collaborator

@michi-covalent michi-covalent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 🚀

just out of curiosity how did you get these debug events? i enabled the debug flag and ran

./hubble observe agent-events --server unix:///var/run/cilium/hubble.sock  -f

but i didn't see get any debug events.

@maintainer-s-little-helper maintainer-s-little-helper bot removed the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 1, 2021
completionExample is a constant text not changed at runtime. Declare it
as const instead of var.

Signed-off-by: Tobias Klauser <tobias@cilium.io>
Move timeFormatNameToLayout and the valid format names into package
time. This will allow to reuse them for the agent and debug event
commands introduced in successive commits.

Signed-off-by: Tobias Klauser <tobias@cilium.io>
Add a new `hubble observe agent-events` subcommand used to observe agent
events using the Hubble GetAgentEvents API.

This works similar to the existing `hubble observe` command for flows and will be
extended to support other events, namely debug events.

Example:

```
$ hubble observe agent-events
TIMESTAMP             TYPE                          DETAILS
Apr 26 13:32:03.225   ENDPOINT_CREATED              id: 3478
Apr 26 13:32:14.281   IPCACHE_UPSERTED              cidr: 10.244.2.221/32, identity: 2806, host ip: 172.18.0.4, encrypt key: 0
Apr 26 13:32:14.805   IPCACHE_UPSERTED              cidr: 10.244.0.80/32, identity: 5027, host ip: 172.18.0.3, encrypt key: 0
Apr 26 13:32:14.866   IPCACHE_UPSERTED              cidr: 10.244.0.164/32, identity: 5882, host ip: 172.18.0.3, encrypt key: 0
Apr 26 13:32:14.923   IPCACHE_UPSERTED              cidr: 10.244.0.174/32, identity: 5027, host ip: 172.18.0.3, encrypt key: 0
Apr 26 13:32:17.620   ENDPOINT_REGENERATE_SUCCESS   id: 142, labels: [reserved:host]
Apr 26 13:32:17.624   ENDPOINT_REGENERATE_SUCCESS   id: 142, labels: [reserved:host]
Apr 26 13:32:18.683   ENDPOINT_REGENERATE_SUCCESS   id: 3478, labels: [reserved:health]
Apr 26 13:32:18.686   ENDPOINT_REGENERATE_SUCCESS   id: 3478, labels: [reserved:health]
Apr 26 13:32:24.296   SERVICE_UPSERTED              id: 2, frontend: 10.96.0.10:53, backends: [10.244.0.174:53], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:32:24.297   SERVICE_UPSERTED              id: 3, frontend: 10.96.0.10:9153, backends: [10.244.0.174:9153], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:32:26.475   SERVICE_UPSERTED              id: 4, frontend: 10.96.214.93:80, backends: [10.244.2.221:4245], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: hubble-relay
Apr 26 13:32:27.986   SERVICE_UPSERTED              id: 2, frontend: 10.96.0.10:53, backends: [10.244.0.174:53,10.244.0.80:53], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:32:27.986   SERVICE_UPSERTED              id: 3, frontend: 10.96.0.10:9153, backends: [10.244.0.174:9153,10.244.0.80:9153], type: ClusterIP, traffic policy: Cluster, namespace: kube-system, name: kube-dns
Apr 26 13:33:02.147   IPCACHE_UPSERTED              cidr: 172.18.0.2/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:33:02.147   IPCACHE_UPSERTED              cidr: 10.244.1.116/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:33:02.148   IPCACHE_UPSERTED              cidr: 0.0.0.0/0, identity: 2, old identity: 2, encrypt key: 0
Apr 26 13:34:02.149   IPCACHE_UPSERTED              cidr: 172.18.0.2/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:34:02.149   IPCACHE_UPSERTED              cidr: 10.244.1.116/32, identity: 1, old identity: 1, encrypt key: 0
Apr 26 13:34:02.150   IPCACHE_UPSERTED              cidr: 0.0.0.0/0, identity: 2, old identity: 2, encrypt key: 0
```

Signed-off-by: Tobias Klauser <tobias@cilium.io>
Add a new `hubble observe debug-events` subcommand used to observe debug
events using the Hubble GetDebugEvents API.

This works similar to the `hubble observe` command for flows and the
`hubble observe agent-events` command for agent events.

Example:

```
$ hubble observe debug-events
TIMESTAMP             FROM         TYPE                     CPU/MARK        MESSAGE
Apr 26 15:58:41.050   ID: 142      DBG_CT_MATCH             01 0x3cbea25    CT entry found lifetime=16811869, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_VERDICT           01 0x3cbea25    CT verdict: Reply, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_1         01 0x3cbea25    Conntrack lookup 1/2: src=172.18.0.3:6443 dst=172.18.0.2:44656
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_2         01 0x3cbea25    Conntrack lookup 2/2: nexthdr=6 flags=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_MATCH             01 0x3cbea25    CT entry found lifetime=16811869, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_VERDICT           01 0x3cbea25    CT verdict: Reply, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_1         01 0x60902282   Conntrack lookup 1/2: src=172.18.0.2:44656 dst=172.18.0.3:6443
Apr 26 15:58:41.050   ID: 142      DBG_CT_LOOKUP4_2         01 0x60902282   Conntrack lookup 2/2: nexthdr=6 flags=1
Apr 26 15:58:41.050   ID: 142      DBG_CT_MATCH             01 0x60902282   CT entry found lifetime=16811869, revnat=0
Apr 26 15:58:41.050   ID: 142      DBG_CT_VERDICT           01 0x60902282   CT verdict: Established, revnat=0
Apr 26 15:58:41.266   ID: 142      DBG_IP_ID_MAP_SUCCEED4   01 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.266   ID: 142      DBG_IP_ID_MAP_SUCCEED4   01 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.266   ID: 142      DBG_IP_ID_MAP_SUCCEED4   01 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.267   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:41.267   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.036   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0xf142bcaf   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   04 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6
Apr 26 15:58:42.275   ID: 142      DBG_IP_ID_MAP_SUCCEED4   00 0x9ec04f8c   Successfully mapped addr=172.18.0.3 to identity=6

```

Signed-off-by: Tobias Klauser <tobias@cilium.io>
@tklauser
Copy link
Member Author

tklauser commented May 1, 2021

lgtm rocket

just out of curiosity how did you get these debug events? i enabled the debug flag and ran

./hubble observe agent-events --server unix:///var/run/cilium/hubble.sock  -f

but i didn't see get any debug events.

AFAIK only starting the Cilium agent with debug flag is not enough, I also needed to enable the debug events using cilium config on the node you're running hubble observe debug-events. Here is what I did:

$ kubectl -n kube-system cp ./hubble cilium-p4vwm:/usr/bin/hubble
$ kubectl -n kube-system exec -it cilium-p4vwm -- bash
# cilium config | grep Debug
Debug                    Disabled
DebugLB                  Disabled
DebugPolicy              Enabled
# cilium config Debug=enable
# cilium config | grep Debug
Debug                    Enabled
DebugLB                  Disabled
DebugPolicy              Enabled
# hubble observe debug-events --server unix:///var/run/cilium/hubble.sock
May  1 13:43:46.162: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 07 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.180: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 02 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.182: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 07 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.182: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 07 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.191: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xa8e62bc3 CPU: 06 (Successfully mapped addr=172.18.0.2 to identity=6)
May  1 13:43:46.191: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xa8e62bc3 CPU: 04 (Successfully mapped addr=172.18.0.2 to identity=6)
May  1 13:43:46.191: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xa8e62bc3 CPU: 00 (Successfully mapped addr=172.18.0.2 to identity=6)
May  1 13:43:46.193: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xa8e62bc3 CPU: 02 (Successfully mapped addr=172.18.0.2 to identity=6)
May  1 13:43:46.193: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xa8e62bc3 CPU: 02 (Successfully mapped addr=172.18.0.2 to identity=6)
May  1 13:43:46.302: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 02 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.303: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 01 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.303: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 01 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.340: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 06 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.340: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 06 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.342: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 01 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.342: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 01 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.465: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 06 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.466: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 07 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.466: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 07 (Successfully mapped addr=172.18.0.1 to identity=2)
May  1 13:43:46.644: ID: 2394 DBG_IP_ID_MAP_SUCCEED4 MARK: 0xd37e8a6d CPU: 06 (Successfully mapped addr=172.18.0.1 to identity=2)

@tklauser tklauser force-pushed the pr/tklauser/agent-debug-events branch from bf87be7 to 8dc2df6 Compare May 1, 2021 13:47
@michi-covalent
Copy link
Collaborator

👍 thanks @tklauser i was able to observe debug events as well:

./hubble observe debug-events --server unix:///var/run/cilium/hubble.sock  -f -o jsonpb  | jq
{
  "debug_event": {
    "type": "DBG_IP_ID_MAP_SUCCEED4",
    "source": {
      "ID": 1692,
      "identity": 1,
      "labels": [
        "k8s:node-role.kubernetes.io/master",
        "reserved:host",
        "k8s:minikube.k8s.io/commit=23f40a012abb52eff365ff99a709501a61ac5876",
        "k8s:minikube.k8s.io/name=minikube",
        "k8s:minikube.k8s.io/updated_at=2021_04_30T22_36_30_0700",
        "k8s:minikube.k8s.io/version=v1.15.1"
      ]
    },
    "hash": 4071380026,
    "arg1": 20031680,
    "arg2": 2,
    "arg3": 0,
    "message": "Successfully mapped addr=192.168.49.1 to identity=2",
    "cpu": 2
  },
  "node_name": "minikube",
  "time": "2021-05-02T00:44:32.769110135Z"
}

Copy link
Member

@gandro gandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks a ton for getting this re-implemented on such short notice! 💯 🚀

Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work @tklauser I tested using your instructions and LGTM!

Question: are we planning on adding hubble observe flows as an alias of hubble observe for consistency's sake in another PR, or is it too difficult?

@gandro
Copy link
Member

gandro commented May 3, 2021

Question: are we planning on adding hubble observe flows as an alias of hubble observe for consistency's sake in another PR, or is it too difficult?

Let's try to do that in a follow-up PR if feasible. This one is already big enough as is. I created a tracking issue: #543

Copy link
Member

@rolinh rolinh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @tklauser !

@kaworu I had a quick session with @tklauser about adding hubble observe flows but it turns out it's not that easy to do and probably not something we want to attempt just before branching v0.8 which is why we decided to postpone it.

@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 3, 2021
@rolinh rolinh merged commit 1561ec2 into master May 3, 2021
@rolinh rolinh deleted the pr/tklauser/agent-debug-events branch May 3, 2021 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⌨️ area/cli Impacts the command line interface of any command in the repository. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/major This PR introduces major new functionality to Hubble.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants