docs: Document --debug-verbose=datapath in debugging datapath section

[navarrothiago]

Adds --debug-verbose=datapath option for debugging.

Signed-off-by: Thiago Navarro navarro@accuknox.com

[qmonnet]

Thanks for the doc update!
Please find some feedback below.

Note also that you're using Markdown-style single backquotes to mark inline literals, but RST uses double backquotes. If you wanted to preview the changes, you can generate the HTML document locally with make -C Documentation live-preview.

[navarrothiago]

@qmonnet, thanks for your comments. Let me know if you have more suggestions!

[qmonnet]

Thanks! A few comments:

• I think cilium_dbg() should be the preferred method over printk(), because we already have many calls in place in the code, and it integrates well in cilium monitor whereas printk() logs may be harder to track in the datapath flow. Plus printk() is supposed to have lower performance. So I would mention cilium monitor/cilium_dbg() first, and bpftool prog tracelog/printk() last.

• --debug and --debug-verbose=datapath are not equivalent. I suspect (but didn't check) that the later also sets the former. My understanding is that --debug-verbose=datapath also defines the LB_DEBUG macros used in bpf/lib/lb.h to provide additional logs about the load-balancer. But I do not believe these LB_DEBUG macros condition the usage of either cilium_dbg() or printk().

• By the way, cilium config debug=true does not seem to enable DebugLB, like --debug-verbose=datapath does. I think that cilium config debugLB=true is needed (but conversely, this won't set Debug).

Overall it is not clear when to use printk() or cilium_dbg(). I'm not sure there is a clear answer to that, I think cilium_dbg() should be favoured, but the more “traditional” printk() method remains available for developers who prefer to handle their debug logs outside of cilium monitor's framework.

I would also try to make the distinction between functions and options more explicit. I do understand the paragraph because I know what the options or debug functions do, but it would be more accessible, I think, if we stated explicitly that:

1. There are options to print logs from the datapath with cilium monitor: ...
2. There are options to get even more logs, for the load-balancer: ...
3. Depending on which options are enabled, developers can also add their own debug messages. Two functions are available: cilium_dbg() and printk(), they behave differently and their output should be retrieved differently and blah blah blah.

What do you think?
[Asking a review from Joe who may have some valuable input here.]

[joestringer]

My understanding is that --debug-verbose=datapath also defines the LB_DEBUG macros used in bpf/lib/lb.h to provide additional logs about the load-balancer. But I do not believe they condition the usage of either cilium_dbg() or printk().

--debug-verbose=datapath enables the datapath DEBUG option (via option.Config.Opts) in a somewhat sneaky way:

$ git grep -i -B 2 debugdatapath
daemon/cmd/daemon_main.go-
daemon/cmd/daemon_main.go-func initEnv(cmd *cobra.Command) {
daemon/cmd/daemon_main.go:      var debugDatapath bool
--
daemon/cmd/daemon_main.go-              case argDebugVerboseDatapath:
daemon/cmd/daemon_main.go-                      log.Debugf("Enabling datapath debug messages")
daemon/cmd/daemon_main.go:                      debugDatapath = true
--
daemon/cmd/daemon_main.go-      bpf.CheckOrMountFS(option.Config.BPFRoot, k8s.IsEnabled())
daemon/cmd/daemon_main.go-
daemon/cmd/daemon_main.go:      option.Config.Opts.SetBool(option.Debug, debugDatapath)
daemon/cmd/daemon_main.go:      option.Config.Opts.SetBool(option.DebugLB, debugDatapath)

(Plus later the option.Config.Opts is serialized into individual endpoint headers to enable debug for the endpoint).

[qmonnet]

Thanks, nice rework! It looks more correct and makes the different options much clearer, I find it much better than the previous versions of the PR.

Now for the small details :). A few observations below, mostly I think the paragraph starts a bit abruptly for users who come here just looking for a way to debug their issue, I think we should put the commands first and the explanations after that. See suggestions below.

[qmonnet]

Ah, one other thing, we are trying to move away from running commands inside of Cilium pods, but to use the cilium CLI instead. I haven't checked if it can be used to enable debug messages (and the debugLB?), would you happen to know? It would be nice if we could use this generic CLI instead of the old one which is just available on the agent pods.

[navarrothiago]

Ah, one other thing, we are trying to move away from running commands inside of Cilium pods, but to use the cilium CLI instead. I haven't checked if it can be used to enable debug messages (and the debugLB?), would you happen to know? It would be nice if we could use this generic CLI instead of the old one which is just available on the agent pods.

Oh! I did not know that. I have never tried to used the cilium CLI. Because of that, could we follow this PR focusing on the idea that we are doing (without CLI)? I believe it will took me a little bit more to understand about it. If you have other suggestion (corrections), please, let me know! And, thank you again :)

[qmonnet]

OK let's keep the cilium-cli for another PR.
I find the current version really neat, thanks a lot for this work!

[brb]

Nitpick: maybe we can unify style of diagrams in the docs? E.g. https://docs.cilium.io/en/v1.10/gettingstarted/cni-chaining-aws-cni/

[errordeveloper]

As someone who doesn't know much about this, I find this additional information very helpful! Thanks very much!