make: strip symbol tables from all binaries by default

[tklauser]

Marked RFC because I'm not sure this is something we want or should do by default. While it doesn't break panic backtraces it makes debugging of production systems harder or even impossible.

Feedback appreciated @tgraf @aanm @joestringer @raybejjani

---

Strip the symbol tables and debug information from all binaries by
default. This will not remove annotations needed for stack traces, so
panic backtraces will still be readable.

If debug information and symbol tables are needed during debugging, use
make NOSTRIP=1 to compile.

This reduces the binary sizes as follows:

before:

-rwxr-xr-x 1 tklauser tklauser 45M Feb 27 12:35 cilium/cilium
-rwxr-xr-x 1 tklauser tklauser 20M Feb 27 12:35 cilium-health/cilium-health
-rwxr-xr-x 1 tklauser tklauser 70M Feb 27 12:35 daemon/cilium-agent
-rwxr-xr-x 1 tklauser tklauser 74M Feb 27 12:35 operator/cilium-operator
-rwxr-xr-x 1 tklauser tklauser 20M Feb 27 12:35 plugins/cilium-cni/cilium-cni

after:

-rwxr-xr-x 1 tklauser tklauser 45M Feb 27 12:37 cilium/cilium
-rwxr-xr-x 1 tklauser tklauser 15M Feb 27 12:37 cilium-health/cilium-health
-rwxr-xr-x 1 tklauser tklauser 51M Feb 27 12:37 daemon/cilium-agent
-rwxr-xr-x 1 tklauser tklauser 54M Feb 27 12:37 operator/cilium-operator
-rwxr-xr-x 1 tklauser tklauser 15M Feb 27 12:37 plugins/cilium-cni/cilium-cni

summary:

cilium           0MB (already stripped previously, see 8596bd9c)
cilium-agent    ~19MB
cilium-health   ~5MB
cilium-operator ~20MB
cilium-cni      ~5MB

Also see https://blog.filippo.io/shrink-your-go-binaries-with-this-one-weird-trick/

Updates #6099
Updates #10056

Signed-off-by: Tobias Klauser tklauser@distanz.ch

---

This change is 

[maintainer-s-little-helper]

Release note label not set, please set the appropriate release note.

[coveralls]

Coverage increased (+0.02%) to 45.549% when pulling 7f89309 on pr/tklauser/strip-symbols-tables-by-default into b676c4c on master.

[tklauser]

test-me-please

[tgraf]

Does this have an impact on the memory consumption of cilium-agent or only on binary size?

[tklauser]

Does this have an impact on the memory consumption of cilium-agent or only on binary size?

AFAICS it only affects binary size (i.e. for #6099). Memory consumption is more or less the same.

[joestringer]

it makes debugging of production systems harder or even impossible

panic backtraces will still be readable.

Do you have example production debug workflows that you're concerned we will be unable to follow if we merge this PR? I think that these would help us to make a decision.

Most debugging workflows I've used involve the cilium-sysdump and the information you gather there; panic backtraces would be an exception, but as you note these should continue to work.

For instance, does this break gops?

[tklauser]

Do you have example production debug workflows that you're concerned we will be unable to follow if we merge this PR? I think that these would help us to make a decision.

I don't have a lot of experience of debugging Cilium in production yet, so I cannot really name an example where debug symbols would be useful. AFAICT panic backtraces, gops and cilium-sysdump would probably be enough for me to get a first idea.

I just wanted to raise this in order to make others aware that e.g. debugging using gdb or delve would no longer work on production binaries.

For instance, does this break gops?

gops will still work AFAICS, e.g. the stackcommand, the top10 and even the list <symbol> command in pprof-heap still resolve the function names.

[aanm]

Do you have example production debug workflows that you're concerned we will be unable to follow if we merge this PR? I think that these would help us to make a decision.

I don't have a lot of experience of debugging Cilium in production yet, so I cannot really name an example where debug symbols would be useful. AFAICT panic backtraces, gops and cilium-sysdump would probably be enough for me to get a first idea.

I just wanted to raise this in order to make others aware that e.g. debugging using gdb or delve would no longer work on production binaries.

Why? Wouldn't it work even if ship the unstripped binary and have it available to be used whenever we need to use delve and / or gdb? AFAIK, delve allows to pass a binary for a running process.

For instance, does this break gops?

gops will still work AFAICS, e.g. the stackcommand, the top10 and even the list <symbol> command in pprof-heap still resolve the function names.

[tklauser]

test-me-please

[tklauser]

Why? Wouldn't it work even if ship the unstripped binary and have it available to be used whenever we need to use delve and / or gdb? AFAIK, delve allows to pass a binary for a running process.

True. And I think the same is possible in gdb with the symbol-file command.

[aanm]

One more note, shouldn't we have 2 docker images (one for the non-stripped and the other with the stripped) created from the same binary? Similar to what it is being proposed here cilium/proxy#16 (comment)

[joestringer]

@tklauser are you waiting on us to merge this or are you following up on some feedback?

[tklauser]

@joestringer sorry, this PR somehow fell between the cracks on my side. As per @aanm's feedback I tried to build 2 docker images for stripped and unstripped binaries but this will need a bit more work (also to provide both images on quay.io). Thus, I'd propose to merge this as is (like the linked cilium/proxy#16 PR) and provide an image with unstripped binaries once the need arises.

Would that be OK with you both?

[tgraf]

One more note, shouldn't we have 2 docker images (one for the non-stripped and the other with the stripped) created from the same binary? Similar to what it is being proposed here cilium/proxy#16 (comment)

Why do we need unstripped container images? Are you expecting users to run them? If so, why?

[tgraf]

Merging this. We can resolve the unstripped binary images separately.

[aanm]

One more note, shouldn't we have 2 docker images (one for the non-stripped and the other with the stripped) created from the same binary? Similar to what it is being proposed here cilium/proxy#16 (comment)

Why do we need unstripped container images? Are you expecting users to run them? If so, why?

@tgraf we need an unstripped binary in case we want to debug a live cilium process. If that live process is running the stripped binary we should have the unstripped binary also available.

[tgraf]

@tgraf we need an unstripped binary in case we want to debug a live cilium process. If that live process is running the stripped binary we should have the unstripped binary also available.

What exact operation is to debug? Are you talking about running a debugger? Are you talking about gops debugging? How commonly do we need this information available right now?

I assume users will typically run the unstripped version and we would require them to run a different image and reproduce? Is a Makefile target to build unstripped images on demand sufficient?