[Feat]: build an additional Netdata nightly image with debug symbols in CI

[ilyam8]

Problem

We have to build a custom Netdata image manually every time we need to debug a segfault when Netdata is running inside a Docker container.

It would be very nice to build a Netdata image with CFLAGS="-O1 -ggdb" in our CI.

Description

A nightly Netdata image build with CFLAGS="-O1 -ggdb".

Importance

really want

Value proposition

We need it to debug segfaults when Netdata is running inside a Docker container.

Proposed implementation

No response

[Ferroin]

Could I get some better clarification about exactly what is desired here that is not working (IOW exactly what debugging functionality is missing that is needed)? -ggdb is an exceedingly heavy option, both at build time and at runtime, and while it is possible to eliminate the runtime overhead it is somewhat complicated to get it right.

If all we care about is working symbolic backtraces and those are not working, then something is broken in the build (binaries are getting stripped when they should not be) and we do not need -ggdb.

If we actually need line number resolution or similar functionality, then we probably do need a -g option, but may not need -ggdb specifically. Regardless of the exact option, we will need to set up detached debugging symbols so that we don’t add a ridiculous additional runtime memory overhead to the binaries, which is not exactly a trivial change to the build process.

Irrespective of that, if we are running into problems with debugging that require us to build with -O1 to resolve, then I would seriously question the quality of our code in the first place. Unless we’re doing stuff resulting in super aggressive inlining of code, we should not be seeing a significant difference

[ilyam8]

@Ferroin I didn't mean building Netdata nightly with CFLAGS="-O1 -ggdb", but an additional nightly image (e.g. netdata/debug:latest)

[Ferroin]

I reiterate though, what specifically is needed here? AIUI, -ggdb is only needed to get things like line number references in function names, hence my question of what debugging functionality is not working here.

On a side note, you probably want -Og, not -O1 for this type of thing.

[Ferroin]

OK, based on discussion in the agent sync meeting I understand better what’s actually needed here.

Current plan is:

• Add a new build argument (DEBUG_BUILD) to the Dockerfile to switch the default CFLAGS from -O2 -pipe to -Og -ggdb -pipe. This way we have a cleanly reproducible to easily test builds locally.
• Add a new docker image named netdata/netdata-dbg, which will mirror netdata/netdata, but use the above-mentioned build argument during the build process. This will give us usable nightlies and stable builds with debug symbols.
• Investigate long-term how we can improve things for native packages and static installs (I think our Debian packages are already set up to generate split debugging symbols properly, so we should just need to add -ggdb there, but our RPM packages and the static builds will be trickier).

[ilyam8]

Thanks, one little thing from me - netdata-debug looks better than netdata-dbg.