Linux: OOM when a kernel debug symbols is used as input

[gcmoreira]

Dwarf2json out of memory when parsing Linux kernel debug symbols.
The QEMU VM used for this have Ubuntu 18.04.4 amd64 stock kernels 4.18.0-25-generic and I also tried with 5.4.0-42-generic.
The command lines were:
./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-42-generic
or
./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-42-generic --system-map System.map-5.4.0-42-generic

The dmesg error was:

[  148.628383] Out of memory: Killed process 1322 (dwarf2json) total-vm:5022196kB, anon-rss:3736096kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8012kB oom_score_adj:0

I tried from inside of the virtual machine first with 2GB of RAM which later increased to 4GB and 8GB but it didn't fix the problem.
Finally, I got dwarf2json working in a 16GB RAM host.
I'm aware Dwarf itself requires a lot of memory but I think it would be great to see if this limitation could be addressed in the future.

[ilch1]

Thanks for reporting the issue. I haven't seen an OOM error before, but my machine has 16 GB of RAM.

I would need to run a profiler to determine what is using the memory. While some optimizations may be possible, the output format is such that the entire symbols structure resides in memory before being written out as JSON. My guess is that is taking up a large chunk of total memory by dwarf2json.

At the very least, we can update documentation with recommended RAM.

[NiklasBeierl]

Experiencing an OOM as well in a 4gig VM, had to run it on the host.

[ilch1]

Documentation added in PR #29.