Support DWARF5 standardization of GNU extension expression opcodes

[rdunklau]

In dwarfv5, a number of GNU extensions have been standardized.
I don't know how to test for them to be honest, but it would be nice to add synonyms to the GNU extension opcodes we already parse.

It seems to me that the following should be added:

DW_OP_GNU_entry_value => DW_OP_entry_value
DW_OP_GNU_const_type => DW_OP_const_type
DW_OP_GNU_regval_type => DW_OP_regval_type
DW_OP_GNU_deref_type => DW_OP_deref_type
DW_OP_GNU_implicit_pointer => DW_OP_implicit_pointer
DW_OP_GNU_convert => DW_OP_convert

[eliben]

PRs welcome

@sevaa

[sevaa]

Those are listed in the enum, not recognized by the expression parser and not dumped by the description generator. I could provide the latter, but where would we get a binary to test? That'd be more of a challenge.

I do have a sample that produces the older GNU counterparts to them (not all those opcodes, just some), both built and the source. Let me try to build the same source with DWARF5, see where that takes me.

[rdunklau]

Thanks ! I did not provide a PR since it's trivial to add, and the problem is actually having a test case for it as you noted. I guess worst case scenario would be to manually replace opcodes in the binary with the GNU_ versions and run through it ?

[sevaa]

Rebuilt the same code for Linux/x64 with DWARF5, got a test binary. All the listed opcodes are there. Stay tuned for a PR.

[eliben]

@sevaa would you like to fix this before 0.29 is released? Or was it already fixed with #411 ?

[sevaa]

I'd like to. Barring more surprises from GNU, this shouldn't take long.

[sevaa]

Update: looks like the new in DWARF5 section .debug_loclists, which replaces .debug_loc, was never implemented. Oh well.

[rdunklau]

I have a rudimentary implementation outside of pyelftools for this, let me make a proper patch for it.

[sevaa]

I'm already halfway :)

[rdunklau]

Great :-) Please ping me for the review if you want.

[vegard]

Hi, I ran into issues trying to use the dwarf_lineprogram_filenames.py example on files from these archives:

• http://ftp.de.debian.org/debian/pool/main/l/linux/linux-image-5.17.0-3-amd64-dbg_5.17.11-1_amd64.deb
• http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-unsigned-5.15.0-37-generic-dbgsym_5.15.0-37.39_amd64.ddeb
  I'm pretty sure the errors are related to DWARF5 as I'm getting various errors (including what looks like opcode-related ones):

On /usr/lib/debug/lib/modules/5.17.0-3-amd64/kernel/drivers/edac/edac_mce_amd.ko from the 5.17.11 deb above:

  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/dwarfinfo.py", line 241, in line_program_for_CU
    top_DIE = CU.get_top_DIE()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/compileunit.py", line 92, in get_top_DIE
    top = DIE(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/die.py", line 94, in __init__
    self._parse_DIE()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/die.py", line 242, in _parse_DIE
    raw_value = struct_parse(structs.Dwarf_dw_form[form], self.stream)
KeyError: 'DW_FORM_line_strp'

and /usr/lib/debug/lib/modules/5.17.0-3-amd64/kernel/net/atm/clip.ko:

  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/dwarfinfo.py", line 241, in line_program_for_CU
    top_DIE = CU.get_top_DIE()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/compileunit.py", line 92, in get_top_DIE
    top = DIE(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/die.py", line 94, in __init__
    self._parse_DIE()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/die.py", line 234, in _parse_DIE
    abbrev_decl = self.cu.get_abbrev_table().get_abbrev(self.abbrev_code)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/elftools/dwarf/abbrevtable.py", line 36, in get_abbrev
    return self._abbrev_map[code]
KeyError: 172

You can use dpkg-deb -x file.[d]deb /tmp/output-dir/ to extract the archives on a dpkg-based systems (Debian/Ubuntu).
Just wanted to add this as it might be a good test.

Edit: attached a zip of the files in question: dwarf5.zip

[sevaa]

@rdunklau: got any test binaries? Preferably one with the whole gamut of DW_LLE_ entry types?

@vegard: we just had a PR with support for the V5 lineprogram logic, merged yesterday. Can you pull the latest master and test with that? Also, normally one creates a new issue, unless you are reasonably sure your error has the same nature as the one in this issue. From looking at the error message, it's not.

[rdunklau]

@sevaa no test binary unfortunately. I started implementing the basics because one of the binaries I'm working with had it (compiled with gcc 12.1).
Not sure what triggers it to generate a location list though...

[sevaa]

.debug_loclists is mandated in place of .debug_loc by DWARF v5, so any build with -gdwarf-5 will do. Still, my GCC (8.3) generates the loclists as if it was V4 loc, with none of that second-level indirection business. Also, -O2 usually brings out the hairiest DWARF; unoptimized builds are straightforward.

[sevaa]

.debug_loclists works, but there is another section that was replaced in DWARF5: .debug_rnglists. Most disturbingly, the fact that it was ignored went undetected by the tests.

[eliben]

It seems like we'll want to wait for these fixes before releasing 0.29

[sevaa]

With two new sections to look at, this with break the DWARFInfo constructor. Consumers that use elftools as a DWARF parser for non-ELF files, myself included, will have to accommodate. No way around it. Won't be the first time, though.

[eliben]

That's OK. pyelftools is fairly low-level infra and not aiming for a stable API. I wouldn't break things just for fun, but if there's a good reason to do so, that's not a big problem.

[sevaa]

I'm also adding -debug-dump=Ranges (sic) to the readelf tests. This should be fun.

[eliben]

I'm also adding -debug-dump=Ranges (sic) to the readelf tests. This should be fun.

Sounds like lots of queued-up changes. Thanks for tackling this!
Please make sure to split it into reasonably-chunked PRs so it's easier to review.

[sevaa]

Catch #418. I'll keep ranges until the next one.

[rdunklau]

Thanks ! I'm closing this.

[Kungergely]

Will the .gnu_debuglink section be implemented as well? It seems to be part of the DWARF5 specification too.

[sevaa]

Make a new issue.