Padding between debug_str_offsets is not supported by DWARF verifier

[felipepiovezan]

If we run the verifier with --all on llvm/test/DebugInfo/X86/dwarfdump-str-offsets.s,
it will claim the input is invalid because it adds padding between debug_str_offsets:

        .long .debug_str_offsets_segment0_end-.debug_str_offsets_base0+4
        .short 5    # DWARF version
        .short 0    # Padding
.debug_str_offsets_base0:
        .long str_producer
        .long str_CU1
        .long str_CU1_dir
        .long str_Subprogram
        .long str_Variable1
        .long str_Variable2
        .long str_Variable3
.debug_str_offsets_segment0_end:
# A 4-byte gap.
        .long 0

Now, if we interpret the DWARF spec literally, it doesn't talk about padding.
It really makes it sound like offset_of_dwarf_version + length must point to
the next contribution. Indeed, this is what the verifier expects (and why this
test fails to verify).

Dwarfdump, on the other hand, takes a different approach. It calls
collectContributionData(Units), which scans all compile units, extracts their
DW_AT_str_offsets_base and uses those to build a list of "this is where the
different debug_str_offset contributions start". It even "subtracts" the header
size from those to find the headers.

So one utility acknowledges (and prints!) those gaps because it uses an indirect way
to find where they are, but the other rejects them. How to reconcile these?

[llvmbot]

@llvm/issue-subscribers-debuginfo

Author: Felipe de Azevedo Piovezan (felipepiovezan)

If we run the verifier with --all on `llvm/test/DebugInfo/X86/dwarfdump-str-offsets.s`, it will claim the input is invalid because it adds padding between debug_str_offsets:

        .long .debug_str_offsets_segment0_end-.debug_str_offsets_base0+4
        .short 5    # DWARF version
        .short 0    # Padding
.debug_str_offsets_base0:
        .long str_producer
        .long str_CU1
        .long str_CU1_dir
        .long str_Subprogram
        .long str_Variable1
        .long str_Variable2
        .long str_Variable3
.debug_str_offsets_segment0_end:
# A 4-byte gap.
        .long 0

Now, if we interpret the DWARF spec literally, it doesn't talk about padding.
It really makes it sound like offset_of_dwarf_version + length must point to
the next contribution. Indeed, this is what the verifier expects (and why this
test fails to verify).

Dwarfdump, on the other hand, takes a different approach. It calls
collectContributionData(Units), which scans all compile units, extracts their
DW_AT_str_offsets_base and uses those to build a list of "this is where the
different debug_str_offset contributions start". It even "subtracts" the header
size from those to find the headers.

So one utility acknowledges (and prints!) those gaps because it uses an indirect way
to find where they are, but the other rejects them. How to reconcile these?

[llvmbot]

@llvm/issue-subscribers-tools-llvm-dwarfdump

Author: Felipe de Azevedo Piovezan (felipepiovezan)

If we run the verifier with --all on `llvm/test/DebugInfo/X86/dwarfdump-str-offsets.s`, it will claim the input is invalid because it adds padding between debug_str_offsets:

        .long .debug_str_offsets_segment0_end-.debug_str_offsets_base0+4
        .short 5    # DWARF version
        .short 0    # Padding
.debug_str_offsets_base0:
        .long str_producer
        .long str_CU1
        .long str_CU1_dir
        .long str_Subprogram
        .long str_Variable1
        .long str_Variable2
        .long str_Variable3
.debug_str_offsets_segment0_end:
# A 4-byte gap.
        .long 0

Now, if we interpret the DWARF spec literally, it doesn't talk about padding.
It really makes it sound like offset_of_dwarf_version + length must point to
the next contribution. Indeed, this is what the verifier expects (and why this
test fails to verify).

Dwarfdump, on the other hand, takes a different approach. It calls
collectContributionData(Units), which scans all compile units, extracts their
DW_AT_str_offsets_base and uses those to build a list of "this is where the
different debug_str_offset contributions start". It even "subtracts" the header
size from those to find the headers.

So one utility acknowledges (and prints!) those gaps because it uses an indirect way
to find where they are, but the other rejects them. How to reconcile these?

[dwblaikie]

Eh - there's been some chatter in the DWARF committee (@pogo59 ) about documenting whether sections can contain gaps, or padding, and how to implement that padding - and Paul's mostly been advocating, I tihnk, for saying that you can't have arbitrary gaps in sections and you should use encoding-valid representations of padding (exactly what is is different in different sections).

Historically, padding has been allowed, somewhat incidentally (because you only need to read the .debug_info (and debug_aranges, and debug_pubnames maybe) sections directly, and everything else is read indirectly) - sections read indirectly naturally support unspecified gaps/padding/garbage because there's no reason for a reader to read those bytes unless it knows some secret contract that suggests it should read it.

Binutils dwarfdump has been pretty good about not dumping these gaps - llvm-dwarfdump less so, I think it does read some sections end-to-end even when they are only needed to be accessed indirectly. I'm surprised we do the right thing with debug_str_offsets - maybe due to the DWARFv4 DWARFv5 encoding changes in debug_str_offsets.dwo, not sure... shrug

I don't think it's super important either way, and I'm not sure my opinion here is the popular one, probably not, I'm fine with not allowing gaps - wouldn't especially mind if llvm-dwarfdump dumped the debug_str_offsets section without consulting an units, just parsing table after table.

[felipepiovezan]

Thank you for the background, that helps a lot!

because there's no reason for a reader to read those bytes unless it knows some secret contract that suggests it should read it.

Yup, this makes sense, I guess it is only the verifier / dump tools that are special in that way.

I don't think it's super important either way, and I'm not sure my opinion here is the popular one, probably not, I'm fine with not allowing gaps

No strong preferences either way from me too, but it would be nice to hear if there are any good use cases for padding.

[jmorse]

Drive-by comment -- something similar came up a long time ago to do with padding in .debug_loc, recorded now at #42635. The tl;dr is that GNU ld will dead-strip code and install holes in .debug_loc, that readelf/dwarfdump can cope with, but the LLVM tools can't. I think the conclusion at the time was that it's just not a section you can read without also reading .debug_info. Less relevant to this issue as DWARF5 uses .debug_loclists instead, but this is certainly an existing category of limitation of the LLVM tools.