>4GB offsets in MCDataFragment in llvm-21 (llvm-dwp OOM)

[MatzeB]

We are having trouble with llvm-dwp running out of memory in llvm-21 for big builds resulting in giant dwp files where some sections are > 4GB in size. This makes creating a reproducer a bit tricky, but I'll describe what we are seeing:

It looks like recent MC layer rewrites, like 9beb467 introduce fields like uint32_t ContentStart to MCEncodedFragment. And my understanding is that in our case llvm-dwp ends up creating a single fragment bigger than 4GB after repeatet emitBytes calls here: https://github.com/llvm/llvm-project/blob/main/llvm/lib/DWP/DWP.cpp#L860
(The OOM situation seems to be accidental by ContentEnd overflowing causing MCEncodedFragment::getContentsForAppending to wrongly grow the contents vector to arbitrary size).

The code seems to have changed lately but I am still seeing various 32bit offsets in the fragment code. Should we increase those fields to 64bits or better find a way to not write everything into a single fragment in this use case?

CC @MaskRay

[MatzeB]

Looking at this some more; I think it's still just multiple smaller fragments adding up to a section bigger than 4GB.

So maybe we just need a fix for llvm-21 to bump the ContentStart and ContentEnd fields of MCEncodedFragment to 64bits...

[llvmbot]

@llvm/issue-subscribers-llvm-mc

Author: Matthias Braun (MatzeB)

We are having trouble with `llvm-dwp` running out of memory in llvm-21 for big builds resulting in giant dwp files where some sections are > 4GB in size. This makes creating a reproducer a bit tricky, but I'll describe what we are seeing:

It looks like recent MC layer rewrites, like 9beb467 introduce fields like uint32_t ContentStart to MCEncodedFragment. And my understanding is that in our case llvm-dwp ends up creating a single fragment bigger than 4GB after repeatet emitBytes calls here: https://github.com/llvm/llvm-project/blob/main/llvm/lib/DWP/DWP.cpp#L860
(The OOM situation seems to be accidental by ContentEnd overflowing causing MCEncodedFragment::getContentsForAppending to wrongly grow the contents vector to arbitrary size).

The code seems to have changed lately but I am still seeing various 32bit offsets in the fragment code. Should we increase those fields to 64bits or better find a way to not write everything into a single fragment in this use case?

CC @MaskRay

[MaskRay]

So maybe we just need a fix for llvm-21 to bump the ContentStart and ContentEnd fields of MCEncodedFragment to 64bits...

The fields are kept as 32-bit to minimize sizeof(MCFragment). This is crucial for maintaining low memory consumption in the integrated assembler, which is the most important user.

A potential solution for DWP is to change MCObjectStreamer::appendContents(ArrayRef<char> Contents) to create multiple fragments, each with size <= 32-bit.