-
Notifications
You must be signed in to change notification settings - Fork 11.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
'Soft Stop' solution on offset overflow issue: By Produceing a truncated but valid DWP file, discarding any DWO files that would not fit within the 32 bit/4GB limits of the format. #71902
'Soft Stop' solution on offset overflow issue: By Produceing a truncated but valid DWP file, discarding any DWO files that would not fit within the 32 bit/4GB limits of the format. #71902
Conversation
✅ With the latest revision this PR passed the C/C++ code formatter. |
7fd2678
to
0271dfa
Compare
The whole point of the We could add another flag that is sort of "best effort" DWP that emits a correct one with as many CUs as it can... but that doesn't seem /super/ useful? |
I agree with @dwblaikie. It is possible to re-construct CU index. It was implemented in LLVM toolchain including LLDB. (String size is a different issue, but there are work arounds for it in llvm). Maybe GDB can do the same? |
Right - if you're using gdb the recommendation would be not to pass this flag, and to make changes to reduce debug info to fit into the current DWARF DWP limit of 4GB per section. We could add another flag that's "give up and provide a valid but truncated dwp at 4GB" which seems a bit awkward, but perhaps no more/less awkward than the current "create a corrupted index that we know can be recovered from". Alternatively you could go and work with gdb to do some similar recovery. (I did just have the thought that our recovery, especially in DWARFv5, could be better - we could tombstone the .debug_info column values in the index (set them to
Right, the What you want is less a |
I think a list of .o/.dwo files can be specified to llvm-dwp to produce .dwp file, correct? |
How about the option AllowOverflow ? |
How is that different from an option that is there now? |
How about calling the new flag 'StopOnCuIndexOverflow'? Since the old patch is to continue packaging dwo when overflow, the new patch will stop packaging. |
Probably need some terminology to disambiguate between the current default "stop" behavior which is to abort/error/fail, and the new flag behavior of "bail out and emit a best-effort DWP". (that said, I'd still ask whether this is the right tool to add at all - I know it's a complicated space for all of us hitting these limits, so I'm not ruling anything out for sure, but trying to reduce all these different slices as much as we can - could you build less code in your situation? Or modify your build system to drop certain debug info to keep the dwp under these limits?) |
I think this could be described as a 'soft stop' (or Graceful Stop), as opposed to the current default 'hard stop' (Abortive Stop).
The business scenario we are dealing with involves a large number of different projects all experiencing overflow issues. We are indeed considering modifying our build system to generate DWP only for the DWO parts we are interested in. |
and using LLDB is no feasible? |
currently only gdb |
Personally I don't have a strong feeling about this, and understand trying to make giant build system work for everyone. |
@llvm/pr-subscribers-debuginfo Author: Jinjie Huang (Labman-001) ChangesWhen 'ContinueOnCuIndexOverflow' enables without this modification, the forcibly generated '.dwp' won't be recognized by Debugger(gdb 10.1) correctly. About patch: Full diff: https://github.com/llvm/llvm-project/pull/71902.diff 2 Files Affected:
diff --git a/llvm/lib/DWP/DWP.cpp b/llvm/lib/DWP/DWP.cpp
index 89101ca7e5736ba..3080939133cfebc 100644
--- a/llvm/lib/DWP/DWP.cpp
+++ b/llvm/lib/DWP/DWP.cpp
@@ -197,11 +197,14 @@ static Error sectionOverflowErrorOrWarning(uint32_t PrevOffset,
return make_error<DWPError>(Msg);
}
-static Error addAllTypesFromDWP(
- MCStreamer &Out, MapVector<uint64_t, UnitIndexEntry> &TypeIndexEntries,
- const DWARFUnitIndex &TUIndex, MCSection *OutputTypes, StringRef Types,
- const UnitIndexEntry &TUEntry, uint32_t &TypesOffset,
- unsigned TypesContributionIndex, bool ContinueOnCuIndexOverflow) {
+static Error
+addAllTypesFromDWP(MCStreamer &Out,
+ MapVector<uint64_t, UnitIndexEntry> &TypeIndexEntries,
+ const DWARFUnitIndex &TUIndex, MCSection *OutputTypes,
+ StringRef Types, const UnitIndexEntry &TUEntry,
+ uint32_t &TypesOffset, unsigned TypesContributionIndex,
+ bool ContinueOnCuIndexOverflow, bool &SeeOverflowFlag,
+ bool StopOnCuIndexOverflow) {
Out.switchSection(OutputTypes);
for (const DWARFUnitIndex::Entry &E : TUIndex.getRows()) {
auto *I = E.getContributions();
@@ -235,6 +238,9 @@ static Error addAllTypesFromDWP(
if (Error Err = sectionOverflowErrorOrWarning(
OldOffset, TypesOffset, "Types", ContinueOnCuIndexOverflow))
return Err;
+ if (StopOnCuIndexOverflow)
+ SeeOverflowFlag = true;
+ return Error::success();
}
}
return Error::success();
@@ -244,7 +250,8 @@ static Error addAllTypesFromTypesSection(
MCStreamer &Out, MapVector<uint64_t, UnitIndexEntry> &TypeIndexEntries,
MCSection *OutputTypes, const std::vector<StringRef> &TypesSections,
const UnitIndexEntry &CUEntry, uint32_t &TypesOffset,
- bool ContinueOnCuIndexOverflow) {
+ bool ContinueOnCuIndexOverflow, bool &SeeOverflowFlag,
+ bool StopOnCuIndexOverflow) {
for (StringRef Types : TypesSections) {
Out.switchSection(OutputTypes);
uint64_t Offset = 0;
@@ -276,6 +283,9 @@ static Error addAllTypesFromTypesSection(
if (Error Err = sectionOverflowErrorOrWarning(
OldOffset, TypesOffset, "types", ContinueOnCuIndexOverflow))
return Err;
+ if (StopOnCuIndexOverflow)
+ SeeOverflowFlag = true;
+ return Error::success();
}
}
}
@@ -583,7 +593,8 @@ Error handleSection(
}
Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
- bool ContinueOnCuIndexOverflow) {
+ bool ContinueOnCuIndexOverflow,
+ bool StopOnCuIndexOverflow) {
const auto &MCOFI = *Out.getContext().getObjectFileInfo();
MCSection *const StrSection = MCOFI.getDwarfStrDWOSection();
MCSection *const StrOffsetSection = MCOFI.getDwarfStrOffDWOSection();
@@ -613,6 +624,7 @@ Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
uint32_t ContributionOffsets[8] = {};
uint16_t Version = 0;
uint32_t IndexVersion = 0;
+ bool SeeOverflowFlag = false;
DWPStringPool Strings(Out, StrSection);
@@ -687,12 +699,17 @@ Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
uint32_t SectionIndex = 0;
for (auto &Section : Obj.sections()) {
if (SectionIndex == Index) {
- return sectionOverflowErrorOrWarning(
- OldOffset, ContributionOffsets[Index], *Section.getName(),
- ContinueOnCuIndexOverflow);
+ if (Error Err = sectionOverflowErrorOrWarning(
+ OldOffset, ContributionOffsets[Index], *Section.getName(),
+ ContinueOnCuIndexOverflow))
+ return Err;
}
++SectionIndex;
}
+ if (StopOnCuIndexOverflow) {
+ SeeOverflowFlag = true;
+ break;
+ }
}
}
@@ -722,6 +739,10 @@ Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
InfoSectionOffset, InfoSectionOffset + C.getLength32(),
"debug_info", ContinueOnCuIndexOverflow))
return Err;
+ if (StopOnCuIndexOverflow) {
+ SeeOverflowFlag = true;
+ break;
+ }
}
UnitOffset += C.getLength32();
@@ -752,6 +773,8 @@ Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
Info.substr(UnitOffset - C.getLength32(), C.getLength32()));
InfoSectionOffset += C.getLength32();
}
+ if (SeeOverflowFlag)
+ break;
}
if (!FoundCUUnit)
@@ -762,7 +785,7 @@ Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
if (Error Err = addAllTypesFromTypesSection(
Out, TypeIndexEntries, TypesSection, CurTypesSection, CurEntry,
ContributionOffsets[getContributionIndex(DW_SECT_EXT_TYPES, 2)],
- ContinueOnCuIndexOverflow))
+ ContinueOnCuIndexOverflow, SeeOverflowFlag, StopOnCuIndexOverflow))
return Err;
}
continue;
@@ -860,9 +883,12 @@ Error write(MCStreamer &Out, ArrayRef<std::string> Inputs,
if (Error Err = addAllTypesFromDWP(
Out, TypeIndexEntries, TUIndex, OutSection, TypeInputSection,
CurEntry, ContributionOffsets[TypesContributionIndex],
- TypesContributionIndex, ContinueOnCuIndexOverflow))
+ TypesContributionIndex, ContinueOnCuIndexOverflow,
+ SeeOverflowFlag, StopOnCuIndexOverflow))
return Err;
}
+ if (SeeOverflowFlag)
+ break;
}
if (Version < 5) {
diff --git a/llvm/tools/llvm-dwp/llvm-dwp.cpp b/llvm/tools/llvm-dwp/llvm-dwp.cpp
index f976298dcadc2f4..8383f5e77cea8b4 100644
--- a/llvm/tools/llvm-dwp/llvm-dwp.cpp
+++ b/llvm/tools/llvm-dwp/llvm-dwp.cpp
@@ -144,6 +144,7 @@ int llvm_dwp_main(int argc, char **argv, const llvm::ToolContext &) {
OutputFilename = Args.getLastArgValue(OPT_outputFileName, "");
ContinueOnCuIndexOverflow = Args.hasArg(OPT_continueOnCuIndexOverflow);
+ SoftStopOnCuIndexOverflow = Args.hasArg(OPT_stopOnCuIndexOverflow);
for (const llvm::opt::Arg *A : Args.filtered(OPT_execFileNames))
ExecFilenames.emplace_back(A->getValue());
@@ -255,7 +256,8 @@ int llvm_dwp_main(int argc, char **argv, const llvm::ToolContext &) {
if (!MS)
return error("no object streamer for target " + TripleName, Context);
- if (auto Err = write(*MS, DWOFilenames, ContinueOnCuIndexOverflow)) {
+ if (auto Err = write(*MS, DWOFilenames, ContinueOnCuIndexOverflow,
+ SoftStopOnCuIndexOverflow)) {
logAllUnhandledErrors(std::move(Err), WithColor::error());
return 1;
}
|
e30f401
to
1d6a206
Compare
@dwblaikie @ayermolo I have merged the 'soft-stop' and 'continue' operations into the 'OnCuIndexOverflow' enumeration.
|
The the original continue-on-cu-index-overflow option's default value has been set to 'continue', please help to review it, thx. |
Generally looks OK to me - I guess this isn't practical to test due to the size needed to exceed the limit? |
back from PTO. I'll take a look later today/tomorrow. |
llvm/lib/DWP/DWP.cpp
Outdated
return Err; | ||
if (AnySectionOverflow) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment as above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Please wait for David to approve also. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Thanks for the help with the review @ayermolo
This is follow up for llvm#71902. The default option --continue-on-cu-index-overflow returned an error --continue-on-cu-index-overflow: missing argument. Changed it so that it is the same behavior as other flags like -gsplit-dwarf. Where --continue-on-cu-index-overflow will default to continue, and user can set mode with --continue-on-cu-index-overflow=<value>.
This is follow up for #71902. The default option --continue-on-cu-index-overflow returned an error --continue-on-cu-index-overflow: missing argument. Changed it so that it is the same behavior as other flags like -gsplit-dwarf. Where --continue-on-cu-index-overflow will default to continue, and user can set mode with --continue-on-cu-index-overflow=\<value>.
This is follow up for llvm/llvm-project#71902. The default option --continue-on-cu-index-overflow returned an error --continue-on-cu-index-overflow: missing argument. Changed it so that it is the same behavior as other flags like -gsplit-dwarf. Where --continue-on-cu-index-overflow will default to continue, and user can set mode with --continue-on-cu-index-overflow=\<value>.
When 'ContinueOnCuIndexOverflow' enables without this modification, the forcibly generated '.dwp' won't be recognized by Debugger(gdb 10.1) correctly.
it looks like there's a problem with processing the dwarf header, which makes debugging completely impossible.
About patch:
When llvm-dwp enables option '--continue-on-cu-index-overflow=soft-stop' and recognizes the overflow problem , it will stop packing and generate the '.dwp' file at once, discarding any DWO files that would not fit within the 32 bit/4GB limits of the format.