feat(crashdump): Add crash dump functionality for fatal errors #1594

slurmlord · 2025-09-18T09:33:24Z

Merge after bugfix(system): Prevent AMD/ATI driver crash on game launch by front loading the system dbghelp.dll #1066
Merge after fix(debug): Implement reference counter and thread safety in DbgHelpLoader to allow for independent users #1804

This change introduces the option to generate crash dumps, aka. mini dumps, on fatal errors.
The main minidump functionality is done by explicitly loading the dbghelp.dll from the system directory, as the dbghelp.dll that is bundled with the game is an older version that does not include this functionality. There is an option to create small dumps or extended info dumps, currently both are created.

Small dumps

These mostly contain stacks for the process threads and some stack variables, or to create dumps with extended info. The use case for these is to quickly determine where a crash occured, the type of crash, if it was already fixed etc. In addition, if the memory allocation structures are corrupted enough, an extended info dump might not succeed while the small dump should. The size of these dumps are typically on the order of 250kB.

Extended info

These contain global values, along with the memory regions allocated via the memory pool factory and the dynamic memory allocator. This makes all in-game objects available to the person debugging the crash dump, so for example dt generalszh!TheWritableGlobalData in WinDbg will show the state at the time the dump was created.

An alternative option could be to not traverse the memory structures "manually" to get to the allocations and instead just specify the MiniDumpWithFullMemory flag to MiniDumpWriteDump, but that increases the file size considerably.

As an example, dump of the generalszh process in the main menu with the shell map in the background yields a ~140MB dump when traversing and ~420MB with MiniDumpWithFullMemory. Beyond that, the ~140MB file compresses to ~20MB with 7Z, so should be relatively easily transferable.

Storage Location

Crash dumps are stored in a new folder called 'CrashDumps' under the userDir ("Documents\Command and Conquer Generals Zero Hour Data"), and on startup it will create this directory if it doesn't exist and delete any older dumps so only the 10 newest small and 2 newest extended info dumps are left. This is to preserve disk space, as the extended info files can be several hundred MB.

Integration points

For VS2022 builds, unhandled exceptions end up in the UnhandledExceptionFilter in WinMain, which then get a reference to the actual exception that occurred and includes that in the dump.
For VC6 builds, unhandled exceptions are caught in the catch(...) blocks of GameEngine::execute which then calls RELEASE_CRASH. As there is no exception data available in this case to populate _EXCEPTION_POINTERS from, an intentional exception is triggered to get the trace of the current thread. This makes the stack traces for VC6 a bit more cryptic than VS2022 builds as the C++ exception handling gets included in the trace.

Limitations

In the longer run we'll probably want to replace this code with a more mature solution, like CrashPad, but that currently depends on a newer compiler than VC6.
As the code is intended to be temporary, it's kept behind a new CMake feature so it can be easily removed. There are also some other decisions made with this in mind:

Minidump is created in-process. Ideally, the dump should be performed by a process external to the crashing/failing process, but to avoid having to ship an extra binary, in-process was chosen instead. It's being performed in a separate thread to hopefully have a clean stack to work with.
Depends on RTS_BUILD_OPTION_VC6_FULL_DEBUG for VC6 builds. The PDBs generated with the default VC6 compile options are lacking in information, making the mini dumps less useful. The option RTS_BUILD_OPTION_VC6_FULL_DEBUG should be enabled for VC6 builds to ensure maximum usability. VS2022 builds produce better PDBs and require no extra options.
Directory management code is contained within the MiniDumper class, not re-usable by other components.
Many Win32-specific types and functions are used directly without regards for portability.
As the MiniDump feature is not available for VC6, a lot of headers have been borrowed from minidumpapiset.h and included in the MiniDumper_compat.h file.
Globals are used for storing the current exception info.
Only enabled for the games, tools are currently not included.

OmniBlade

Looks good, tested dump generation with a 2022 build.

slurmlord · 2025-10-09T12:41:24Z

Another approach I realized after publishing could be to move the GameMemory allocations from using GlobalAlloc to instead create a separate GameMemory heap and use HeapAlloc.
The benefit would be a lot less code required to traverse the allocations for inclusions the dump, as it could be done with HeapWalk instead for only the GameMemory heap.

xezon

Looks good overall. Just a bunch of small comments.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Source/Common/System/Debug.cpp

Generals/Code/Main/WinMain.cpp

Core/GameEngine/Include/Common/MiniDumper.h

Generals/Code/Main/WinMain.cpp

Core/GameEngine/Include/Common/GameMemory.h

Core/GameEngine/Source/Common/System/GameMemory.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

slurmlord · 2025-10-14T19:54:04Z

Pushed a commit attempting to address most of the comments in the previous review:

Made TheMiniDumper a pointer, declared in MiniDumper.cpp
Moved extern TheMiniDumper to MiniDumper.h
Moved MiniDumper ctor body to MiniDumper.cpp
(Attempted to) find and correct all the stylistically misplaced "{"
Added enum for dump type
Fall back to DUMP_TYPE_FULL when DUMP_TYPE_GAMEMEMORY is requested and game memory implementation is turned off
Removed cmake feature dependency on RTS_GAMEMEMORY_ENABLE
Removed thread Id from dump file name
Replace hard-coded list of executable names to match with GetModuleFileNameW
(Attempted to) clean up log format
Added Enum values for dump thread exit codes
Removed MiniDumper::m_endRangeIter

In addition:

Added new static init/shutdown methods, using placement new on the process heap
Prefixed win32 api function calls with global namespace ::
Added asserts for dumping thread active state

Remaining:
Merge with #1066 when merged.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Skyaero42 · 2025-11-01T12:45:39Z

I think this should be enabled by default for now and included in our weekly pre-releases and legi.cc/patch distributions. The patch still crashes now and then without being able to find a cause (replays are send to us, but there is no crash to be found). For example, Legi (and only Legi) crashed yesterday on stream while being on the patch. No cause was found.

Players on the patch (with crashdump on) can send us the crashdumps and thus help finding where the issues/instabilities are.

slurmlord · 2025-11-04T19:48:09Z

I think this should be enabled by default for now and included in our weekly pre-releases and legi.cc/patch distributions. The patch still crashes now and then without being able to find a cause (replays are send to us, but there is no crash to be found). For example, Legi (and only Legi) crashed yesterday on stream while being on the patch. No cause was found.

Players on the patch (with crashdump on) can send us the crashdumps and thus help finding where the issues/instabilities are.

Agreed, that's the main use case I was hoping this could address. Once this is merged, the remaining hurdle would be to get the weekly builds to enable the Vc6FullDebug cmake option.

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h

xezon

I think the cleanup in MiniDumper class needs another look. Make it simple and robust.

xezon · 2025-11-16T11:07:19Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 		m_quitting = NULL;
 	}

 	DbgHelpLoader::unload();


if (m_loadedDbgHelp) ?

cmake/config-memory.cmake

Core/GameEngine/Source/Common/System/MiniDumper.cpp

xezon

Code looks good to me. A few minor comments that could be looked into.

xezon · 2025-11-19T18:31:52Z

Core/GameEngine/Source/Common/System/Debug.cpp

+		TheMiniDumper->TriggerMiniDump(DUMP_TYPE_GAMEMEMORY);
+	}
+
+	MiniDumper::shutdownMiniDumper();


This code block exists twice in this file. Perhaps consolidate?

xezon · 2025-11-19T18:44:43Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+#elif RTS_ZEROHOUR
+	Char product = 'Z';
+#endif
+	Char dumpTypeSpecifier = dumpType == DUMP_TYPE_MINIMAL ? 'M' : 'X';


There are 3 types, but only identifiers here. Maybe use 3? Maybe use a lookup array or function for these? M, X is also used in InitializeDumpDirectory

xezon · 2025-11-19T18:49:20Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		// Only include data segments for the game and ntdll modules to keep dump size low
+		if (output.ModuleWriteFlags & ModuleWriteDataSeg)
+		{
+			if (::StrCmpIW(input.Module.FullPath, m_executablePath) && !::StrStrIW(input.Module.FullPath, L"ntdll.dll"))


Maybe also keep some other dll's, like kernel32 ?

xezon · 2025-11-19T18:55:42Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+#endif
+	memset(m_dumpDir, 0, ARRAY_SIZE(m_dumpDir));
+	memset(m_dumpFile, 0, ARRAY_SIZE(m_dumpFile));
+	memset(m_executablePath, 0, ARRAY_SIZE(m_executablePath));


This is fine, but it would also be sufficient to just do m_executablePath = "" or m_executablePath[0] = 0

xezon · 2025-11-19T18:57:24Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+Bool MiniDumper::InitializeDumpDirectory(const AsciiString& userDirPath)
+{
+	constexpr Int MaxExtendedFileCount = 2;
+	constexpr Int MaxMiniFileCount = 10;


constexpr const (because VS6 does not see constexpr)

xezon · 2025-11-19T18:59:51Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+#if RTS_GENERALS
+	Char product = 'G';
+#elif RTS_ZEROHOUR
+	Char product = 'Z';


Can make const

OmniBlade approved these changes Sep 23, 2025

View reviewed changes

xezon added Major Severity: Minor < Major < Critical < Blocker Gen Relates to Generals ZH Relates to Zero Hour Debug Is mostly debug functionality System Is Systems related labels Oct 8, 2025

xezon reviewed Oct 9, 2025

View reviewed changes

xezon reviewed Oct 16, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/MiniDumper.cpp Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

xezon added the Approved Pull Request was approved label Oct 21, 2025

slurmlord added 5 commits November 3, 2025 20:47

feat(crashdump): Add crash dump functionality

7e85db6

Address review feedback

68aa823

Fix build break when game memory is disabled

0216b55

Review feedback

a5a2419

Moved dbghelp interaction to DbgHelpLoader

90a0b49

slurmlord force-pushed the feature-crashdump branch from 7bb8e8c to 90a0b49 Compare November 4, 2025 19:43

Fix build break when crashdumps are disabled

f96f2d4

slurmlord mentioned this pull request Nov 5, 2025

build(weekly): Enable complete debug symbols for VC6 weekly builds #1803

Merged

xezon reviewed Nov 5, 2025

View reviewed changes

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

slurmlord added 2 commits November 9, 2025 20:05

Merge remote-tracking branch 'origin/main' into feature-crashdump

f58d620

Merge with updated DbgHelpLoader behavior

c4398cf

slurmlord commented Nov 12, 2025

View reviewed changes

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h Show resolved Hide resolved

xezon reviewed Nov 16, 2025

View reviewed changes

slurmlord added 2 commits November 18, 2025 21:12

Simplified shutdown and cleanup

d1a1048

Lowered PDB info text from WARNING to STATUS

6923fcd

xezon approved these changes Nov 19, 2025

View reviewed changes

feat(crashdump): Add crash dump functionality for fatal errors #1594

Are you sure you want to change the base?

feat(crashdump): Add crash dump functionality for fatal errors #1594

Uh oh!

Conversation

slurmlord commented Sep 18, 2025 • edited by xezon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Small dumps

Extended info

Storage Location

Integration points

Limitations

Uh oh!

OmniBlade left a comment

Choose a reason for hiding this comment

Uh oh!

slurmlord commented Oct 9, 2025

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slurmlord commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Skyaero42 commented Nov 1, 2025

Uh oh!

slurmlord commented Nov 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

xezon Nov 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

xezon Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xezon Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xezon Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xezon Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xezon Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

xezon Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

slurmlord commented Sep 18, 2025 •

edited by xezon

Loading