[Extractors] Thread map-extractor and vmap-extractor tile processing - #5
Merged
Merged
Conversation
Add a -t/--threads option to map-extractor and convert each map's ADT tiles across a worker pool (0 = auto-detect cores, default; 1 = serial), mirroring the movemap generator's threading. - per-map tile queue drained by N workers; each .map output file is independent, so the result is identical regardless of thread count - per-tile scratch buffers (area/height/liquid grids) are thread_local - success/failed counters are std::atomic - MPQ reads are serialised by one mutex held only around the archive read in ConvertADT (StormLib's shared per-archive file position has no internal lock, so concurrent reads from one handle are unsafe); the parse/pack/write that follows runs in parallel - zero every per-tile scratch buffer up front so a tile's output never inherits residual bytes from the prior tile (which made the .map bytes depend on tile order -- non-deterministic across threads) Mirrors the same change in the mangosthree in-tree extractor, where full extraction at --threads 1 vs auto was verified byte-for-byte identical.
Add -t/--threads to vmap-extractor and process each map's ADT tiles across a worker pool (0 = auto-detect cores, default; 1 = serial). - one mutex in the MPQFile ctor serialises all archive reads (StormLib's shared per-archive file position has no internal lock); parsing and geometry conversion run in parallel - model dedup: a worker claims a model, extracts it, and any worker referencing the same model blocks on a condition variable until its file is on disk -- the placement written next (ModelInstance) opens that file and drops the spawn if it is missing - each tile writes its own temp dir_bin file (a placement is several fwrites, so a shared handle would interleave records), concatenated in tile order afterwards - GenerateDoodadUniqueId's shared map + counter are mutex-guarded - failedPaths is accumulated per worker and merged - g_WmoDoodads is populated by the serial ExtractWmo pass that runs first, so it is read-only during the threaded tile pass Mirrors the mangosthree in-tree change, verified on-client: same model and vmap file sets and identical dir_bin records (doodad ids aside) vs serial.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Threads both map-extractor and vmap-extractor — each adds
-t/--threads(0 = auto-detect cores, default; 1 = serial) and processes a map's ADT tiles across a worker pool, mirroring the movemap generator. Ports the same change shipped in the mangosthree in-tree extractors.map-extractor (commit 1)
.mapfile is independent → identical output regardless of thread countthread_local; countersstd::atomic; one MPQ-read mutex around the archive read inConvertADT.mapbytes tile-order-dependent → non-deterministic across threads)vmap-extractor (commit 2)
MPQFilector serialises archive reads; parsing + geometry conversion run in parallelModelInstancedrops the placement otherwise (this dropped ~156 placements in testing before the fix)dir_binfiles concatenated in order (avoids interleaved records);GenerateDoodadUniqueIdmutex-guarded; per-workerfailedPathsVerified: both build green (MSVC). The identical mangosthree changes were validated on-client — map output byte-for-byte identical (serial vs auto); vmap produces the same model + vmap file sets and identical
dir_binrecords (doodad-id values aside, which are order-dependent).This change is