feat(dataminer): migrate generators to extract::Installation#5
Merged
Conversation
Replace legacy KeyBif/Folder/RIM/ERF containers in dataminer with the Installation index API, add shared lookup helpers, and fix TSL nwscript loading to use the TSL chitin path instead of KotOR.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Pull request overview
This PR migrates src/apps/dataminer generators off legacy resource containers onto extract::Installation / FileResource, adds public Installation accessors for enumerating indexed resources, and introduces shared helper utilities to keep lookup order consistent (chitin-only vs override-then-chitin). It also addresses a pre-existing issue where TSL nwscript.nss could be read from the wrong game’s chitin.
Changes:
- Replaced
KeyBifResourceContainer/FolderResourceContainerusage in dataminer generators withextract::Installationlookups and enumeration. - Added
Installationindex accessors (chitinResources(),moduleArchives(),overrideResources()) and a newinstallation_helpers.h. - Fixed the TSL
nwscript.nsslookup bug by using the correct game installation context.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/apps/dataminer/routines.cpp | Switches nwscript parsing to Installation chitin lookup (and fixes the prior TSL/K1 mixup). |
| src/apps/dataminer/models.cpp | Migrates model enumeration/loading to Installation chitin index and FileResource reads. |
| src/apps/dataminer/installation_helpers.h | Adds shared search-scope helpers and resource enumeration helpers for dataminer generators. |
| src/apps/dataminer/guis.cpp | Migrates GUI resource enumeration and lookup to Installation (override-then-chitin for TSL). |
| src/apps/dataminer/gffparsers.cpp | Migrates GFF ingestion to Installation chitin/modules indexing (but currently misses module ingestion due to moduleRoot semantics). |
| src/apps/dataminer/2daparsers.cpp | Migrates 2DA enumeration/loading to Installation chitin index and FileResource reads. |
| src/apps/dataminer/CMakeLists.txt | Adds the new helper header to the dataminer target. |
| include/reone/extract/installation.h | Exposes public accessors for already-indexed chitin/modules/override collections. |
| docs/plans/2026-05-29-dataminer-installation-plan.md | Adds an implementation plan/status doc for the dataminer migration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
289
to
+292
| try { | ||
| auto mdlData = keyBif.findResourceData(ResourceId(resId.resRef, ResType::Mdl)); | ||
| auto mdxData = keyBif.findResourceData(ResourceId(resId.resRef, ResType::Mdx)); | ||
| if (!mdlData || !mdxData) { | ||
| auto mdlData = resource.readData(); | ||
| auto mdxLocation = installation.resource(ResourceId(resource.id().resRef(), ResType::Mdx), chitinOnlyOrder()); | ||
| if (!mdxLocation) { |
Comment on lines
241
to
+243
| } | ||
| twoDAs.push_back({resId.resRef, reader.twoDA()}); | ||
| } | ||
| twoDAs.push_back({resource.id().resRef(), reader.twoDA()}); | ||
| }); |
Comment on lines
259
to
+261
| } | ||
| twoDAs.push_back({resId.resRef, reader.twoDA()}); | ||
| } | ||
| twoDAs.push_back({resource.id().resRef(), reader.twoDA()}); | ||
| }); |
Comment on lines
+301
to
318
| Installation installation(GameID::TSL, k2dir); | ||
| installation.loadChitin(); | ||
| installation.loadModules(); | ||
|
|
||
| auto ingestGff = [&](const FileResource &resource) { | ||
| if (resource.id().type != resType) { | ||
| return; | ||
| } | ||
| auto bytes = keyBif.findResourceData(resId); | ||
| auto stream = MemoryInputStream(*bytes); | ||
| auto bytes = resource.readData(); | ||
| auto stream = MemoryInputStream(bytes); | ||
| auto reader = GffReader(stream); | ||
| reader.load(); | ||
| trees[resId.resRef.value()] = reader.root(); | ||
| } | ||
| trees[resource.id().resRef.value()] = reader.root(); | ||
| }; | ||
|
|
||
| auto modulesPath = getFileIgnoreCase(k2dir, "modules"); | ||
| for (auto &entry : std::filesystem::directory_iterator(modulesPath)) { | ||
| if (!std::filesystem::is_regular_file(entry)) { | ||
| continue; | ||
| } | ||
| auto extension = boost::to_lower_copy(entry.path().extension().string()); | ||
| if (extension == ".rim") { | ||
| auto rim = RimResourceContainer(entry.path()); | ||
| rim.init(); | ||
| for (auto &res : rim.resourceIds()) { | ||
| if (res.type != resType) { | ||
| continue; | ||
| } | ||
| auto bytes = rim.findResourceData(res); | ||
| auto stream = MemoryInputStream(*bytes); | ||
| auto reader = GffReader(stream); | ||
| reader.load(); | ||
| trees[res.resRef.value()] = reader.root(); | ||
| } | ||
| } else if (extension == ".erf") { | ||
| auto erf = ErfResourceContainer(entry.path()); | ||
| erf.init(); | ||
| for (auto &res : erf.resourceIds()) { | ||
| if (res.type != resType) { | ||
| continue; | ||
| } | ||
| auto bytes = erf.findResourceData(res); | ||
| auto stream = MemoryInputStream(*bytes); | ||
| auto reader = GffReader(stream); | ||
| reader.load(); | ||
| trees[res.resRef.value()] = reader.root(); | ||
| } | ||
| } | ||
| } | ||
| forEachResource(installation.chitinResources(), resType, ingestGff); | ||
| forEachResource(installation.moduleArchives(), resType, ingestGff); | ||
|
|
Comment on lines
289
to
+292
| try { | ||
| auto mdlData = keyBif.findResourceData(ResourceId(resId.resRef, ResType::Mdl)); | ||
| auto mdxData = keyBif.findResourceData(ResourceId(resId.resRef, ResType::Mdx)); | ||
| if (!mdlData || !mdxData) { | ||
| auto mdlData = resource.readData(); | ||
| auto mdxLocation = installation.resource(ResourceId(resource.id().resRef(), ResType::Mdx), chitinOnlyOrder()); | ||
| if (!mdxLocation) { |
Comment on lines
241
to
+243
| } | ||
| twoDAs.push_back({resId.resRef, reader.twoDA()}); | ||
| } | ||
| twoDAs.push_back({resource.id().resRef(), reader.twoDA()}); | ||
| }); |
Comment on lines
259
to
+261
| } | ||
| twoDAs.push_back({resId.resRef, reader.twoDA()}); | ||
| } | ||
| twoDAs.push_back({resource.id().resRef(), reader.twoDA()}); | ||
| }); |
Comment on lines
+301
to
318
| Installation installation(GameID::TSL, k2dir); | ||
| installation.loadChitin(); | ||
| installation.loadModules(); | ||
|
|
||
| auto ingestGff = [&](const FileResource &resource) { | ||
| if (resource.id().type != resType) { | ||
| return; | ||
| } | ||
| auto bytes = keyBif.findResourceData(resId); | ||
| auto stream = MemoryInputStream(*bytes); | ||
| auto bytes = resource.readData(); | ||
| auto stream = MemoryInputStream(bytes); | ||
| auto reader = GffReader(stream); | ||
| reader.load(); | ||
| trees[resId.resRef.value()] = reader.root(); | ||
| } | ||
| trees[resource.id().resRef.value()] = reader.root(); | ||
| }; | ||
|
|
||
| auto modulesPath = getFileIgnoreCase(k2dir, "modules"); | ||
| for (auto &entry : std::filesystem::directory_iterator(modulesPath)) { | ||
| if (!std::filesystem::is_regular_file(entry)) { | ||
| continue; | ||
| } | ||
| auto extension = boost::to_lower_copy(entry.path().extension().string()); | ||
| if (extension == ".rim") { | ||
| auto rim = RimResourceContainer(entry.path()); | ||
| rim.init(); | ||
| for (auto &res : rim.resourceIds()) { | ||
| if (res.type != resType) { | ||
| continue; | ||
| } | ||
| auto bytes = rim.findResourceData(res); | ||
| auto stream = MemoryInputStream(*bytes); | ||
| auto reader = GffReader(stream); | ||
| reader.load(); | ||
| trees[res.resRef.value()] = reader.root(); | ||
| } | ||
| } else if (extension == ".erf") { | ||
| auto erf = ErfResourceContainer(entry.path()); | ||
| erf.init(); | ||
| for (auto &res : erf.resourceIds()) { | ||
| if (res.type != resType) { | ||
| continue; | ||
| } | ||
| auto bytes = erf.findResourceData(res); | ||
| auto stream = MemoryInputStream(*bytes); | ||
| auto reader = GffReader(stream); | ||
| reader.load(); | ||
| trees[res.resRef.value()] = reader.root(); | ||
| } | ||
| } | ||
| } | ||
| forEachResource(installation.chitinResources(), resType, ingestGff); | ||
| forEachResource(installation.moduleArchives(), resType, ingestGff); | ||
|
|
ResRef is a struct member on ResourceId, not a callable; fixes dataminer build on Linux and Windows CI.
Record stuck wasm-ci recovery and merge-ready state for PR #5.
Record self-hosted runner recovery and pending wasm-ci after 45m queue stall.
th3w1zard1
added a commit
that referenced
this pull request
May 30, 2026
Replace pre-merge pass 19 draft with final landed state after PR #5 merge.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extract::InstallationInstallationindex accessors for chitin/modules/override enumerationinstallation_helpers.hwith chitin-only and override-then-chitin lookup scopesnwscript.nsswas read from KotORchitin.keyTest plan
KeyBifResourceContainer/FolderResourceContainerusage in dataminerdataminertarget compiles on CI runners