Skip to content

Port unique updater scripts from PoB2#9480

Open
EtherealCarnivore wants to merge 33 commits intoPathOfBuildingCommunity:devfrom
EtherealCarnivore:port-unique-updater-scripts
Open

Port unique updater scripts from PoB2#9480
EtherealCarnivore wants to merge 33 commits intoPathOfBuildingCommunity:devfrom
EtherealCarnivore:port-unique-updater-scripts

Conversation

@EtherealCarnivore
Copy link
Contributor

Summary

Ports the unique item updater scripts from PathOfBuilding-PoE2 (PRs #54, #65, #175), building on the work started in #8700 by @Wires77. Extends coverage to all 20 PoE1 item types and fixes multiple bugs found during testing.

New scripts

  • uTextToMods.lua -- converts unique item text to mod IDs (run when adding new uniques)
  • uModsToText.lua -- converts mod IDs back to text from GGPK data (run when game data updates)

Changes to existing files

  • mods.lua -- adds ModItemExclusive.lua and ModTextMap.lua generation
  • statdesc.lua -- handles ! negation limits and ranges starting/ending at 0

Bug fixes over original PoE2 scripts

  • uModsToText.lua: last item in each file wasn't getting mods flushed (no separator at EOF)
  • uModsToText.lua: mod ID regex misidentified base type names containing hyphens (e.g. "Two-Point Arrow Quiver") as legacy mod ranges
  • uModsToText.lua: unresolved text lines lost their position among stat-ordered mods
  • uModsToText.lua: nil access on statOrder when processing legacy-only mods
  • uTextToMods.lua: greedy {.+} tag stripping consumed multiple tags; fixed to lazy {.-}
  • uTextToMods.lua: mod selection logic could skip better matches; separated into two-pass approach
  • uTextToMods.lua: usedMods was global; modTextMap reloaded inside loop unnecessarily
  • Enabled all 20 PoE1 item types (original only had axe enabled with PoE2 types commented out)

Bug fixes in Item.lua (pre-existing)

  • excluded/exclude variable name mismatch creating an accidental global (lines 103-143)
  • Wrong Lua pattern ^%[a ]+ corrected to ^[%a ]+ for jewel radius parsing
  • Space indentation converted to tabs in jewel radius block

Data update

  • All unique item text files re-exported with current GGPK data
  • Adds stat ordering, updated mod wording, and catalyst tags for amulets/rings

Addresses review feedback from @Nightblade on #8700. Supersedes #8700.

Test plan

  • Full test suite: 195/195 passing, 0 failures
  • No "unrecognised item" warnings during load
  • PoB loads and runs without errors
  • Spot-check unique items in PoB UI for correct mod display
  • Verify catalyst tags on amulets/rings are accurate

Port the unique item updater scripts from PathOfBuilding-PoE2 (PRs PathOfBuildingCommunity#54,
PathOfBuildingCommunity#65, PathOfBuildingCommunity#175) and fix multiple bugs for PoE1 compatibility.

Scripts added:
- uTextToMods.lua: converts unique item text to mod IDs (run when adding
  new uniques)
- uModsToText.lua: converts mod IDs back to text with fresh GGPK data
  (run when game data updates)

Changes to existing files:
- mods.lua: add ModItemExclusive.lua and ModTextMap.lua generation
- statdesc.lua: handle '!' negation limits and ranges starting at 0

Bug fixes over the original PoE2 scripts:
- Fix uModsToText.lua not flushing mods for last item in each file
- Fix mod ID regex misidentifying base type names containing hyphens
  (e.g. "Two-Point Arrow Quiver") as legacy mod ranges
- Fix unresolved text lines losing position among ordered mods
- Fix nil access on statOrder when processing legacy-only mods
- Fix uTextToMods.lua greedy tag stripping pattern and mod selection
- Make usedMods local and move modTextMap load outside loop
- Enable all 20 PoE1 item types (original only had axe enabled)

Bug fix in Item.lua:
- Fix excluded/exclude variable name mismatch creating accidental global
- Fix wrong Lua pattern ^%[a ]+ to correct ^[%a ]+ in jewel radius
- Fix space indentation to tabs in jewel radius block
- ModItemExclusive.lua: unique/exclusive mod data generated by mods.lua
- Export/Uniques/ModTextMap.lua: mod text to mod ID mapping
- Export/Uniques/*.lua: intermediate mod ID format for all item types
Re-export all unique item text files using the updater scripts with
current game data. Updates stat ordering, mod wording, and adds catalyst
tags to amulets and rings.
@Wires77
Copy link
Member

Wires77 commented Mar 3, 2026

I'm going to push some changes to this branch to start fixing a lot of these uniques:

  • Reverted the changes to the unique text files themselves, as they were missing tags and other things. Going through each item type one-by-one is a lot more maintainable and until then we don't want to lose the original text.
  • Updated the scripts to remove several PoE2-exclusive concepts/tags
  • Overhauled the TextToMods script to be more robust and accurate

Even with these changes, I'm just going to go through each item type one at a time so they can be inspected thoroughly without one-shotting the whole thing. There are a lot of manual fixes that are needed, which is why the uTextToMods script was left out of the PoE2 one. At the end I'll probably disable it again once we have the mod files in a good state.

I'm even considering commenting out the statOrder sorting to make diffs more readable until we know the text and parsing itself is in a good state. Would've been nice, but it's too much code to bother with. The ordering might actually help find the correct mods too.

Known issues:

  • Mods don't get converted to generic parsing unless it's a variant. So far this has been most visible on old prophecy uniques, because their base unique got a variant, but obviously we didn't add one for the upgraded version
  • permyriad type stats are a bit flaky and I probably missed some
  • Mods with multiple lines of stats are not handled at all. Usually this results in the first line being properly converted to a mod, leaving the others. Then when it's converted back into unique text, you have duplicate lines
  • Added flat lightning mods aren't getting legacy values applied. Probably fixable, but so far I've been fixing manually
  • "Reduced" and "Less" legacy mods aren't getting their negative signs in the values

This script now handles sorting mods on preference to remove as much
manual mod entry as possible.  There are still many caveats with using
it, so item types have been commented out for now with the intent to
tackle each type one at a time until we are confident in its data.
@Wires77
Copy link
Member

Wires77 commented Mar 4, 2026

One thing that is near-impossible to automate is when GGG did the base item implicit upgrade. The mods are full of implicits like this:

Implicits: 2
{variant:1}ElementalDamagePercentImplicitSceptreNew16[10,10]
{variant:2,3}ElementalDamagePercentImplicitSceptreNew16

When you search on the trade site and sort by the implicit, you'll see that the 10% implicits are really 20%, and are likely ElementalDamagePercentImplicitSceptre1 for a mod. You still have to put the legacy brackets in for that, which is why it likely won't matter for our use case unless they go back and update those old implicit values.

@PJacek
Copy link
Contributor

PJacek commented Mar 5, 2026

I did some work here: https://github.com/PJacek/PathOfBuilding/tree/port-unique-updater
The updated script reads how many implicits there are in the input file and outputs the number of implicit lines actually written.

@EtherealCarnivore
Copy link
Contributor Author

Added jewels on top of body armour, boots, and gloves. Rest of the item types (helmets, shields, quivers, belts, rings, amulets) still to go - skipping flasks and tinctures since someone else is on those.

@EtherealCarnivore
Copy link
Contributor Author

Re-exported body, boots, gloves, and jewel through the full pipeline on Windows:

  1. Ran uTextToMods.lua (Data -> Export) to convert remaining raw text lines to mod IDs
  2. Ran uModsToText.lua (Export -> Data) to regenerate Data files with proper stat ordering and legacy mod resolution

Had to use Windows for this since uModsToText.lua needs dat() from the GGPK to resolve legacy mods with [min,max] ranges - those can't be stubbed out on macOS.

Test suite: 194/195 passing (1 pre-existing TradeQueryCurrency failure, unrelated).

Wires77 and others added 4 commits March 4, 2026 21:26
Pattern was missing dots so mods with values like 0.2 (leech rates etc)
would fail to parse and crash describeStats with nil comparison.
Copy link
Member

@Wires77 Wires77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a bunch of duplicated lines in these latest commits, due to the multiline bullet in my other comment. The output should look nearly identical to the existing item unless there are some mod changes we didn't catch in earlier leagues.

We'll also want to commit the Export/Uniques file for future automation

{variant:3}20% less Effect of your Curses
Socketed Hex Curse Skills are Triggered by Doedre's Effigy when Summoned
Hexes from Socketed Skills can apply 5 additional Curses
{variant:2}20% less Effect of Curses from Socketed Hex Skills
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check your output for duplicated lines. These already exist above, because the one mod is three lines in one.

Wires77 and others added 3 commits March 4, 2026 23:37
When a mod ID resolves to N lines of text, the export file also has
N-1 plain text continuation lines after it. Without skipping those,
they'd get output twice (once from the mod resolution, once as
unresolved text).
@Wires77
Copy link
Member

Wires77 commented Mar 5, 2026

One last thing I've noticed on a handful of mods is that GGG has set their value to 0, like this section for Lightpoacher explains on the wiki. This affects us by making the mod not appear at all in ModItemExclusive, so we can't describe it. Could look into making them appear there, but otherwise I've just left the raw text in Export/Uniques

Wires77 and others added 3 commits March 5, 2026 02:01
Rewrote io.linesBackward to read the whole file into memory instead
of the old chunk-based approach - the 4KB-chunk reader broke on
Windows because text-mode \r\n conversion threw off the seek math,
producing garbled lines at chunk boundaries.

After resolving a multi-line mod to its ID, strip stale continuation
lines from the output. Also handles sibling mods that share the same
first line (Dream/Nightmare jewels have 4 mods starting with the
same text but different continuations).

Reverted the skipLines workaround in uModsToText - it was eating
legitimate variant override lines (e.g. GrantShaperSkill_1 variants).

Fixed a few ModItemExclusive wording mismatches (Violent Dead
recovery rate/speed, unarmed attack/attacks pluralization, DNT tag).
@EtherealCarnivore
Copy link
Contributor Author

Fix multi-line mod handling in export round-trip
Rewrote io.linesBackward to read the whole file into memory instead
of the old chunk-based approach - the 4KB-chunk reader broke on
Windows because text-mode \r\n conversion threw off the seek math,
producing garbled lines at chunk boundaries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants