nwn_gff unhandled exception

[WilliamDraco]

Attempting to use nwn_gff on Windows 10 produces the following error (command included)

nwn_gff -i .nasher\tmp\module.ifo -o module.ifo.json
gff.nim(534) readGffRoot
Error: unhandled exception: Expectation failed: header.structOffset == 56 [ValueError]`

Also attempted by someone on Linux for whom nwn_gff works as expected on their own files.
File is attached. It is from a 'default mod' - the bare minimum the toolset creates before you can save and close. Also tried the area001.are from the same and it failed with the same error.

module.ifo.zip

[GoLoT]

Same issue when trying to run nwn_gff installed from nimble on Win10. Tried compiling it with mingw-w64 and MSVC, both resulting binaries giving me the header.structOffset == 56 error.

The precompiled binaries (1.2.7) from the releases tab are working with some issues. Mod files can be unpacked, files can be converted to json but it fails to convert some files back to GFF:

nwn_gff -i Repute.json -o Repute.fac --verbose
D [2019-10-12T17:56:00] NWN file encoding: windows-1252
D [2019-10-12T17:56:00] Other file encoding: windows-1252
fatal.nim(39)            sysFatal
Error: unhandled exception: value out of range [RangeError]

Running nwn_gff on linux on that same file works without issues. I tracked down the problem to the GFF->JSON conversion, where invalid values are written as signed (-1) instead of unsigned (4294967295) values.

Linux output, working:

      {
        "__struct_id": 1,
        "FactionGlobal": {
          "type": "word",
          "value": 1
        },
        "FactionName": {
          "type": "cexostring",
          "value": "Hostile"
        },
        "FactionParentID": {
          "type": "dword",
          "value": 4294967295
        }
      },

Windows output, not readable:

      {
        "__struct_id": 1,
        "FactionGlobal": {
          "type": "word",
          "value": 1
        },
        "FactionName": {
          "type": "cexostring",
          "value": "Hostile"
        },
        "FactionParentID": {
          "type": "dword",
          "value": -1
        }
      },

EDIT: RE: the header.structOffset == 56, I noticed that the extracted files (using a self-compiled version of the tools) when unpacking a module are incomplete. At least for areas (.are and .git files) the file length is 1/3 of what it should be and the struct offset value has the right value on the first byte and garbage on the following 3 bytes, causing the assertion to fail. It only happens on windows.

EDIT2: Laputian Bird confirmed that the latest precompiled binaries throw an error when converting a faction file from GFF->JSON->GFF on Win7 too.

[urothis]

Also experiencing this issue on windows 10 pro.

  Extracting DungeonEternalX.mod to C:\Users\urothis\Documents\Projects\dex
      Error: Could not parse .nasher\tmp\011.uti: gff.nim(534)             readGffRoot
         ... Error: unhandled exception: Expectation failed: header.structOffset == 56
         ... [ValueError]`

[squattingmonk]

I'm trying this file on Linux and getting the same error:

# nwn_gff -i module.ifo -o module.ifo.json --verbose
D [2019-10-12T18:20:40] NWN file encoding: windows-1252
D [2019-10-12T18:20:40] Other file encoding: UTF-8
gff.nim(534)             readGffRoot
Error: unhandled exception: Expectation failed: header.structOffset == 56 [ValueError]

[GoLoT]

The problem with that file is that it was corrupted because it wasn't extracted properly from the .mod file, causing the header to hold unexpected values. Linux binaries seem to be working fine.

I don't know what could cause the linux and windows versions to behave differently but the issue is with the windows version not reading files properly (this includes unpacking mod files, as I explained in my previous comment) and writing the json files differently from the linux version (in the case of FAC files it writes signed values instead of unsigned values).

PS: To summarize, windows self compiled binaries truncate and corrupt files extracted from .mod files. The precompiled binaries extract the .mod files properly but write incorrect .json files that can't be converted back to GFF. The problems are isolated to the windows binaries, I tested the same modules on linux and had no issues at all.

[GoLoT]

Tracked down the problem to the writeFile() calls resulting in incomplete files. There is a closed issue ( nim-lang/Nim#12315 ) with details on the fix. Compiling nim from devel branch and recompiling the tools with the devel build fixes the issue.

[WilliamDraco]

Compiling nim from devel branch and recompiling the tools with the devel build fixes the issue.

Confirmed that this resolved the issue. As it is a Nim issue and not these tools specifically, I'd suggest the issue can be closed.

Suggestion to @niv to update the binaries to resolve this alongside the issues GoLoT noted in the pre-compiled binaries.

EDIT: Nim has released 1.0.2, which fixes the compiler issue at the root of this.

[niv]

https://github.com/niv/neverwinter.nim/releases/tag/1.2.8

Thanks everyone for chasing this down. Hopefully that'll be the end of that.