OverflowError: can't convert negative int to unsigned #134

nitro322 · 2023-07-12T18:17:35Z

I've encountered this issue with two NSPs using version 4.3:

$ nsz -C -l 22 -L -B -s 25 -K -t $(nproc) -V file.nsp
Failed to extract TitleID/Version from filename "file.nsp". Use -p to extract from Cnmt.
Block compressing (level 22 ldm) file.nsp -> file.nsz
[ADDING]     45455a52b6be1b4b6647c5fb5f7ae72f.cnmt.xml 941 bytes to NSP
[ADDING]     b1abe96d2142ccefc51a23f55a290bcf.ncz 502906880 bytes to NSP
Compressing 100%|█████████████| 479/479 MiB [00:15<00:00, 32.95 MiB/s]compressed 47% 502906880 -> 236514061  - b1abe96d2142ccefc51a23f55a290bcf.nca
[ADDING]     45455a52b6be1b4b6647c5fb5f7ae72f.cnmt.nca 3584 bytes to NSP
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.11/site-packages/nsz/BlockCompressor.py", line 213, in blockCompressNsp
    with Pfs0.Pfs0Stream(container.getHeaderSize() if removePadding else container.getFirstFileOffset(), str(nszPath)) as nsp:
  File "/home/user/.local/lib/python3.11/site-packages/nsz/Fs/Pfs0.py", line 34, in __exit__
    self.close()
  File "/home/user/.local/lib/python3.11/site-packages/nsz/Fs/Pfs0.py", line 62, in close
    self.write(self.getHeader())
               ^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/nsz/Fs/Pfs0.py", line 90, in getHeader
    h += (f['offset'] - headerSize).to_bytes(8, byteorder='little')
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: can't convert negative int to unsigned

That's obviously with block compression, but get the same error with the default solid compression. This has occurred in 2 of the 371 NSPs I've tried compressing with NSZ v4.3. One NSP is a game title, the other DLC. Both install successfully via goldleaf, so they seem to be fine, but don't know any other good way to test and verify them.

The text was updated successfully, but these errors were encountered:

nicoboss · 2023-07-16T16:08:24Z

This is an extreamly interesting issue describing a case which should be impossible. Let's look at the following Python code:

stringTable = '\x00'.join(file['name'] for file in self.files)
headerSize = 0x10 + len(self.files) * 0x18 + len(stringTable)
remainder = 0x10 - headerSize % 0x10
headerSize += remainder
(...)
for f in self.files:
    h += (f['offset'] - headerSize).to_bytes(8, byteorder='little')

First we calculate the stringTable by joining all filenames together separated by a Null character.
We then calculate the headerSize by first adding 16 bytes for the PFS0 header prior to the FileEntryTable. We then add the size of the FileEntryTable by multiplying the amount of files by 24 bytes (size of an entry inside the FileEntryTable). Finally we add the length of the stringTable which is the last part of the PFS0 header. After this we round up to the next 16 bit block. After this the variable headerSize should exactly contain the size of the PFS0 header as described on https://wiki.oatmealdome.me/PFS0_(File_Format)

The only way f['offset'] - headerSize can be negative is if the offset of a file stored inside the PFS0 container would be inside the PFS0 header which should be impossible. Let's assume the offset would be specified wrongly then reading this file must result in currupted data as it would start within the PFS0 header. I also can't se any way how above described headerSize calucaltion could be wrong.

Maybe the tool which packed your game to PFS0 forgot padding the stringTable to the next 16 byte block? Actually we should be padding to the next 32 byte block according to https://wiki.oatmealdome.me/PFS0_(File_Format) but not sure if correct as everyone seams to be padding to the next 16 byte block.

This issue also reminds me of your "Decompressed NSP checksum mismatch" issue as there we also had an PFS0 file where the offset of the first file was 0 instead of headerSize. I wounder why this issue didn't occur there and why the PFS0 header hash was unable to detect a missmatch as the FileEntryTable is clearly part of the PFS0 header hash.

I just updated the --info command on latest master to include headerSize and the offset of every file. Could you please check out latest master and poste the result of --info on this file? Please censure/remove all sensitive information like "titleKey", "titleKeyDec" and all "key Block" before posting or better PM me the result on GBAtemp or "nicobosshard" on Discord. Please also redo the --info output of the original and broken NSP file from your "Decompressed NSP checksum mismatch" issue with the latest master so I can collect more information regarding your other issue as well.

swiblet · 2023-07-17T04:39:29Z

I also am getting this error message, although the rest of the text seems to be different. I didn't want to create a new issue if this is the same problem, so forgive me if this actually ends up being a different problem. I know nothing about programming.

Here is what NSZ.exe says before outputting an invalid nsz file:

``Solid compressing (level 0) M:\PATH -> M:\PATH.nsz
[ADDING] 010073401175f002000000000000000b.tik 704 bytes to NSP
[ADDING] 010073401175f002000000000000000b.cert 1792 bytes to NSP
[ADDING] 8ab888acce274615b76e7ff97d190963.cnmt.nca 3584 bytes to NSP
[ADDING] 1e0ec52206adc14f043f674a3ac982b3.ncz 114688 bytes to NSP
Compressed 14.61181640625% 114688 -> 16758 - 1e0ec52206adc14f043f674a3ac982b3.nca
Traceback (most recent call last):
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 132, in solidCompressNsp
with Pfs0.Pfs0Stream(container.getHeaderSize() if removePadding else container.getFirstFileOffset(), str(nszPath)) as nsp:
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 34, in exit
self.close()
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 62, in close
self.write(self.getHeader())
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 90, in getHeader
h += (f['offset'] - headerSize).to_bytes(8, byteorder='little')
OverflowError: can't convert negative int to unsigned

nut exception: [WinError 32] The process cannot access the file because it is being used by another process: "M:\PATH.nsz"
Process Process-2:
Traceback (most recent call last):
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 132, in solidCompressNsp
with Pfs0.Pfs0Stream(container.getHeaderSize() if removePadding else container.getFirstFileOffset(), str(nszPath)) as nsp:
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 34, in exit
self.close()
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 62, in close
self.write(self.getHeader())
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 90, in getHeader
h += (f['offset'] - headerSize).to_bytes(8, byteorder='little')
OverflowError: can't convert negative int to unsigned

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self.kwargs)
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz_init.py", line 36, in solidCompressTask
outFile = solidCompress(filePath, compressionLevel, keepDelta, removePadding, useLongDistanceMode, outputDir, threadsToUse, statusReport, id, pleaseNoPrint)
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 16, in solidCompress
return solidCompressNsp(filePath, compressionLevel, keepDelta, removePadding, useLongDistanceMode, outputDir, threads, statusReport, id, pleaseNoPrint)
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\SolidCompressor.py", line 138, in solidCompressNsp
nszPath.unlink()
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\pathlib.py", line 1206, in unlink
self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: "M:\PATH.nsz"
Exception ignored in: <function BaseFile.del at 0x000001ABC338D000>
Traceback (most recent call last):
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\File.py", line 41, in del
self.close()
File "C:\Users\swibl\Desktop\nsz_v4.3.0_win64_portable\lib\site-packages\nsz\Fs\Pfs0.py", line 62, in close
self.write(self.getHeader())

Compressing 100%|████████████████████████████████████████████████████████████████████| 0/0 MiB [00:00<00:00, 0.00 MiB/s] h += (f['offset'] - headerSize).to_bytes(8, byteorder='little')ackages\nsz\Fs\Pfs0.py", line 90, in getHeader
OverflowError: can't convert negative int to unsigned

Compressing 100%|████████████████████████████████████████████████████████████████████| 0/0 MiB [01:34<00:00, 0.00 MiB/s]``

Also I know it says that the file is being used by another process. It's definitely not, though.

nicoboss · 2023-07-17T23:27:03Z

@swiblet Thanks a lot for reporting your findings. This is indeed the same issue. I unfortunately still haven't found what causes this. To help me investigate this issue it would be awesome if you could provide some more information. To do so either run latest master from source or get the latest portable Windows release and replace BaseFs.py with https://raw.githubusercontent.com/nicoboss/nsz/master/nsz/Fs/BaseFs.py and File.py with https://raw.githubusercontent.com/nicoboss/nsz/master/nsz/Fs/File.py. Then execute --info on your file or press the "Info" button if you use the GUI. Please censure/remove all sensitive information like "titleKey", "titleKeyDec" and all "key Block" entries before posting or PM me the result on GBAtemp (nicoboss) or Discord (nicobosshard).

wsteel · 2023-07-21T10:39:56Z

PFS0

magic = b'PFS0'
fsType = None
cryptoType = Crypto.NONE
size = 6091060036
headerSize = 388
offset = 0 - (0)

        ****************************************************************


        Files:


        Ticket

        File Path: 1.tik
        File Size: 704
        File Offset: 388
        signatureType = TicketSignature.RSA_2048_SHA256
        keyType = 0
        masterKeyRev = 16 (master_key_0f)
        ticketId = 0000000000000000
        deviceId = 0000000000000000
        rightsId = 0
        accountId = 00000000
        titleId = x
        titleKey = x
        titleKeyDec = x

        ****************************************************************

        File Path: 1.cert
        File Size: 1792
        File Offset: 1092

        ****************************************************************


        NCA Archive

        File Path: 1.cnmt.nca
        File Size: 3584
        File Offset: 2884
        magic = b'NCA3'
        titleId = x
        rightsId = b'00000000000000000000000000000000'
        isGameCard = 0x0
        contentType = Content.META
        cryptoType = Crypto.NONE
        Size: 3584
        crypto master key: 2
        crypto master key2: 16
        key Index: 0
        key Block: b'00000000000000000000000000000000'
        key Block: b'00000000000000000000000000000000'
        key Block: 1
        key Block: b'00000000000000000000000000000000'

        ****************************************************************


        NCA Archive

        File Path: 1.nca
        File Size: 6090571776
        File Offset: 6468
        magic = b'NCA3'
        titleId = x
        rightsId = 1
        isGameCard = 0x0
        contentType = Content.PROGRAM
        cryptoType = Crypto.NONE
        Size: 6090571776
        crypto master key: 2
        crypto master key2: 16
        key Index: 0
        key Block: 1
        key Block: 1
        key Block: 1
        key Block: 1
        build Id: None

        ****************************************************************


        NCA Archive

        File Path: 1.nca
        File Size: 311808
        File Offset: 6090578244
        magic = b'NCA3'
        titleId = x
        rightsId = b'00000000000000000000000000000000'
        isGameCard = 0x0
        contentType = Content.CONTROL
        cryptoType = Crypto.NONE
        Size: 311808
        crypto master key: 2
        crypto master key2: 16
        key Index: 0
        key Block: b'00000000000000000000000000000000'
        key Block: b'00000000000000000000000000000000'
        key Block: 1
        key Block: b'00000000000000000000000000000000'

        ****************************************************************


        NCA Archive

        File Path: 1.nca
        File Size: 169984
        File Offset: 6090890052
        magic = b'NCA3'
        titleId = x
        rightsId = b'00000000000000000000000000000000'
        isGameCard = 0x0
        contentType = Content.MANUAL
        cryptoType = Crypto.NONE
        Size: 169984
        crypto master key: 2
        crypto master key2: 16
        key Index: 0
        key Block: b'00000000000000000000000000000000'
        key Block: b'00000000000000000000000000000000'
        key Block: 1
        key Block: b'00000000000000000000000000000000'

        ****************************************************************

wsteel · 2023-07-21T10:46:26Z


PFS0

magic = b'PFS0'
fsType = None
cryptoType = Crypto.NONE
size = 3997030602
headerSize = 266
offset = 0 - (0)

        ****************************************************************


        Files:


        Ticket

        File Path: .tik
        File Size: 704
        File Offset: 266
        signatureType = TicketSignature.RSA_2048_SHA256
        keyType = 0
        masterKeyRev = 16 (master_key_0f)
        ticketId = 0000000000000000
        deviceId = 0000000000000000
        rightsId = 
        accountId = 00000000
        titleId = 
        titleKey = 
        titleKeyDec = 

        ****************************************************************

        File Path: .cert
        File Size: 1792
        File Offset: 970

        ****************************************************************


        NCA Archive

        File Path: .cnmt.nca
        File Size: 3584
        File Offset: 2762
        magic = b'NCA3'
        titleId = 
        rightsId = 
        isGameCard = 0x0
        contentType = Content.META
        cryptoType = Crypto.NONE
        Size: 3584
        crypto master key: 2
        crypto master key2: 16
        key Index: 0
        key Block:
        key Block:
        key Block:
        key Block:

        ****************************************************************


        NCA Archive

        File Path: .nca
        File Size: 3997024256
        File Offset: 6346
        magic = b'NCA3'
        titleId = 
        rightsId = 
        isGameCard = 0x0
        contentType = Content.PUBLICDATA
        cryptoType = Crypto.NONE
        Size: 3997024256
        crypto master key: 2
        crypto master key2: 16
        key Index: 0
        key Block:
        key Block:
        key Block:
        key Block:

        ****************************************************************

nicoboss · 2023-07-21T22:56:37Z

Your headerSize is 266 and the offset of the first file is on 266 as well. Let's do the calculation to see if this PFS0 header matches the standard:

We have 16 bytes for the PFS0 header prior to the FileEntryTable. We then add the size of the FileEntryTable by multiplying the number of files by 24 bytes which in this case is 4 * 24 => 72

So we know that the Array of null-terminated filename strings, padded to 0x20 must be 266 - 72 => 194 bytes long. 194 cannot be divided by 16 or 32 (only 2 and 97).

With this we proofed that whatever tool created this NSP file did not follow the PFS0 standard as documented on https://wiki.oatmealdome.me/index.php?title=PFS0_(File_Format) in 2018. There unfortunately is no documentation about this padding inside https://switchbrew.org/w/index.php?title=NCA to verify if it must be there and if it is supposed to 0x20 or 0x10 big.

We are making it us too easy by just not supporting what could be considered invalid NSP files. As long most applications see them as valid we should support them. We however need to keep in mind that bit-identical recreation means that we also have to recreate this missing padding. This can luckily easily be done.

I implemented a potential fix for this issue inside #135. @nitro322, @thatch or @wsteel Could anyone of you please test the latest commit on the removal-of-string-table-padding branch and let me know if this fixes your issue? If it does please hash the file before compression and after decompression to make sure they are still bit-identical recreated. If it doesn't work please post an --info while being on the removal-of-string-table-padding branch. Thank you so much for your help investigating this issue.

wsteel · 2023-07-22T05:52:46Z

I checked the two files.
The first one has an error

[ADDING]     010---.tik 704 bytes to NSP
[ADDING]     010---.cert 1792 bytes to NSP
[ADDING]     bf9---.cnmt.nca 3584 bytes to NSP
Skipping not packed cc1---.nca
[ADDING]     cc1---.nca 6090571776 bytes to NSP
[ADDING]     0b7---.nca 311808 bytes to NSP
[ADDING]     cf8---.nca 169984 bytes to NSP

the second one works prefect, and all files hash are correct.

swiblet · 2023-07-23T07:21:47Z

@swiblet Thanks a lot for reporting your findings. This is indeed the same issue. I unfortunately still haven't found what causes this. To help me investigate this issue it would be awesome if you could provide some more information. To do so either run latest master from source or get the latest portable Windows release and replace BaseFs.py with https://raw.githubusercontent.com/nicoboss/nsz/master/nsz/Fs/BaseFs.py and File.py with https://raw.githubusercontent.com/nicoboss/nsz/master/nsz/Fs/File.py. Then execute --info on your file or press the "Info" button if you use the GUI. Please censure/remove all sensitive information like "titleKey", "titleKeyDec" and all "key Block" entries before posting or PM me the result on GBAtemp (nicoboss) or Discord (nicobosshard).

Very sorry for the delay, I'll do this right now!

swiblet · 2023-07-23T08:43:03Z

Alright, so after downloading the newest release (the same one I used before) and replacing those two files you linked me, I tested with like 10 NSP's and they all compressed perfectly. Previously, like 9 out of 10 of them were having the error I posted above. I'm not sure if those files were updated with a fix, but if they were, I'd say it worked!

nicoboss · 2023-08-05T11:27:00Z

@nitro322, @swiblet, @wsteel This issue should be fixed in the latest NSZ 4.4.0 release. Can you please test and confirm that this issue got fixed in the latest release?

rlaphoenix · 2023-08-05T11:35:33Z

The odds that I was testing this as you release 4.4.0 hahaha

rlaphoenix · 2023-08-05T12:31:16Z

I can personally confirm, it has fixed it in my case.

nitro322 · 2023-08-05T16:57:18Z

Out of town right now, but should be able to test early next week. Thanks a bunch for looking into this.

…

On August 5, 2023 6:27:12 AM CDT, Nico Bosshard ***@***.***> wrote: @nitro322, @swiblet, @wsteel This issue should be fixed in the latest [NSZ 4.4.0](https://github.com/nicoboss/nsz/releases) release. Can you please test and confirm that this issue got fixed in the latest release? -- Reply to this email directly or view it on GitHub: #134 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

nitro322 · 2023-08-08T04:09:09Z

NSZ compression works on one of the problematic NSPs from my original report. I can't find/remember which other NSP failed to compress, so assuming that one would work as well.

Thumbs up here. Thanks for fixing!

Just as an FYI, I still have that problem reported here with a couple NSPs failing to decompress back to the original checksum. The NSP that failed to compress for this issue is now a third suffering from that issue. I know you didn't report it fixed, but just wanted to test to confirm. Would it be helpful to create a separate bug report for that?

nicoboss · 2023-08-10T19:17:05Z

@rlaphoenix @nitro322 Thanks a lot for testing. I'm really glad this fix worked.

@nitro322 Oh sorry. I only looked at the PFS0 header diff between original and decompressed NSP so far and realized that there were extremely strange values for the PFS0 file offsets. From the 26 bytes that where different almost all where related to file offsets and increased by 0x10. I then got stuck as I thought I lack any further information as I didn't encounter any such case in any of my own dumps. Likely because this issue can't occur for games dumped using nxdumptool. However, I just realized that you provided much more information which I completely missed. I was now able to finally recreate your issue for the first time and can confirm its existence. Feel free to create separate issue regarding those rare cases where bit-identical PFS0 recreation fails if you want. I will give my best to fix this issue during the next weekend and do another release once this is fixed.

nicoboss self-assigned this Jul 13, 2023

nicoboss added the bug Something isn't working label Jul 13, 2023

nicoboss mentioned this issue Jul 21, 2023

Removal of stringTable padding #135

Merged

nicoboss closed this as completed in #135 Jul 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OverflowError: can't convert negative int to unsigned #134

OverflowError: can't convert negative int to unsigned #134

nitro322 commented Jul 12, 2023

nicoboss commented Jul 16, 2023 •

edited

swiblet commented Jul 17, 2023 •

edited

nicoboss commented Jul 17, 2023 •

edited

wsteel commented Jul 21, 2023 •

edited

wsteel commented Jul 21, 2023

nicoboss commented Jul 21, 2023

wsteel commented Jul 22, 2023

swiblet commented Jul 23, 2023

swiblet commented Jul 23, 2023

nicoboss commented Aug 5, 2023

rlaphoenix commented Aug 5, 2023

rlaphoenix commented Aug 5, 2023

nitro322 commented Aug 5, 2023 via email

nitro322 commented Aug 8, 2023

nicoboss commented Aug 10, 2023

OverflowError: can't convert negative int to unsigned #134

OverflowError: can't convert negative int to unsigned #134

Comments

nitro322 commented Jul 12, 2023

nicoboss commented Jul 16, 2023 • edited

swiblet commented Jul 17, 2023 • edited

nicoboss commented Jul 17, 2023 • edited

wsteel commented Jul 21, 2023 • edited

wsteel commented Jul 21, 2023

nicoboss commented Jul 21, 2023

wsteel commented Jul 22, 2023

swiblet commented Jul 23, 2023

swiblet commented Jul 23, 2023

nicoboss commented Aug 5, 2023

rlaphoenix commented Aug 5, 2023

rlaphoenix commented Aug 5, 2023

nitro322 commented Aug 5, 2023 via email

nitro322 commented Aug 8, 2023

nicoboss commented Aug 10, 2023

nicoboss commented Jul 16, 2023 •

edited

swiblet commented Jul 17, 2023 •

edited

nicoboss commented Jul 17, 2023 •

edited

wsteel commented Jul 21, 2023 •

edited