zstandard versions newer than 0.15.2 break compatibility with all title installers except DBI #120
Comments
|
Can you please --verify the original file and check if NSZ marks it as corrupted? If it marks it as corrupted make sure your dump is clean and not modified in any way. If this only affects a single game maybe let me know which one. |
|
I tried and still the same.
If I go to dbi and using network install with the same file from nginx (protocol http also) then I can run without problem
|
|
|
|
I'm having the same issue. All packages were packaged and verified with level 22, they install just fine, NCAs are clean and confirmed, then when trying to start, I get the corrupted data message. 4.1.0 with 14.1.2 keys. Update: I got corrupted install for all installers, tinleaf, tinwoo, etc etc. I'll try tomorrow with a exFAT drive to see if it helps. All my NSZ are solid compression. |
|
Update: DBI 415 works perfectly either through USB (MTP listener) or USB Drive, no other title installer software seems to be working and are all producing a corrupted installation when using level 22 compression with 4.1.0, solid compression. Didn't try any other options yet, will do soon. |
|
Can you post the error that you got from tinfoil? Should be in the console section. |
|
Sure thing, will reinstall a title this afternoon. |
|
@blawar , sorry for the delay, I just tried it again, got the new Tinfoil update for 15.0, restarted, but still having the same issue, the nsz's install just fine (no complaints about signature, hash, nothing), no errors are shown during the installation, however, after intalling the software, they start briefly and then the corrupted data comes up. What I noticed is that Tinfoil installed much quicker than DBI itself, not sure if those are related. Everything shows green on DBI so I believe this might not be due to the nsz algorithm itself, but rather due to some change which wasn't implemented on the title installers after the 4.1.0 update (?). Not sure, though with nsz 4.0.1 I never got this corrupted data error with any Tinfoil version. What I noticed is that the used storage after installing with Tinfoil was much lower (as in, more storage was still available on the Switch) than with DBI (installation with Tinfoil took 4 GB of storage, installation with DBI took 6.2 GB). |
|
Can you please try to compress the exact same game with NSZ 4.0.1 to see if it works there? Does this issue occur with every game or only for some specific ones? If it only occurs for specific games please give an example. If you conveard such a non-working NSZ back to NSP does it still have issues with Tinfoil? There should be no changes to the file format between NSZ v4.0.1 and v4.1.0 (except it's now skipping the content meta xml like many other NSZ compressers do) so if this issue only occurs in NSZ v4.1.0 something is likely quite wrong that version. It's really strange that this only seems to happen for Tinfoil (and related projects like tinleaf) while there is no issue with other title installers. Tinfoil is known to have quite good NSZ support so this is quite surprising. I would expect a corrupted NSZ file to lead to a corrupted installation no matter what title installer you use. Given that --verify marks it as fine indicates that there is no corruption. The size difference makes it look as if Tinfoil forgets to install something other title installers do. Honestly no idea but I will give my best to fix this once I can reproduce it unless it’s a bug in Tinfoil in which case blawar would need to fix it. |
|
similiar problem exists with atmoxl or the very outdated awoo. |
|
I just tested the new versions of tinfoil and lithium and I couldn't proceed at all. The same 2TB ExFAT HDD that was recognized by the softwares back in 14.1.2 doesn't show up any longer so I can't even test installing. DBI does see the HDD, it installs NSZs perfectly, however, it cannot install XCZs, either block or solid, but I'm assuming that's a problem with DBI #rashevskyv/dbi#145 so I opened an issue there. Using Atmosphere 1.4.0, HOS 15.0.0, nsz 4.1.0. I tested TinLeaf as well, it does see and installs the XCZs, however, the same error as before happens, 2002-4000 I believe. @blawar , Neither Tinfoil nor Lithium (both v 15.0) are closing properly with HOS 15.0.0 (EDIT: downgraded to 14.1.2, both errors persist) either, both hang on the "Shutting down" screen forever and never go back to hbmenu (I waited 1 minute to no avail with both), at least for me. |
|
@nicoboss , another update after some testing this morning. Summary of my findings is here: rashevskyv/dbi#145 (comment) |
|
@jacoghi Can you or anyone else test if NSZ 4.0.1 also has this issue or if this is something introduced with the latest NSZ 4.1.0 update. Does this issue affect every game or only certain games? I'm now taking this issue very seriously as it seems to affect many users and affects booth NSZ and XCZ. |
nicoboss
added
bug
|
According to rashevskyv/dbi#145 there should be a wrong hash-table in the XCZ header and the hfs0_header_size be wrong. I don't remember to even have touched any of that in the last update so knowing if NSZ 4.0.1 has the same issue would be quite important. But even if true there also seam to be issues with NSZ files. They don't even have a hfs0_header_size field so there has to be more to this. |
|
NSZ 4.0.1 had this issue as well. I started noticing this issue almost a year ago when I upgraded my Python from 3.7 to 3.10. If I went back to 3.7 then it seemed fine. Don't know if it's due to the newer version of Python or the newer version of the dependency module. |
This is actually really interesting. Never thought about something like that. I always assumed Python versions to behave equally. The CI/CD pipeline only checks Python 3.6.0, 3.6.8, 3.7.6 and 3.8.3 so if there is something wrong with Python 3.10 I would never have noticed. One of the few major NSZ 4.1.0 changes that isn’t mentioned in the changelogs was packing the Windows release with Python 3.10 instead of a much older version. If true this affects everyone who uses the latest Windows release. This could also explain why I'm having troubles reproducing this issue. This is definitely something I will try. |
|
@nicoboss, I personally don't recall ever running into this issue with 4.0.1, however, as @tonyzzz321 wisely brought up, I always used Py 3.9.3 (pre-bundled or cloned from repo), and recently upgraded to 3.10.5 when I started using 4.1.0. I assumed this was related to the nsz package since all title installers are giving me headaches except for DBI, and at least when it comes to nszs, I haven't faced any issues with that installer, just xczs. I guess we could try cloning the repo and using an older version of Python to test? |
|
@jacoghi It would be very helpful if you could test with some older Python versions and test NSZ 4.0.1 with a game you know has the issue. Do you get this issue with every game or only for some specific games? If it's only for some games can you please give me an example? I myself will try with the latest Python version and the latest dependencies and see if I can finally reproduce this. |
|
@nicoboss, every single xci game I tried. Every nsz as well, using installers other than DBI. It's kinda complicated to narrow it down since I upgraded my keys file, Python and nsz all at once. I'll give 3.9 a go with 4.1.0 from source tonight and see if it helps for both formats using other title installers. |
|
I am 90% sure you have a sig patch problem. |
|
@blawar, would it make any sense for the sigpatches to only affect nsz / xcz (and some title installers) and not nsp / xci (which install just fine with every single title installer)? |
|
@nicoboss, I cloned the repo and checked out 4.1.0. Uninstalled Py 3.10 and went back to 3.9.13. Decompressed and recompressed. First interesting thing, the files compressed with 3.9.13 are 1 ~ 1.5 MB smaller than the the ones produced with 3.10, exactly the same settings. Unfortunately, that didn't solve the problem, they still install correctly but give an error when running for the first time. Downgrading to latest 3.7, will post results later. |
|
@jacoghi which version of python zstandard do you have installed? Newer versions are bugged, you have to use older version. try zstandard==0.15.2 |
|
@blawar The one currently being used for 3.7 is 0.19.0, didn't check the newer Python versions. I'll give that version a shot, I just compressed the first ones using 3.7 and they did go further. The games opened, but eventually crashed, something that didn't happen before. I'll try downgrading zstandard and we'll see how it goes, I just checked out version nsz 4.0.1 and am compressing with the same packages to see if nsz is directly the culprit or not. |
|
You misunderstand @jacoghi , it’s not the version of python, it’s the version of zstandard. Install zstandard 0.15.2 |
|
This is still broken, with these settings: Using DBI to install the compressed nsz it installs fine and plays. I then tried installing with tinwoo/atmoxl/tinfoil and all of these failed. I could install OK, so I dumped the installed nsp with nxdumptool and then extracted the nsp on my computer. What I found is that the main nca file is corrupt, so that's why it doesn't work - the header is fine but the sections are all corrupt. Also when you install with dbi it takes far longer to install, probably because this works (extracts the NCA) properly. |
Please try using the latest AtmoXL-Titel-Installer which got released yesterday. It will take some time for all title installers to fix this issue. |
|
Thanks, I tried that latest version from yesterday, it's still broken with those settings. So is Tiinfoil. I could send you the compressed nsz file and you can see for yourself, DBI installs and plays properly, everything else fails - it will install just fine, but crash when starting using other installers. NCA is getting corrupted. |
You are using Block compression. This is an extension to the NSZ file format not supported by AtmoXL and TinWoo. Please try without block compression. Block compression in Tinfoil never had this issue and should work. I will test if Block compression still works in latest Tinfoil. |
|
Yep, installs work fine with block compression turned off, however on some "shops" these installs with block compression will fail in tinfoil. I attempted to DL 2 nsp's yesterday with tinfoil from a shop and the game would start but would crash. I'll try updating tinfoil again and see if that solves the issue. Out of curiosity do you think you'll release some code for AtmoXL to fix/add block compression support? |
The NSZ block compression variant was manly developed to be used for emulators and to execute compressed games on real hardware once fs is reimplemented in Atmosphere. I still did not have the time to implement block compressed NSZ mounting for PC but plan to do so soon. fs reimplementation seems like it could happen next year if SciresM isn't distracted with irl things that much. Due to the use of block compressed NSZ files currently being quite limited it did not gain broad adoption so far. Now that I have a development setup for AtmoXL I could add support for it. It would not be that much effort to do so just kind of annoying as there is no open-source title installer with NSZ block compression support so I would need to implement this from the ground up. It's definitely something I'm willing to do if there is enough demand for it. @mrdude2478 I saw that you are the author of TinWoo. Your nca_writer.cpp is also based on Adubbz Tinfoil. Because of this your title installer is also affected by the memory misalignment issue of the encrypt functions input with zstd 1.5.0. Are you able to port the changes of dezem/AtmoXL-Titel-Installer#36 over to TinWoo by your own or do you need any help with that? |
I can confirm that on latest Tinfoil block compression is broken. I tested installing block compressed titles with Tinfoil 15.0 R5 and R7 (I wasn't able to obtain older versions). I also tested compressing them with python-zstandard 0.15.2 and even tried the really old nsz v4.0.1 release. All tests I performed resulted in a corrupted installation. @blawar Can you please look why block compression on latest Tinfoil is broken? |
|
Is tinfoil broken when installing a file block compressed with nut @nicoboss ? |
I only tested installing block compressed titles I copied over to the SD-Card so far. They always resulted in corrupted installations. I can test with nut as well if you expect this to have any different results. Diffrent bug I noticed while testing this: If you have a folder starting with # on your SD-Card Tinfoil trims it away and doesn't let you open it. |
Thanks, I just used the updated nca_writer.cpp from Atmoxl which was updated yesterday (thanks), it's working fine, just block decompressing is not working. |
You are being very generous here @nicoboss , he took the work of Adubbz, Xortroll, Hunterb, and myself, then renamed the app. Tinwoo is a ripoff/copy of Tinleaf. He isn't the author of anything. @mrdude2478 is the reason I never updated the public nca writer class to support block compression. I had started to release more open source code to this scene til he reminded of why I stopped in the first place. tldr; the reason Tinwoo doesn't support block compression is because he just copies other people's work, and cant write anything himself, and no one has posted a public implementation of block compression yet. |
Exactly! Every single open-source title installer so far seems to just be using the Adubbz NCA writer to which you added solid NSZ support. This is exactly why they all have the exact same memory misalignment issue. It would be quite easy for me to modify NCA writer to support NSZ block compression but I agree it sucks that no open-source title installer in all those years spent the few hours it takes to implement this themselves. It seems like a fun side-project to implement it so I might do it. |
I just tested with NUT. I can confirm that installing block compressed NSZ games over NUT also results in corrupted installations. This isn’t really surprising as in the end it’s still Tinfoil who is extracting and encrypting the NCZ files. |
It was a fork of the work that was on github, Awoo/Tinleaf hadn't been updated in a long time and didn't work on never versions of libnx, so I forked the installer and made quite a few changes to it. If you didn't want tinleaf to be forked, why did you even bother putting it on github? Surely that's the entire point. I remember contacting you on GBATemp about some things in the app that were broken but I got some witty reply, so that's why I ended up needing to fix some stuff myself. Still if you are salty about it, I am happy you are getting it off your chest. maybe you should update tinleaf once in a while then people wouldn't need to create forks such as atmoxl or tinwoo. Still it's good to see you are going to update tinfoil, I'll be sure to use that and dbi installer. PS DBI is the best, hopefully you can add some of the stuff that's in that app to tinfoil. Also, I am here to report an issue, with block compression - not looking for an internet fight, life is too short for that. |
Fork, copy, publicly available, tomato tomato. Tinleaf existed and did everything Tinwoo did, you just copied Tinleaf and renamed it. When you launched, you were a carbon copy of Tinleaf. Your whole schtick was "it's not blawar" even though it was my work. You stripped my name from it, and a look at your commit history since you did that shows that your "contribution" to the project was basically a plethora of readme edits. You have to understand the concept of a rude fork. For one, you copied my work without any attribution, presented it as your own which I am pretty sure is a violation of my software license. But even not withstanding that, the end result of a rude fork is the original author abandoning the project. Now you have nothing to copy from, and the scene lost from having fewer open source developers doing actual work. You suggest I should update Tinleaf. Why would I update it just for you or someone else to copy it? I also have no need to, that project was for the scene's benefit for people who only wanted to use open source software. I have Tinfoil, I do not need Tinleaf. You are why the two best title installers in the scene are closed source. This happened before you too, and it chased Adubzz / original Tinfoil author out of the scene when someone did the exact same thing to him (although that person at least did a lot more work than you did). That said, there is nothing to get off of my chest. I actually find the situation funny and you saved me from maintaining other open source projects that I did not need. It is the scene who you hurt, not me. |
|
@Blawr, didn't you clone Awoo installer - and rename in Tinleaf. Also what about this: You seem to be OK with cloning others projects but are getting salty when people clone yours??? I also didn't start tinwoo from tinleaf as far as I remember, I forked Awoo and copied some of the stuff that was in tinleaf to add to it and popped in some of my own changes. Anyway, this will be my last post on the subject, I was here to report an issue, that's been done now so hopefully you'll fix Tinfoil and the scene can benefit from it. |
I just verified, I am looking into it now. |
I added NSZ support to Goldleaf because the author refused to add it, I also changed the USB backend to use NUT instead of Quark. The upstream author was credited for his work. I modified awoo to remove the code that explicitly made it crash on SXOS for no reason, removed some furry and other inappropriate stuff, and added support for USBHDD. You copied all of that work from me. And the project was under development at the time you forked it. At no point did you approach me about adding something that was missing or talk about forking. I also credited the original authors. The difference between you and me is: I only fork when I absolutely have to, to add new features (add things of value). You fork without adding any new features for the purpose of just changing the name and stealing code. You add nothing of value. That is why your "project" is a rude fork. Had you not rudely forked Tinleaf to "create" Tinwoo, I would have continued development and Tinleaf (and the scene in general) would have much better open source software today. To be sure, it was not me you harmed, it was the scene. |
|
@blawar, I am totally agree with you. |
|
It appears that Tinfoil is writing the decrypted chunks out of order: 4000 -> 204000 Bit odd because this was verified 3 years ago, something might have changed on either end. Either way i'll fix.
|
|
@nicoboss Tinfoil 15.0 R8 fixes the blocked compression. The issue was an optimization you added, that wasn't in the reference that I programmed against. Specifically, the leaving of a block plaintext if it's compressed block is larger than the plaintext block. I actually did try to program for this optimization, but at the time I did not have anything to test against so I wrote it blind. It mostly worked, the plaintext blocks just bypassed the caching layer and resulted in the blocks being written out of order. |
Thank you so much for fixing block compression in Tinfoil. I can confirm that on latest Tinfoil v15.0 R9 block compression is working perfectly fine. I absolutely love Tinfoil and am really happy to see block compression working with it. I'm really sorry for being unclear about the plaintext rule. In hindsight I should have communicated and documented this way better. I will definitely improve the documentation about it. For me it was only clear because nsZip (the predecessor of NSZ) had this rule as well. It also didn't help that the development version from 12th October 2019 (ab6b6e4) still missed this rule and only later on 17th October 2019 (baaeaa6) I added support for it. Part of NSZ block compression support in Tinfoil likely got implemented in the few days NSZ had block compression but without this crucial rule. |
I'm busy doing a ps4 app for now:
When I get around to finishing that I might do it. First I need to read up on block compression to see how it works - it's a low priority for now. |
Looks nice @mrdude2478 . I encourage you to write new code and develop apps. Don't steal code or rename other people's projects. If you want to atone for what you did, write code to add nsz block compression to tinwoo. |
|
Just for information. DKP now has libzstd 1.4.4, but even the next release (1.4.5) has major decompression speed improvement (15-50%, especially on ARM SoC). |
I did not notice that @duckbill007, thx. I am skeptical that we can fully utilize these improvements due to the switch's trash hardware though. The bottleneck appears to be the disk IO controller, USB and the network stack. Even with USB 3 enabled, USB NIC's have terrible transfer speeds. And Disk IO seems to top out around 70-80MB/sec peak burst with the very fastest SD cards. I had already updated Tinfoil to the latest 1.5.X when I fixed the bug here (I stopped using DKP packages a long time ago, due to their stability issues and their hostile attitude towards historical DKP version backups). |
|
With the release of NSZ v4.2.0 I consider this issue to be resolved. All modern non-abandoned title-installers fixed the memory alignment issue caused by the use of block splitting in zstd 1.5.0 and later. If you know any other popular title installers that need to fix this issue, please create an issue on their GitHub repository and ping me so I can look into it. You can also link them here as I will keep monitor this issue despite closing it. Regarding the implementation of block compression support for open-source title-installers please continue the discussion in #42 |
I use nsz 4.1 to compress my NSP file then upload to my webserver.
then I try to install it with Tinfoil install is fine but when run it, I got corrupt error.
But when I use dbi to install over network then it run without problem
the command I use to compress is :
nsz -l 22 -V -m 1 -o -C
Ps: I use NSC_builder is work for both dbi or Tinfoil.
The text was updated successfully, but these errors were encountered: