Issue unpacking and repacking chunk0.dat

[cstBipBop]

Hey apparently unpacking and repacking the chunk0.dat game files with the current QAR dictionary project results in an infinite load screen according to other modders. Supposedly doesn't occur with QARTool so I'm curious if you have any idea what the issue is. Brought up by tinmantex and investigated by others.

"so theres an issue unpacking then repacking chunk0 with gzstool with a current dictionary? simply doing that with no modifications causes a hang on load. doing the same thing with an empty dictionary and its fine.

BobDoleOwndU - Today at 12:30 AM
Huh. Interesting. I knew that repacking caused an infinite hanging screen. But I didn't know that it doesn't happen with an empty dictionary. QAR Tool doesn't have that issue though

Joey35233 - Today at 2:32 PM
@HeartlessSeph Have you gotten anywhere with bone weights?

hoopyboop - Today at 3:27 PM
@TinManTex that's kind of bizarre. Maybe an improper definition being used somewhere? Not sure why that'd be an issue though, since hash collisions should mean it shouldn't matter. Though if QAR Tool doesn't share the issue I guess it is some weird bug. Maybe something to report to Atvaark to see if he knows what's up with it.

tinmantex - Today at 6:56 PM
So (after a bunch of divide and conquer) the offending entry is /Assets/tpptest/pack/location/empty/empty
which matches legit fpk/d which is a mission pack used for init, which seems to match up with where if fails on loading
so i'm guessing gzstool is packing that with a different hash for some reason?
confirmed it by keeping the dictionary and removing just that entry and also an dictionary with just that entry.

BobDoleOwndU - Today at 6:59 PM
According to my debug version of GzsTool, that name's hash is 1978c01cfffc2. Do you know if that's supposed to be what it is?

hoopyboop - Today at 7:07 PM
Well mgsvPathHasher.exe is reporting:
/Assets/tpptest/pack/location/empty/empty 978c01cfffc2

So I'm not sure what the issue is.

BobDoleOwndU - Today at 7:08 PM
unknown's path hasher uses the same algorithm as GzsTool. So they both come up with the same result(edited)

hoopyboop - Today at 7:09 PM
So an issue with the lead 13th digit is what you're saying?
As I mentioned before at some point, Sergeanur said he isn't interested in modding MGSV at this time, but having his input would be great on this matter.

BobDoleOwndU - Today at 7:10 PM
Nono, unknown added functionality to strip the extra digit to make it easier to search for file names in a hex editor. Because the extra digit is added to the extension hex part of the name, unless you know what the extension digits are, you can't search for that part of the name

hoopyboop - Today at 7:11 PM
Oh gotcha
@unknown I don't write C-type programs. Where exactly did you do that (for stripping digits)? Or does anyone else know? Unless I'm reading that wrong anyway.
unknown - Today at 7:15 PM
idk
https://github.com/unknown321/mgsv_path_hasher/blob/master/main.cs#L76 here I think
GitHub
unknown321/mgsv_path_hasher
Contribute to mgsv_path_hasher development by creating an account on GitHub.

BobDoleOwndU - Today at 7:16 PM
It's this part here

ulong testStrCode32 = GetStrCode32(text);
hash = testStrCode32.ToString("x");
if (hash.Length < 12)
hash = hash.PadLeft(12,'0');
hash = hash.Substring(hash.Length-12,12);
unknown - Today at 7:16 PM
ye
I don't remember the exact reason why I did it though

BobDoleOwndU - Today at 7:17 PM
Makes it easier to search for hashes in a hex editor
unknown - Today at 7:17 PM
no
there was an issue and I don't remember
anyway, sergeanur has the definitive hashing function

BobDoleOwndU - Today at 7:18 PM
Well, even if that wasn't the orginal intention, it does make it easier to search for hashes 😛
Do we know if there's a difference between sergeanur and Atvaark's hashing functions (other than QAR Tool being written in C++ I think)? They should be the same I would think. I do think there's a difference in the way the tools pack things though(edited)

hoopyboop - Today at 7:28 PM
idk, at any rate I'll rebuild unknown's main.cs file to see what results from it though.
Did a FOR /R %A (*.cs) DO (csc "%A") loop on every folder on my HDD C:\ folder until I interrupted it. RIP every program.(edited)
Figured out why it wasn't running proper. Needed to do "csc /r:CityHash.dll main.cs" according to unknown's GitHub page.
Rebuilt with my edits. Don't know what issue he was referring to. Seems to work fine. Will try with the problem entry.
Got
/Assets/tpptest/pack/location/empty/empty 1978c01cfffc2
So in other words I have no idea. Might be an issue with GZsTools algorithm like others have mentioned since QAR Tool apparently has no issue with chunk0.dat.(edited)
My changes (aside from changing default hash reversal) were just this:
ulong testStrCode32 = GetStrCode32(text);
hash = testStrCode32.ToString("x");
if (hash.Length < 13)
hash = hash.PadLeft(13,'0');
hash = hash.Substring(hash.Length-13,13);
Perhaps running Sergeanur's QAR Tool with a blank dictionary will reveal the issue.

hoopyboop - Today at 8:02 PM
QAR_Tool's .inf file lists those entries as
522d978c01cfffc2|chunk0\1978c01cfffc2.fpk key=0 version=0 compressed=1
ed55978c01cfffc2|chunk0\1978c01cfffc2.fpkd key=0 version=0 compressed=1
Which goes back to: "hell if I know"
Might be a major bug with GZsTool. I'll contact Atvaark and see what he knows if no one else has or wants to.(edited)"

[Atvaark]

Thanks for the report.

Some files (e.g. foxfs.dat) use a flag encoded inside their hashes.
This flag is missing in /Assets/tpptest/pack/location/empty/empty due to a bug somewhere here in the code:

GzsTool/GzsTool.Core/Utility/Hashing.cs

Line 178 in 8a0656c

if (text.StartsWith(assetsConstant))

I could maybe include the flag as an XML attribute or fix the check in Hashing.cs, but I need to do some more testing.

QAR Tool doesn't have this issue because it requires pre-calculated hashes (and thus doesn't support adding new files to archives.)

[Atvaark]

Please test if release v0.5.3 fixes this issue.

[Atvaark]

Did anyone test the the new version yet?
@cstBipBop @TinManTex

[cstBipBop]

Sorry I only see replies on GitHub when looking at the actual page (not sure why it doesn't give a notification on the site instead of an email). Will test it tonight and let you know.

[TinManTex]

Yes works well, thank you.
Still a bit worrying how arbitrary it seems.