Repacked .hpk does not work in Surviving Mars (loads forever)

[zocker-160]

When extracting armstrong.hpk and packing it again, the game refuses to load into the menu.

Looking at your file format specifications, it seems like something weird is going on:

The original file does have the data of the first file right after the one BPUL header starting at 0x24.
The newly created file though does contain two BPUL headers and then the data for the first file, which then starts later (0x48 I guess?)

here the first 200 bytes of each file:

original

4250554C2400000001000000FFFFFFFF000000000000000001000000CA267A0448050000444B4946000020005650383080073804905F0100010000000000000000000080C80D0000000000000000000030B7019D012A80073804004708858588858488020200061604F70681649F6BDB9B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B

repacked

4250554C2400000001000000FF0000000000000000000000010000006EB07204480500004250554C2400000001000000FFFFFFFF0000000000000000010000008E763000580000004C5A34200E000000000002001000000072657475726E20313031303738324C5A34208E250A000000020024000000D36200003EB60000BA130100C06F010045D40100FF16734200000014000000456E7469746965732F41636164656D792E656E740D0A4D35000000181F000042446F6F7223006FF7310000002023000050466C

Also I noticed that the first header in the repacked archive does have 0x000000FF instead of 0xFFFFFFFF.

Any idea where this difference comes from and if that could be the cause of the problem?

EDIT:
when running it from terminal, the game seems to crash at loading the DLCs, which this file is part of.

.local\share\Steam\steamapps\common\Surviving Mars\CommonLua\Dlc.lua(702): field LoadDlcs
[string ""](-1):  <>
        --- end of stack

EDIT2:
I also noticed, that there is a size difference between the original and the repacked file (the bak is the original one):

[nickelc]

Also I noticed that the first header in the repacked archive does have 0x000000FF instead of 0xFFFFFFFF.

What happens when you change these four bytes to 0xFFFFFFFF?

I also noticed, that there is a size difference between the original and the repacked file (the bak is the original one)

The different file size comes from residual data in the original file. The data is not used and is some leftover from how Haemimont edits the files instead of re-creating the files.

The command hpk print --header-only armstrong.hpk.bak should show the residual info from the header and looks like this for armstrong.hpk.

  fragments_residual_offset: 0x0
  fragments_residual_count: 0

[zocker-160]

original file:

$ ./hpk print --header-only armstrong.hpk.bak 
reading file: armstrong.hpk.bak
header:
  data_offset: 0x24
  fragments_residual_offset: 0x0
  fragments_residual_count: 0
  fragments_per_file: 1
  fragments_filesystem_offset: 0x47A26CA
  fragments_filesystem_length: 1352
filesystem entries: 169

generated wtih hpk:

$ ./hpk print --header-only armstrong.hpk
reading file: armstrong.hpk
header:
  data_offset: 0x24
  fragments_residual_offset: 0x0
  fragments_residual_count: 0
  fragments_per_file: 1
  fragments_filesystem_offset: 0x472B06E
  fragments_filesystem_length: 1352
filesystem entries: 169

What happens when you change these four bytes to 0xFFFFFFFF?

Sadly no difference, the game still fails to load.

[nickelc]

Did you extract the file with --fix-lua-files?
Because then you have to use --cripple-lua-files on creation.

--fix-lua-files makes it possible to decompile the lua files with unluac or luadec.

[zocker-160]

Do I didn't, all I did was use hpk extract and hpk create with no additional parameters or changes to the files.

[nickelc]

I could reproduce it with DLC/dde.hpk and I think I found the problem.
It seems SM doesn't like compressed lua files in the DLCs.

With the parameter --extensions "" no files are compressed and SM should load.
hpk create armstrong DLC/armstrong.hpk --extensions ""

The newly created file though does contain two BPUL headers and then the data for the first file, which then starts later (0x48 I guess?)

The armstrong.hpk contains other hpk files and the re-creation writes the files in a different order.

[zocker-160]

With the parameter --extensions "" no files are compressed and SM should load.
hpk create armstrong DLC/armstrong.hpk --extensions ""

that works! 🎉

thanks for the help, might be good to document that in the readme. Should I close issue then?

[zocker-160]

OT: would you be interested in a PKGBUILD generating a DEB for Debian systems? Would make installation a single click.

[nickelc]

thanks for the help, might be good to document that in the readme. Should I close issue then?

I'll add a parameter --no-compression and mention the problem for SM DLCs in the readme.

OT: would you be interested in a PKGBUILD generating a DEB for Debian systems? Would make installation a single click.

sure, PR is welcome.