Fixed Guild Bug, enabling Win <-> Linux transfer

[Magiwarriorx]

1. Fixed the Guild bug. Guild data is stored in the level.sav as a raw byte array; member player GUIDs are stored as little endian 32 bit values within that array i.e.:
   byte 4, byte 3, byte 2, byte 1, byte 8, byte 7, byte 6, byte 5, byte 12, byte 11, etc..
   Most everywhere else, GUIDs are hex, thus each byte corresponds to 2 characters of the GUID. Additionally, the JSON tool converts these bytes to decimal. Bodged the code to convert players' hex GUIDs to decimal, order them little endian, then search/replace the old GUID in every guilds' byte array.

   The "give your friend ownership of the Guild" workaround works when moving from a Windows host to a Windows server, or Linux to Linux, but Windows and Linux generate different GUIDs for all players. As such, it was impossible to preserve Guild progress when moving from Windows host to Linux server. This change fixes that by directly changing the guild membership GUIDs.

2. Added new argv argument for replacing non-000....00001 IDs (i.e. when replacing someone's Windows GUID with their Linux one or vice versa). If you're just looking to use the script for its original function, leave this arg out and it'll default back to the 000....0001 value.

I apologize for my spaghetti, Python is not my forte. Tested confirmed working moving a save from Windows co-op to G-Portal (Linux).

[Magiwarriorx]

This is experimental, might brick your save, burn your house down, and send Mercury into retrograde.

My code has yet to be reviewed by anyone that has actual Python knowledge. Use at your own risk, and make plenty of backups. I dont expect this to work on any patch past PC v0.1.2.0, but hey maybe it will.

[Magiwarriorx]

Note this does not fix the Pal bug; all Pals must be dropped and picked back up to continue working at the base.

[mel0n]

I saw the same extra occurrence of the 00000...00001 GUID in a hex editor, and also just ran the hex script in the latest commit which showed up at this location:
[root][properties][worldSaveData][Struct][value][Struct][GroupSaveDataMap][Map][value][35][value][Struct][Struct][RawData][Array][value][Base][Byte]

I am a C/C# guy so python is not my thing either but your PR changes mimicked what I was just working on (adding extra GUID's since I am converting from gamepass -> co-op on steam -> dedicated linux server) however when running your script it seems to make all the modifications, yet when I load in to the game I get the new character screen, almost like the game is rejecting something. I have run through it a few times from fresh and get the same result. If I run the original script on the same save files, I load into the map just fine as myself (Host) but with the guild glitch which is going to be tricky to try and fix since we didnt use a guild and every1 in the server's GUID is changing technically since I came from gamepass (I.e. impossible to pass the guild ownership off to anybody since all of the players have the guild glitch, and adding a new steam player to the mix in co-op mode.... well they cant be added to a guild if nobody has access lol).

[Magiwarriorx]

My background is C/C++; we think alike I guess :P

Very interesting. I didn't encounter anything like that. Since you're the co-op host, were you running the script without the optional arg?

[carsong24]

Can confirm this works. Also the ability to migrate any guid to another was very helpful. Hopefully the saves stay intact for future updates.

[mel0n]

I tried it without the arguement at first, but I also just tried the following:
python fix-host-save.py C:\Users\mel0n\.cargo\bin\uesave.exe R:\Projects\palworld-save-conversion\linux-pull-test-2 812XXXXX000000000000000000000000 00000000000000000000000000000001

However still got the new character screen when I loaded it up. I am wondering if the byte offsets could be different for gamepass saves vs steam native saves. I will have to do some further digging on this. I have also checked the files after the script runs and it's not really making any sense because some of the values dont change for whatever reason. When I check the Hex decode the 00000....00001 section I found before is still present.

[Magiwarriorx]

My understanding is that character data (i.e. appearance, location, whether to display the "make a new character" screen) is tied to the .sav's inside the Players folder. The only change I made to how the script handles those was replacing the hardcoded 00000000000000000000000000000001 with that last arg; everything else affects only the Level.sav file, where guild data is stored.

The fact the original script works for you rules out a lot of options for Game Pass differences. However, since the player data is in Players, and guild in Level.sav, try taking the Level.sav generated by this PR and replace the Level.sav generated by the original script, leaving the Players folder untouched.

[mel0n]

That was a good idea, however with the known-good player folder, and the level.sav produced by your modified script, it sent me to the character creation screen. I will have to diff the level.sav files and comb through them in the morning. Looking at your script changes I dont quite understand why its doing this as not much changed from the original aside from the edit to the byte array. I am considering making a fresh steam co-op save to look for any specific differences that might stand out in that level.sav file when compared to the gamepass produced one. Gamepass is 2 patches behind at this point too iirc.

[xNul]

Thanks for the PR! I'll test it out later to confirm it works before merging. Strangely, I ended up trying something almost exactly like this, but it wouldn't work. Maybe I didn't get the signedness correct?

host_guid_bytes = list(bytes(host_guid.encode('utf-8')))
host_old_guid_bytes = list(bytes('00000000000000000000000000000001'.encode('utf-8')))

# Replace an instance of the 00001 GUID which corresponds to guild data with the former host's actual GUID.
group_save_data_map_len = len(level_json['root']['properties']['worldSaveData']['Struct']['value']['Struct']['GroupSaveDataMap']['Map']['value'])
for i in range(group_save_data_map_len):
    level_grouptype = level_json['root']['properties']['worldSaveData']['Struct']['value']['Struct']['GroupSaveDataMap']['Map']['value'][i]['value']['Struct']['Struct']['GroupType']['Enum']['value']
    if level_grouptype == 'EPalGroupType::Guild':
        level_bytes = level_json['root']['properties']['worldSaveData']['Struct']['value']['Struct']['GroupSaveDataMap']['Map']['value'][i]['value']['Struct']['Struct']['RawData']['Array']['value']['Base']['Byte']['Byte']
        if level_bytes[20:52] == host_old_guid_bytes:
            level_json['root']['properties']['worldSaveData']['Struct']['value']['Struct']['GroupSaveDataMap']['Map']['value'][i]['value']['Struct']['Struct']['RawData']['Array']['value']['Base']['Byte']['Byte'][20:52] = host_guid_bytes

[Spaenny]

I just tested this fix, and it worked flawlessly. I was able to recover full chars and the guild with it.

[LockonZero]

Same here, used this and everything seems to work as intended. My char works, not sure if the host works yet since he is not online yet, but there should be no problem

[danxvv]

Works for me and my friends; we all remain in the same guild with all our pals.

[maxiking913]

works like a charm thanks!

[Magiwarriorx]

@xNul No problem! Thank you for the original script.

The guild byte array for my save was about 18KB, with the membership stored at the end. Had to search the full thing for replacement; I don't think fixed offsets will work.

Additionally, they are storing it as little endian 32-bit numerical values instead of strings. So:

00000000000000000000000000000001
would be in the array in the JSON as:
[0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0]

For a non-host example:

548DDBCA000000000000000000000000
becomes:
[202 219 141 84 0 0 0 0 0 0 0 0 0 0 0 0]

202 = 0xCA, 219 = 0xDB, 141 = 0x8D, 84 = 0x54; the first four bytes of the GUID in reverse order.

Bizarrely, the player/guild names are stored as in-order strings shortly after the corresponding GUID. Thank goodness they are, or I'd have never found this.

EDIT: There is a 00000000000000000000000000000001 string at the start of the array, but truthfully I don't know what it's for, and I didn't edit it.

EDIT 2: That string seems to be tied to the creator, not the owner. A file that had guild ownership transferred off-host still has that string there. However it isn't just creator's GUID; a new player created a second guild on the save, and its equivalent string is:

'000000000000000000000000 + ��±�', with the last 4 bytes corresponding to the creator's GUID in the same pattern as above (slightly fuzzed for their privacy).

[Ulryck57000]

Hello,
After script running we only need to upload the new 548DDBCA000000000000000000000000.sav or we need to re-upload all the save in the repertories 0?

[Magiwarriorx]

@Ulryck57000 The entire save folder. Some changes are made outside the Players folder.

[mel0n]

@xNul No problem! Thank you for the original script.

The guild byte array for my save was about 18KB, with the membership stored at the end. Had to search the full thing for replacement; I don't think fixed offsets will work.

Additionally, they are storing it as little endian 32-bit numerical values instead of strings. So:

00000000000000000000000000000001 would be in the array in the JSON as: [0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0]

For a non-host example:

548DDBCA000000000000000000000000 becomes: [202 219 141 84 0 0 0 0 0 0 0 0 0 0 0 0]

202 = 0xCA, 219 = 0xDB, 141 = 0x8D, 84 = 0x54; the first four bytes of the GUID in reverse order.

Bizarrely, the player/guild names are stored as in-order strings shortly after the corresponding GUID. Thank goodness they are, or I'd have never found this.

EDIT: There is a 00000000000000000000000000000001 string at the start of the array, but truthfully I don't know what it's for, and I didn't edit it.

EDIT 2: That string seems to be tied to the creator, not the owner. A file that had guild ownership transferred off-host still has that string there. However it isn't just creator's GUID; a new player created a second guild on the save, and its equivalent string is:

'000000000000000000000000 + ��±�', with the last 4 bytes corresponding to the creator's GUID in the same pattern as above (slightly fuzzed for their privacy).

Yes that second string at the start of the array was the one I had ben looking at. I had noticed that the guilds can be added to the file out of order if guild changes were made by players in different guilds as well. I think your method of searching may be the best approach though with host could lead to that second occurrence in the binary blob, or even potentially false finds due to the nature of it being all 0's followed by a 1. A pattern of 0's occurs a lot in the file, you just have to be unlucky enough to have a 1 follow one of those somewhere for a false positive.

I believe the file size difference and potential for additional/lacking information in the gamepass save file on the older build of the client led to corruption of my levels file which was causing that new character screen?

I will be testing this on non-host users this afternoon when a buddy gets off work, so will see if that search method works properly and fixes the guild for that player as well.

[xNul]

@Magiwarriorx ahh that makes sense! I was able to test the code and it's working on my end as well! I'll go ahead and merge it. Thanks for the help!

[Ricool06]

Well done @Magiwarriorx and @xNul. I should have checked this thread much sooner!

[mel0n]

Just a quick update, I was able to fix the guild associations of 1 non-host player who I had manually migrated/converted via json previously (I only ran the new guild modification code on level.sav) and he can now touch all of his old stuff. I ran the full updated script on another player who had not been converted yet and it works properly, he only has the pal bug.

After these successes I tried running only the guild portion of the code on the level.sav of a previously migrated host with just the guild glitch present, and it broke like before. So then I stepped back and tried on a fresh/unmodified save straight from gamepass, running the entire thing on just the host again, and it broke/character screen again.

It seems this method can be hit or miss for host depending on the save file, but definitely works perfectly for non-host player migration even from gamepass -> steam dedicated server.

Thanks for the work everyone, much appreciated!

[xNul]

@mel0n that's important information, thanks for writing it. I was able to migrate my host from a Steam co-op save and it worked perfectly, so it's able to work on hosts too, but maybe not gamepass hosts? I'm not sure because I haven't done any testing at all on gamepass saves.

The best and safest solution is being able to understand and parse every byte of the guild data so maybe we can make some headway toward that. It would be nice if we had someone capable enough to reverse engineer the game itself to help us figure that out because if not, then it will just be trial and error testing every byte.

[aksdad]

If I leave in the guild related code, I have to create a new character and lose pretty much everything. Commenting it out and having another person from co-op give give me the guild master rank again works as a workaround. Was there any patches that might have moved the bytes around in the save file?

[rub3z]

@Magiwarriorx @mel0n Do you know if I can use any of this information help my friend be able to log back in and play in the server? He's encountering the infinite loading screen bug because someone left the guild while he was offline. Will modifying the guild information in some way allow him to get in and keep his progress?

What happened was, I migrated my server and all of its save data to a new machine for reasons (needed more RAM). Everything worked fine: myself and a few other guys were able to hop in, and all of our pals and progress were intact (the only thing reverted was map exploration, and the tutorials were appearing again).

Unfortunately, two out of our 4 guild members decided to start horsing around and leave the guild to start a new one while our fourth was offline, so now he has this infinite loading screen bug (none of us knew about the issue at the time).

What can I do to help him get back in?

[bodnjenie14]

this fix doesn't work - if u have transferred/mirgrated your save from Microsoft/gamepass to steam previously

[mel0n]

If I leave in the guild related code, I have to create a new character and lose pretty much everything. Commenting it out and having another person from co-op give give me the guild master rank again works as a workaround. Was there any patches that might have moved the bytes around in the save file?

@aksdad Are you the host? If so, it sounds like you had a similar experience to myself including the work around. Were you on Steam or Gamepass when the co-op world was originally created? As far as patches go, I do not think so since the new code is simply grabbing the byte array which represents the guild membership list, then stepping through each entry in the list which is of fixed size (GUID in decimal), so that size/offset should not have changed.

@rub3z Unfortunately I do not know how to fix this, and its probably related to the issues with removing/changing ownership of a guild to an offline member. This code may be able to help if you know the GUID of the previous owner of the guild, and the GUID of the broken player. You could try running just the guild fix portion of code added here, replacing the old owner GUID with the broken player's GUID. Aside from that, rolling back to a backup save would probably be the best solution. I am frequently backing up my saves (about every 20 mins) on my server since I moved the entire thing onto a ramdisk due to the insane I/O pressure the game places on the server among other save/game destroying glitches such as this.

@bodnjenie14 please see my other comments on this as I migrated from gamepass and it does work on all players except host. For the host, you will need to have a fresh steam player join, add them to the host's guild as in co-op mode, make them the owner, then have the host leave the guild, convert your host save without the guild fix code (comment that section out), then load the save onto your dedi server, join the steam player's guild, then have them give you/host ownership back. This was the only way I could get my host to work again and it has to be done first when converting from co-op to dedi. Once the host is migrated/fixed, you can migrate the other co-op players with this new script and it seems to fix their guild associations properly but they will still have the pal glitch r equiring dropping all palls and picking back up.

[aksdad]

@mel0n I am the host, and this is on the steam version of the game.
Hmm, your explanation makes sense. I’ll try comparing the byte array from the original and the save modified by the dedicated server.

[bodnjenie14]

But this pr states it fixes guild bug however it does not you still need to transfer ownership to a other user if you have previously migrated your save - how ever i got mine to work

by using the guild fix some one crossed out on the readme this pr needs correcting for allready converted saves

also having to do this to fix character creation bug

After character creation on the dedicated server, it overrides your player save. Take that new file, grab your backed up old local file, put it into the folder where you're running your script. Rerun script. and only copy the player file back in

example"\70FF2F994A4BECC8AF1380B6D0827DEB\Players"

If I leave in the guild related code, I have to create a new character and lose pretty much everything. Commenting it out and having another person from co-op give give me the guild master rank again works as a workaround. Was there any patches that might have moved the bytes around in the save file?

@aksdad Are you the host? If so, it sounds like you had a similar experience to myself including the work around. Were you on Steam or Gamepass when the co-op world was originally created? As far as patches go, I do not think so since the new code is simply grabbing the byte array which represents the guild membership list, then stepping through each entry in the list which is of fixed size (GUID in decimal), so that size/offset should not have changed.

@rub3z Unfortunately I do not know how to fix this, and its probably related to the issues with removing/changing ownership of a guild to an offline member. This code may be able to help if you know the GUID of the previous owner of the guild, and the GUID of the broken player. You could try running just the guild fix portion of code added here, replacing the old owner GUID with the broken player's GUID. Aside from that, rolling back to a backup save would probably be the best solution. I am frequently backing up my saves (about every 20 mins) on my server since I moved the entire thing onto a ramdisk due to the insane I/O pressure the game places on the server among other save/game destroying glitches such as this.

@bodnjenie14 please see my other comments on this as I migrated from gamepass and it does work on all players except host. For the host, you will need to have a fresh steam player join, add them to the host's guild as in co-op mode, make them the owner, then have the host leave the guild, convert your host save without the guild fix code (comment that section out), then load the save onto your dedi server, join the steam player's guild, then have them give you/host ownership back. This was the only way I could get my host to work again and it has to be done first when converting from co-op to dedi. Once the host is migrated/fixed, you can migrate the other co-op players with this new script and it seems to fix their guild associations properly but they will still have the pal glitch r equiring dropping all palls and picking back up.

[mcam435]

Not sure if maybe this isn't fixed, or if I've found an edge case/done something wrong. So we've had an issue after transferring our saves where none of us show up as being in the guild. We've tried disbanding the guild and starting a new one. We're not able to see each other on the mini map either.
Interestingly when one of the players created their second character they game it a different name, which is what they show up as in the guild list, but in the world their character name over their head shows their old name.

[Ricool06]

@mel0n that's important information, thanks for writing it. I was able to migrate my host from a Steam co-op save and it worked perfectly, so it's able to work on hosts too, but maybe not gamepass hosts? I'm not sure because I haven't done any testing at all on gamepass saves.

The best and safest solution is being able to understand and parse every byte of the guild data so maybe we can make some headway toward that. It would be nice if we had someone capable enough to reverse engineer the game itself to help us figure that out because if not, then it will just be trial and error testing every byte.

I think the Guild information in Level.sav is an Unreal TMap type from my investigation. UE5 is open source, so deserialising that into a TMap might yield some information.

level_json["root"]["properties"]["worldSaveData"]["Struct"]["value"]["Struct"]["GroupSaveDataMap"]

Should be a TMap.

[teepisutt]

But this pr states it fixes guild bug however it does not you still need to transfer ownership to a other user if you have previously migrated your save - how ever i got mine to work

by using the guild fix some one crossed out on the readme this pr needs correcting for allready converted saves

also having to do this to fix character creation bug

After character creation on the dedicated server, it overrides your player save. Take that new file, grab your backed up old local file, put it into the folder where you're running your script. Rerun script. and only copy the player file back in

example"\70FF2F994A4BECC8AF1380B6D0827DEB\Players"

@bodnjenie14 Thanks, By did what you said. Rerun the script using "ONLY" New dedi player save files .sav but this time put it in old Solo back-up files together with where 00001.sav are. And replace "ONLY" the new compiled .sav back to the dedi \players folder did bring all progress and carried Pals back. However, I lose base as it is said "belong to other guild".

[bodnjenie14]

But this pr states it fixes guild bug however it does not you still need to transfer ownership to a other user if you have previously migrated your save - how ever i got mine to work
by using the guild fix some one crossed out on the readme this pr needs correcting for allready converted saves
also having to do this to fix character creation bug
After character creation on the dedicated server, it overrides your player save. Take that new file, grab your backed up old local file, put it into the folder where you're running your script. Rerun script. and only copy the player file back in
example"\70FF2F994A4BECC8AF1380B6D0827DEB\Players"

@bodnjenie14 Thanks, By did what you said. Rerun the script using "ONLY" New dedi player save files .sav but this time put it in old Solo back-up files together with where 00001.sav are. And replace "ONLY" the new compiled .sav back to the dedi \players folder did bring all progress and carried Pals back. However, I lose base as it is said "belong to other guild".

to fix this issue

Open CO-OP server again.
Ask friend to join
Have friend join your guild
Pass ownership onto him
You leave the guild (only him will be left on the guild)
Leave & close the server
Redo ALL this crap again.
Pray that it works
Boot up the server again and have your friend join.
Have him invite you to the guild again
Change ownership back to you and you can kick him from the guild if you want.

[gahhbors]

Guys what's the workaround for my bug, i tried to understood but i'm lost.
I transfered the game and players normally, but who was in another 2 guilds after the transfer is now without a guild, if they goes to their base says "you're not in this guild".

[Simpuhl]

I moved mine from Windows to Linux, everything is good except the guild and left click bug.

My base no longer belongs to me. I am the only player so far. Any help?

[JimLin00]

I found a way to fix anyone who can't pick up pals and can't see guild member on map for temporary.
Just wait until the up right coner indicate that it was saving game and leave the game immediately.

[Lynxaa]

Just wanted to contribute my reversals so far.

I made a script to find all "EPalGroupType::Guild" entries inside of "GroupSaveDataMap", and then I dumped the "RawData" bytes into a binary file so I could open it in a hex editor.

There appears to be a few strings present in each guild raw data, at the top, there's a GUID, presumably this is the guild owner.
At the end of the file, is an array of names, these are the guild members.

An example of a "broken" guild, i.e., this player experienced the bug where you cannot pick up pals after transferring saves:

An example of the original guild:

I'll continue to work on a script that will hopefully be able to display this information to you and give you the option to delete certain guilds, or possibly, automatically detect these "corrupt" guilds.

I'm not sure if this will actually fix the issue of not being able to pick up pals, or if it'll prevent the server from automatically creating new guilds.

EDIT:

It appears the GUID at the start of the RawData is the owners old GUID, so another thing that does need to be fixed.

EDIT2:

After using the library created by @cheahjs (https://github.com/cheahjs/palworld-save-tools), the RawData entry (along with a lot of other structures) are properly parsed, which allows you to very easily fix incorrect data or identify problems.

An example of an empty guild:

{
  "group_type": "EPalGroupType::Guild",
  "group_id": "57f30c05-4478-c9e3-76d0-9392496d05de",
  "group_name": "51E92883000000000000000000000000",
  "individual_character_handle_ids": [
    {
      "guid": "51e92883-0000-0000-0000-000000000000",
      "instance_id": "10815989-4f04-448b-af62-1ba71feb037f"
    }
  ],
  "org_type": 0,
  "base_ids": [],
  "base_camp_level": 1,
  "map_object_instance_ids_base_camp_points": [],
  "guild_name": "Unnamed Guild",
  "admin_player_uid": "51e92883-0000-0000-0000-000000000000",
  "players": [
    {
      "player_uid": "51e92883-0000-0000-0000-000000000000",
      "player_info": {
        "last_online_real_time": 3604065260000,
        "player_name": "Lynxaa"
      }
    }
  ]
}

If you use the repo by @cheahjs to dump your level.sav and search for your old guid (after running the fix tool), you'll see many occurrences of it still present, it's likely that all these instances that remain unfixed are cause for a fair few bugs, likely including this guild bug, resulting in not being able to pick up pals.

[j0lle]

I moved mine from Windows to Linux, everything is good except the guild and left click bug.

My base no longer belongs to me. I am the only player so far. Any help?

I think you'll need another player and transfer the guild to them. then run the fix again.
I did it today and was successful with a win co-op --> linux dedicated transfer.

I only needed to fix the host character, other players were fine.
the map and tutorials have been reset, and I needed to do the transfer/leave/rejoin guild trick, as well as the pick-up-and-drop trick to be able to interact with previously caught pals

[Simpuhl]

I moved mine from Windows to Linux, everything is good except the guild and left click bug.
My base no longer belongs to me. I am the only player so far. Any help?

I think you'll need another player and transfer the guild to them. then run the fix again. I did it today and was successful with a win co-op --> linux dedicated transfer.

I only needed to fix the host character, other players were fine. the map and tutorials have been reset, and I needed to do the transfer/leave/rejoin guild trick, as well as the pick-up-and-drop trick to be able to interact with previously caught pals

Dang, I don't know anybody else. I wish you didn't start off having a "Guild". I didn't even make one, it just creates one for you that you can't leave.

[Lynxaa]

Based upon my previous post, it appears if you remove the "empty" guilds for players that are already in a guild, the lifting bug will be fixed.

i.e., I'm in a guild with 5 other people, but inside of GroupSaveDataMap there's a guild with just myself and it's level 1, deleting this guild fixed the issue for me.

I've created a script to do this for every player, I'll clean it up and post it here once my friends play more and we can confirm there's no side effects.

[L00NEYB0T]

Open CO-OP server again. Ask friend to join Have friend join your guild Pass ownership onto him You leave the guild (only him will be left on the guild) Leave & close the server Redo ALL this crap again. Pray that it works Boot up the server again and have your friend join. Have him invite you to the guild again Change ownership back to you and you can kick him from the guild if you want.

@bodnjenie14 Can you explain what you meant by this“ convert your host save without the guild fix(comment that section out)” did everything had friend take over guild but when they join guild is no longer under them