Support for XCOM2 War of the Chosen #30
Comments
|
note, the 24th above might be the mission name or something. |
|
Hi, please see #21 for details on why XCOM2 isn't supported. It's not so much a problem with understanding the format as that it can't be unambiguously parsed without also having information from the script packages and that is a big job to implement. I am not actively working on this project other than bugfixes, so that isn't something I have any plans to take on myself. I would definitely accept PRs that add full xcom2 support, though. |
|
Currently I have no custom mods installed, so that's fine by me. I would appreciate if you could help or point me in the right direction as to the actual header & inlined blocks structures. Also, regarding the scripts mentioned on your reply, can you elaborate further on those ? |
|
Can anyone help identify the yellow fields ? |
|
The scripts I mentioned are the unrealscript packages for the game and for any mods that have custom code. The big one is xcomgame.upk, and you can build it yourself by installing the xcom2 mod tools SDK. A big chunk of the save file is serialized versions of these unrealscript objects. You don't necessarily need to understand all the fields in the save, as long as you know how big they are you can just read that many bytes and treat them as some unknown byte values that need to be put back in exactly the same way when regenerating the save. Even for EW many of the fields are marked 'unknown' and the values are just passed through as-is. It does make it harder to use if those bytes represent something that you might want to change in the save, though. I don't remember exactly if we even know the size of all these unknown structs to be able to treat them as opaque, it's been many years since I looked into this. Some helpful tools for investigating unreal files are UE Explorer (https://github.com/UE-Explorer/UE-Explorer) which you can use to dump info about a script package and these can be helpful to try to interpret what the serialized form of the data will look like. There are also some useful wiki pages on nexusmods (https://wiki.nexusmods.com/index.php/Modding_XCOM:EU_2012) although that's EW specific XCOM2 uses the same engine and the formats are only slightly different. |
|
thanks for reply with info. |
|
weird, neither UE Explorer or UE viewer is able to view xcomgame.upk file. Not sure if it's because somehow compressed, or a change in serialization format made during move to XCOM2 (i've read Unreal engine is 3.5, compared to version 3 on enemy unknown). If i try to read enemy unknown files, it's goes little bit further but also "dies" Are you able to view XCOM2 files on your pc ? |
|
If you're using the ones from the shipping game you need to decompress them first, there's a decompress tool here: https://www.gildor.org/downloads If you build the packages from the mod tools SDK they are uncompressed by default. |
Thanks for that, you were right, it was compressed. The tool however only seems to work on XComGame.upk file for Enemy Within & Enemy unknown & not for XCOM2 :( Will attempt to contact Gildor author & see if he can help with new's format... |
|
Is the decompression failing or ue explorer failing to read the decompressed file? I was able to use ue explorer on the uncompressed ones I built myself. If it's the decompression failing it may be that xcom2 uses a different compression algorithm. I do remember seeing this but thought it was only Chimera Squad that used the different compression type, but it might be XCOM2 too. There may be another version of the decompressor that supports the newer algorithm somewhere. |
decompression of the original (XCOM2 War of the chosen) XcomGame.upk fails. Searching on the tool's authors (Gildor) forum, found some 2nd approval of that: https://www.gildor.org/smf/index.php/topic,7127.new.html#new Attempted to bump it there, will see what he answers. Seems that the decompressor detect the file as if it's UE1 (which obviously it is not) then it fails... |
|
found another thread there with hope: https://www.gildor.org/smf/index.php/topic,2942.0.html EDIT: if i filter out texture, some texture there seem to crash it, not sure why EDIT2: now crash after complaining cannot find SDL2.dll, will debug this one, not sure why not finding it... EDIT3: got it decompressed with original decompressor using "-game=xcom2" tag. |
|
I need some help. You can see working extractor on code here: https://github.com/obi0ne/xcomsave/blob/61191112d2e5dea1f5059b62b0047a695b23c74a/xcomreader.cpp#L125 That's output correct "output.dat" file with actor's table. begin of file looks like: |
|
Ah, magic numbers! I took a quick look at a couple of saves I had handy and I also see the same 0x34d and 0x78 numbers in every single one so they are not likely to be an actor count. The 3rd number changes from save to save, and looks to me like it might be the actual actor count (the number tends to grow if you look at saves much later in a campaign, and the size of the table also seems to grow so that tracks). Opening up xcomgame.u in UE explorer and looking at the package I can see "Version: 845" and "Licensee version: 120", so that explains those values, although it's interesting that they get emitted as part of the actor table (then again, the actual file structure is more or less all guesswork too so they might be a header in front of the actor table, or what I call the actor table might not even be an actor table). Just by looking at the format of the entries I can already see they are different from the ones in EU and EW but each entry does appear to have the same size other than the variable string length. My advice would be to just keep plugging away with it and recording the values without trying to figure out what they all are ahead of time. As long as you can decode the layout of the table without necessarily understanding what each field really is you may still be able to make progress. |
|
yea, it's looks to be like: Weird thing is there seems to be 108 same entires, with following: up till offset 0xFE0 108 missions ? having some place holders ? Afterward i begin seeing some different entries, such as: XComGameStateContext_ArchiveHistory , XComGameState_Item etc. I'll try to play with it further, thanks for your help. |
|
Yeah that is a little weird. From what I remember from EW saves the number following the class name was the instance number for that object, and every one was different (maybe not entirely accurate, it's been a while). These ones being identical copies is a bit strange, and I don't know why that would be. When I looked at one save yesterday it did seem to change further down: if you scroll down to where the pattern starts to change and look at the last few entries they did seem to have increasing instance numbers. I didn't check to see where the pattern changes, though, or if it really does look to be an instance number.
In XCOM2 the save game encodes not just the current state but all game states, almost like a git repo that records the entire history of the whole campaign. So you won't have just one representation for a particular entity (like a soldier) but a whole bunch of those states for the entire history of that soldier. [There are some exceptions, generally the state transition history for a tactical mission is not kept in strategy because it's so big, but some parts are and the details of how this works actually changed between the original xcom2, the dlcs, and wotc]. So when you convert to json you'd see many different copies of that soldier, and only one of them is the current state that you'd want to change if you are trying to modify something. The flat json structure does not know anything about xcom's history data structures. |
|
ok, here's what i got for this actors table.
decoded the entire actor table, attached here: the numbers don't make sense, likely to be representnig enums ? here's for example snippet of values from table: Alot of these "magic numbers" are somehow repeating, mostly 117, 118, 400, 300, 316. unlike previous game versions, the number represented instance index. EDIT: to add to this, and so on. Only question come to mind is why some fields serialized & other don't ? From explorer, eg: Why for example they are not serializing "CurrentIndex" ? |
I don't think they're enums. The string is a class name, the first number after that is relatively small, and then the next number after that is also relatively small. It also looks like the 2nd number is always the same for the same class name, but the first number changes. My guess: The first number is the index of the parent object. The history is the root at index 0 (parent -1), the children of the history are a bunch of game states (all parent 0), and the other entries after that all link their parent back to one of the game states. This makes sense to me: the history contains The second number being fixed per class type made me suspect it might be a size, but that might not be correct (in particular I wouldn't expect all
XComGameStateHistory is a native class (see the |
|
looks like you are right, what throw me off, the index of parent isn't serialized sequential, the first object is 118, The second number could be size, even for different soldier name, if each class is serialaized with place holders after names. Thanks again for your help. |
|
question: Now, looking on Hex editor, i see indeed those extra 8 bytes: followed by 4 FF FF FF FF to separate to next object, probably. However, this does not behave as previous xcom games, this does not seem to be padding, as half of it it non zeros. First number is zero. Any thoughts on what should i do with this ? another magic number ? EDIT: EDIT2: EDIT3: found source of exception, I need prop builder type of "InterfaceProperty", which does not exists. EDIT4: seems UE explorer has no info about that type: Ideas on how to decode this one ? EDIT5: EDIT6: To debug, just go into xcomreader.cpp & conditional breakpoint inside function named "read_actors_state_table_xcom2()" with condition of iActorIdx == 121,
|
|
Did you add code to support |
|
Hi, thanks for reply. |
|
I have some problem decoding the binary file: Do you think there is a chance the "None" belongs to the InterfaceProperty ?, and then we're simply needs to browse further for the next proeprty(ies) ? Next data isn't a normal propertries: I have "Team Assignment", and afterwards something named "Move this turn" |
|
Both 'MovesThisTurn' and 'TeamAssignment' are special unit value names. UnitValues are stored on units in a special native map data structure mapping strings to floats: The float value 2 is 0x40000000 and value 1 is 0x3f800000 so that is what those values are. This may be using special custom serialization logic since it's not a regular unrealscript array. |
|
should I assume this is the missing 59 bytes & attempt to decode as some property field (UnitValues), |
|
I don't know, you'll probably have to examine several different saves to see if there is anything in common for these extra bytes to be able to process them correctly. The 2 value is interesting because it looks like an int 2, immediately followed by the length-prefixed string "TeamAssignment". But after the float value 0x40000000 there is a byte value 1 immediately followed by the next length-prefixed string for "MovesThisTurn". Those don't really line up, so maybe the 2 is not an int but either a byte or a short and indicates the number of elements in the map. Then either 1 or 2 bytes unknown, then the last 0 is not part of the int but is actually an index? All speculation. If these are 1-byte index values that precede the map entries then I'd expect the total size to also only be a byte, so that leaves 2 zeroes of unknown use after the size. It's hard to guess from only a single instance, it might be more useful if you can locate more entries of this kind in the save preferably with different unit value counts to see if you can find any patterns. From the SDK sources "TeamAssignment" seems to be used only for the lost, but player soldiers often have many different unit values on them so if you can find a soldier (try searching for the name of a known soldier and look around there) you might be able to get a better idea. If you don't have the SDK installed i do recommend you install it so you can get all the base game unrealscript sources. Looking at those source files can give you a much better understanding of what you should expect to see in the save. |
|
Just to let know i'm still on it - currently on hold till I get some spare time. for last few months had no time to play almost any game(s) due to "stress load" on real life work, working coding late nights. |
Hi,
Found out your awsome work, thank you for that.
I've been pooking around the binary in attempt to support XCOM2: Wa of the Chosen save data as well.
Figures, i'll share what I have, would appreciate if you guys could share more knowledge on how you've obtain the knowledge, and hoping togther we can support new format.
And i'll up Pull Request with required changes.
Heres what's i've come by from what i've seen, regarding header info:
25): xcom_str : unknown ? has "NeutralizeFieldCommander" on my side. maybe invetory llisting ?
Any owner can shed light on how he obtained the other knowledge, i might help with this one...
cheers.
The text was updated successfully, but these errors were encountered: