[Use consultation] How do you parse only part of the data?

[zaigie]

First of all, I would like to extend my heartfelt thanks for your project, which has solved many of my problems! I have developed a visualization tool for server management palworld-server-tool based on your project.

In the part that I wrote myself, particularly in structure.py, I only utilized CharacterSaveParameterMap and GroupSaveDataMap. I did not use any of the other data.

Before you updated the code on parsing WorkSaved data, my entire parsing process took less than 60 seconds, but now it generally exceeds 90 seconds.

I suspect that the increase in parsing data is the reason behind the longer processing time and higher memory usage (I am also aiming to keep memory usage as low as possible).

Hence, I am raising this issue to seek your advice on customizing the parsing of certain data, as I am somewhat at a loss.

[cheahjs]

In your case, when you call GvasFile.read(raw_gvas, PALWORLD_TYPE_HINTS, PALWORLD_CUSTOM_PROPERTIES), you can replace PALWORLD_CUSTOM_PROPERTIES with the subset of RawData structs that you're interested in, eg

PALWORLD_CUSTOM_PROPERTIES: = {
    ".worldSaveData.GroupSaveDataMap": (group.decode, group.encode),
    ".worldSaveData.CharacterSaveParameterMap.Value.RawData": (
        character.decode,
        character.encode,
    ),
}

This would skip processing of the other data that I have added.

In the future, I might look into lazily evaluating the RawData structs such that you don't incur the parsing cost until you access it (if you consume the library, or if you dump it as JSON), or provide configuration options in the CLI to pick and choose.

[zaigie]

Thanks, that helps!

[vegeto079]

Thanks for working on 4f8234f, that made this better!

For context, I am building a C# app that runs palworld-save-tools on a frequently-scheduled basis to pull out JSONified server data for monitoring.

On my server I am finding that the processing time is faster with the new --custom-properties option, but saving the JSON is still quite long, due to it saving >500MB of data. The saving action now takes 90% of the total execution time.

If I understand correctly, --custom-properties is reducing the PALWORLD_CUSTOM_PROPERTIES list, but regardless of that, everything inside PALWORLD_TYPE_HINTS is still being processed and saved.

My thought would be that passing in properties would exclude all other properties from the result. Ideally, I could pass .worldSaveData.CharacterSaveParameterMap.Value.RawData and receive a JSON result that would only contain that key path and it's children.

Would you have any idea on how to accomplish that, aiming to reduce the overall execution time by reducing the payload needing to be saved?

[cheahjs]

You can take a look at what https://github.com/magicbear/palworld-server-toolkit does to skip processing various properties.