Implement a patch system to generate PCK files with only the required data

[ghost]

Describe the project you are working on:
Community Content-heavy RPG
Describe the problem or limitation you are having in your project:
I have designed my game with the idea that anyone can download the game's contents and create their own DLC with it. A large variety of assets will be made available, including various scenes containing ready-made models/collision to compose new levels with, music/sfx, animations, icons, and so on. The designer can then export their creations as a .pck, which the game's players can download (through some distribution system or another) into their user:// and enjoy.

This works great, thanks to ProjectSettings.load_resource_pack(), but there's a really big catch: there's no built-in way to prevent a designer from exporting much of, if not all, of the project's contents in their .pck. In the best-case scenario, only the relevant content is exported, which means that all dependencies are exported, too. Because the community designer is expected to not need to create their own assets when they want to create content for the game, but instead may use ready-made assets already included with the game, they wind up exporting duplicate assets due to the obligation that resources are exported with their dependencies. Worse: these dependencies may or may not be desirable, as a designer will most likely need some of those dependencies included, while other dependencies not.

For the sake of an example, let's say the project provides 1GB of assets to play with, and your typical community-designed DLC may reference 100MB of those assets, but itself provide few new assets (so just a lot of data for scenes, scripts and simple resources, in effect.) As the system is designed right now, 100MB of assets which the game already has will be duplicated into the .pck because they are dependencies for the designer's scenes. It's not only a lot to distribute, but it's not required for the game to work as it'll just overwrite what's already there, and what's more: it forces the player to be conservative about what community content they play at any given time since they may not want to have multiple gigabytes of (duplicated) content on their HDD.

That's just one aspect of the problem; another aspect is when the game's own designers want to release a patch for their game using a .pck, and that patch, just as well, references a lot of excess, unmodified content just by the nature of exporting. This example is probably less extreme than the previous one, but the idea that making a minor change to a scene can result in the inclusion of a large amount of duplicate content is a little worrying. At some point, you might think that you're better off patching your game by re-releasing the entire thing and overwriting the last one, than continually building up duplicated content until the patches alone outweigh the game's own size.

Describe how this feature / enhancement will help you overcome this problem or limitation:
It seems that Godot already has a feature that should solve this problem easily, via the patches tab in the Export dialogue:

But, unfortunately, it seems that this tool hasn't been functional for quite a long time, assuming it ever was:

godotengine/godot#22394

This tool would've been used to indicate what constitutes as the "base game" content, where you would then add more patches as you go along with adding content packs. From then on, exporting content would take into account what is already within these patches, and avoid duplicating any assets which are otherwise entirely unmodified. Now our .pck's are small and efficient, containing only essential data. Hurrah!

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
According to a tweet by Juan back in the 2.0 days, this is how it would've been used:

Aside from "add previous patches" being a little confusing, I think this design should work just fine as-is.

Describe implementation detail for your proposal (in code), if possible:
The foundational code for this system is somewhere in the engine already, though I'm not sure where it might be located. It just needs to be completed/updated.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Apparently it's possible to create your own plugin which uses the PCKPacker API to do something similar to what the aforementioned tool would do. From my experiences with creating plugins, I can say it definitely can't be worked around with a few lines of script, but it is probably doable with a lot of lines of script. Regardless, it's definitely something a lot of Godot developers could use, especially those who are creating medium to large games, or any game where modability stems from using Godot as the mod editor.

Is there a reason why this should be core and not an add-on in the asset library?:
It's in "essential but missing feature" territory.

Also, sorry for opening a proposal for a feature that's practically halfway in the engine already; I figured it could help bring more attention to it, and give some insight as to the struggle developers are facing without it, or why you might need it sooner or later.

[girng]

Yes, 100%. Great post and very well-written! Love it!! Another example: My launcher will simply download a new .pck from the server to update the client. This works nice because it's so simple and the filesize is not large. However, I could totally see this being problematic with very large games (similar to what you stated in your 3rd paragraph).

A good example of this in action is the patching system in Path of Exile. Instead of downloading the entire .pck file, individual files are downloaded and appended to the client's .ggpk file. In our case, it would be .pck.

I don't know if this is possible with .pck, however, I think managing such a patch system would be very complex now that I think of it. Luckily for me, simply releasing a new .pck works for my case because the filesize is not large, godspeed to you, my friend 👍 .

[Calinou]

This works nice because it's so simple and the filesize is not large. However, I could totally see this being problematic with very large games (similar to what you stated in your 3rd paragraph).

You can use a binary diffing algorithm, which should work well with PCK files as long as you don't compress them before sending (the algorithm will do it for you). See the "One last thing" paragraph in this article for an example where Embark Studios used casync for this purpose.

[simpuid]

The Patch System is one of the ideas for GSoC 2020 proposals, so I did some digging in the exported zip of a C# example project and found that all C# scripts get compiled to a single dll which makes C# unfit for mod/DLC like patches. Is there any other way for DLC like patches with C#?

[Janders1800]

It would be nice projects load the resources from a patch, so it can be modified on editor, meaning.

• I release a game with his GameContent.pck and a copy of Godot.
• Some user opens Godot and loads the GameContent.pck (essentially Godot fills res// with the pack contents) and makes a mod ModA.pck
• Some other user downloads ModA, opens Godot and loads GameContent.pck and ModA.pck (Godot fills res// with the contents of GameContent and substitutes conflicts with ModA, per load order) and makes ModB.
• user opens Godot and loads the GameContent.pck and makes a mod ModC.pck
• etc.

Some gamer downloads ModA B And C, ModA gets updated by ModB and if ModC don't have conflicts with ModA or B great, if it has conflicts with ModA or ModB substitutes files per load order and plays the modded game.

GameContent
 |        |
ModA     ModC
 |
ModB

If you don't want people stealing your game loading a pack it can be an option to lock a pack or encrypt so it doesn't fill res// but still can make mods if you provide the Api for it.

Is there a milestone for this? It will be nice to not "keep forgetting".

[Calinou]

Is there a milestone for this? It will be nice to not "keep forgetting".

We don't use milestones in this repository. In open source projects, people work on what they want to work on 🙂

[kuruk-mm]

Hi

I'm making a MMORPG and we make a patch system, that is working.
The problem is we made it for our needs, and we want to contribute it to Godot. So we're going to adapt it to the generic needs of Godot.

We try to do it with less modifications to the core engine, but it was a bit difficult. And our main requirement was 'change everything we want on the game', if we want to change the game completely, we can.

So, there is a list of problems that we had:

• class_name: Class names are in the project.settings, so if you add/remove a new class_name, currently you don't have a way to reload that.
• singletons / auto loads: The autoloads are in the project.settings too, so if you add/remove/change an autoload, you can't reload those. We tried to make some GDScript tricks playing with reload nodes, but it was very awkward.
• every project.settings parameters: if we change a parameter in the project.settings it will not take effect.
• FileAccessPack class in the core engine depends on its own singleton, and you cannot instance another FileAccessPack because you change the Singleton (look FileAccessPack constructor)... so we recode it in a new class.

So, to resolve all those problems, we tried to make some new functionality to the ProjectSettings class, adding the ability of reload the class_names, auto_loads... but we drop this decision because it's impossible with some parameters like graphic parameters, or networking (Everything after globals->setup(...) in Main::setup of the core engine).

Our next approach was hardcode a load_resource_pack in the "ProjectSettings::_setup", that it resolve all ours needs.
But the problem with this, is that we lose the ability to load packages from GDScript, and we resolve it reloading the game completely from GDScript like the Editor does with "Save and restart" on the ProjectSettings.
And this solutions is working well on the Desktop, we don't know the result of it in mobile. We didn't test it yet.

To download the patches we do it from GDScript, and to merge from it too, we add some bindings to do that from GDScript.
To make the patches, we add some parameters to Godot command line, generating the differences from two packages from the command line. We can do what Reduz said, and do it from the Editor. But for us was better like this to make our patch system from the CI with a godot headless compilation.

So, I want to make a repository with all of this before to make a PR, and with an example project in GDScript to download new patches. And some bash scripts to make the patches, zip those, and upload. (i'll try to do it in september)
I need a bit of help on how to load the patches in a generic way.

I don't give the code yet, because I want to contribute it correctly (we're using some std stuffs, and things that are not allowed in Godot) and we added an encryptation for PCKs that need to be removed to the contributed code.

I'll appreciate any feedback. Thanks!

[fire]

If you can provide a branch or a pr, interested people and I can review it.

[kuruk-mm]

If you can provide a branch or a pr, interested people and I can review it.

I will try to do as soon as possible. The problem now is I can't give support to that code until september. And I need to remove the PCK encryption that I have.

My main reason to discuss here, is try to find a way to load patches to all needs and understand the requirements for a Godot patch system. What I have done, it's a md5 differential on changes, and delete files if needed. And it'll remove new files on old patches.

[fire]

@kuruk-mm Is your project still active?