Patching AGS games

[i30817]

apparently patches to AGS games on GoG are almost as large at the game itself, which apparently causes some pain

If whatever compression process you're using could be made to be for instance rsyncable friendly; for instance, using parellel gzip rsyncable support the idea of which could be used for a patcher that only replaces files that changed (using rsync or not).

The first link also has some problems the author of primordia found, like weak support for localization and other annoyances.

[ghost]

I think one can use rofl0r's agsutils: https://github.com/rofl0r/agsutils

On game developer's machine:

• take new and older versions of the game
• unpack in separate directories
• compare binary files to find which are different
• take those files, pack into plain archive (e.g. zip) and ship to users

On user's machine:

• unpack old game
• merge in new files
• pack back.

This may be automated with scripts of some sort. What will be left for useability is a small plugin for the Editor which lets to run this process from main menu, or something like that.

Of course, this might also come beneficial if the game development process is altered so that Editor tracked project changes, etc, but that's other thing.

[tobihan]

One could try using a binary patch format such as beat:
http://byuu.org/programming/beat/

On 02/26/2014 11:47 AM, i30817 wrote:

apparently patches to AGS games on GoG are almost as large at the game
itself, which apparently causes some pain
http://www.rpgcodex.net/forums/index.php?threads/primordia-a-point-and-click-adventure-now-available.49740/page-22#post-3135923

If whatever compression process you're using could be made to be for
instance rsyncable friendly; for instance, using parellel gzip rsyncable
support
http://mail.zlib.net/pipermail/pigz-announce_zlib.net/2012-January/000003.html
the idea of which could be used for a patcher that only replaces files
that changed (using rsync or not).

The first link also has some problems the author of primordia found,
like weak support for localization and other annoyances.

—
Reply to this email directly or view it on GitHub
#129.

[i30817]

Beat is not compression aware. In fact none of the compression agnostic binary patchers in the romhacking community are. Since you do control the compression format used on ags games you can do far better (but using a standard like resyncable gzip could lead to good things if it later can work with steam or the gog downloader).

[tobihan]

Is it possible to disable compression in AGS? In that case you could do
that (and compress the game yourself for the download).

On 02/26/2014 07:01 PM, i30817 wrote:

Beat is not compression aware. In fact none of the agnostic patchers in
the romhackin gcommunity are. Since you do control the compression
format used on ags games you can do far better (but using a standard
like resyncable gzip could lead to good things if it later can work with
steam or the gog downloader.

—
Reply to this email directly or view it on GitHub
#129 (comment).

[i30817]

edit: nevermind i see what you mean, but that is just the same process, but less intentional. You might get away with not understanding the virtual file-system if your patcher is very good about identifying binary resemblances, but you're going to have to have a very stable VFS to end with that result, and you're probably decompressing/recompressing the target prior to patching anyway, since the reason engines use VFS is for (IO) performance too.

original:
What's the point? The reason for the complaints is the size of the patch downloads. There are two ways to fix that. Compressing more (which is already quite impossible due to some theorems of information theory), or patching intelligently. To patch intelligently you have to comprehend the game virtual filesystem and only distribute changes that happen inside it, not outside; no way around it with games that use virtual file-systems.
You can also use so called 'solid' compression to compress slightly more, but that prevents using 'intelligent' patching (because decompressing depends on previously decompressed bytes) and patching intelligently is a much larger download win when you have to patch a game.

[tobihan]

If the game is not compressed a changed game will have a small delta to
the previous version so binary patches will be small. You can still
compress manually so the initial download should not be much larger.

Am 26.02.2014 19:32, schrieb i30817:

What's the point? The reason for the complaints is the size of the
downloads. There are two ways to fix that. Compressing more (which is
already quite impossible due to some theorems of information theory), or
patching intelligently. To patch intelligently you have to comprehend
the game virtual filesystem and only distribute changes that happen
inside it, not outside; no way around it.

—
Reply to this email directly or view it on GitHub
#129 (comment).

[i30817]

But the patcher will still have to decompress and recompress to update a setup file (if you want to support that usecase) and the patcher will have to be damn good at spotting binary patterns on large files (very large files), which is not exactly amenable to efficient patching. Now consider the rsync scenario: rsync is a battle hardened filesystem patcher (it not only supports single files) that even supports compressed filesystems patching inplace (with the right compression though), and can be used remotely (eventually with the Gog downloader?).

[i30817]

Bumping this to say that i don't know what i was thinking talking about delta and compression aware patchers... for 'easy' patching you only need a VFS that allows replacing files, something like the override folder in SCI and Infinity Engine. Something to think about in future AGS versions probably.

[ghost]

AGS supports overriding files since the beginning. You can place file in the game's folder and it will have higher priority than the one in the game package.

The problem here is that some resources are packed into another big file, for example, sprites are stored like that.

[i30817]

Well, then i was badly informed. Maybe it's actually 'making' these files/patches that's difficult for the average game maker. Because if that works, there is no reason for this.

[ghost]

Hmm, not just difficult, rather there is no automated way to do this.

EDIT: Also, to think of it, it does not support overriding packages, only raw files in the folder, which may be not acceptable for some game developers.

[ivan-mogilko]

I'd close this, as there are certain ways to patch the games, replacing certain files in it; there were agsutils, and now there are our own versions of few of these utils in the repository (packing/unpacking game "file libraries" etc).

It is also now possible to have compiled game scripts repacked separately from the rest of the data (both script modules and room scripts).

If further improvement is desired then there should be a task opened with more clear goals.