Media file versioning

[TheJJ]

When changes to the media files are done, the engine does not realize the media files need to be reconverted and strange errors may occur. E.g. #202 changes the alpha values of the sprites and thus requires a reconversion. We can't enforce it currently, this is why some versioning needs to be introduced.

Proposal: introduce some media_version variable which hast to be incremented each time a reconversion needs to be done. This version is stored in the assets/ folder and checked by the engine.

Other suggestions?

• asset versioning (openage/convert/changelog.py)
• gamespec format hashing

[franciscod]

If the change only affects, say, sounds, it should be nice not to reconvert all the stuff, instead only reconvert sounds. We might want to fragment the version based on media type.

[TheJJ]

Indeed, so this is closely related to #79 and #188.

[franciscod]

untested idea code:

import pickle

target_media_version = {  # these numbers should be bumped when changing media code
    'graphics': 1,
    'terrain': 1,
    'sounds': 1,
    'gamedata': 1,
    'interface': 1,
}

def get_current_media_version():
    return pickle.load(open('assets/versions.p', 'rb'))

def set_current_media_version(media_version):
    return pickle.dump(media_version, open('assets/versions.p', 'wb'))

def get_mismatching_media_modules():
    mismatching = []
    current_media_version = get_current_media_version()

    for modulename in target_media_version:
        if target_media_version[modulename] != current_media_version[modulename]:
            mismatching.append(modulename)
    else:
        return None

    return mismatching

def update_media_module_version(modulename):
    version = get_current_media_version()
    version[modulename] = target_media_version[modulename]
    set_current_media_version(version)

# after finishing converting a module
update_media_module_version('sounds')

# list of modules to be converted
for module_name in get_mismatching_media_modules():
    please_start_converting_this_one(module_name)

This pickling might be improved when nyan arrives though

[franciscod]

this numeric approach has a caveat:

• let's say at master we have graphics version 5.
• some branch A modifies something about graphics, sets their asset version to 6
• another branch B modifies another graphic thing, also sets version 6
• A gets merged
• when merging B, we should really check that the graphics gets bumped to 7 at least

[TheJJ]

We should hash the version whereever possible, this is persistent across merges.
For non-hashable versions like graphics, the numeric approach is, as you proved, bad.

The problem we have to solve is that some changes to the algorithms used don't require reconversion, but others do. How do we separate between them automatically (probably we'll hit the halting problem..)?

Anyone any other ideas?

[inakoll]

This is how I would do it :

Let say we have a file containing hash (sha1?) of all the converted media files. This hash file is under source control and is merged automagically.

1. Let say this file is up to date on all branches.
2. The convert script could read this file compute the hash and only reconvert necessary files.
3. If the hash is still bad after the reconversion, everything needs to be re converted and the hash file should be updated and committed.
4. Some trusted people should maintain this hash file up to date each time it is really necessary on master.
5. Others don't care about media files and let the build system figure it out.
6. The "version" of the media could simply be the hash of this special file.

[TheJJ]

The media version might be different accross people. Especially if they use mods, which just work transparently after conversion. In the future, when we officially support mods, this attempt will also fail and is inconsistend accross people.

@franciscod showed that using simple numbering might lead to problems, but thinking over it, it is a non-issue: If two people increase the number, this will lead to a merge conflict. And the human resolving the merge conflict has to increase the number to the appropriate following number.

-> The simple numeric approach will work i'd say.

The hashing for the gamedata format is done by storing the parsed version of the empiresdat.py struct.

[TheJJ]

done in #323, the hashing from #265 is pending.

[TheJJ]

Upstream with e59f5c8.