Game uses crazy binary data files

[willglynn]

Mostly these are images which have already been converted by the but2png utility, but for which the game still uses the original data files. Some of these actually contain data.

Crazy binary data is bad because it has endainness and structure packing issues. It also discourages modification. This issue is intended as a meta-issue to centralize discussion and track progress towards eliminating these files.

[willglynn]

We've made a bunch of progress on this. Random data files have been replaced with PNGs one piece at a time, and #100 gives us a way to manipulate the graphics system independent of these crazy formats.

[rnyoakum]

I've gone through RIS and eliminated the reliance on the .but and .img files that were hanging around (merged via commit 49a85a5). Those have been replaced with pngs that had been extracted previously by the RIS team or that I extracted. I haven't deleted the original files yet.

[hweimer]

With serialization via cereal being included, it is possible to make progress on this. In many cases, the game uses files containing a big binary array of some internal data structure. For example, the file mission.dat is one big array of struct mStr. So if you want to get the mission with code 42, the game will seek() to 42*sizeof(struct mStr) and read the next bytes into the proper struct. There are several possibilities how we can proceed to convert these files into human-readable JSON files:

1. Load all data files into memory when the game starts. Will significantly increase memory usage, but it's not 1992 anymore.
2. Perform live deserialization of the required data file. Might be too slow for some of the bigger files, although rapidjson is pretty fast.
3. Split the file into many small pieces, each containing one element of the array (e.g., mission_42.json). More cumbersome to work with these files.
4. Keep the existing data formats, but automatically generate the datafiles from the corresponding JSON files during compile-time.

Opinions?

[peyre]

It would be exciting to finally ditch some of this hex file nonsense. I'm not a big fan of 3 or 4: replacing the hexadecimal files with easy-to-edit plaintext would be better, I think, and I'm not enthusiastic about splitting them into umpteen text files. I'm not sure I quite understand 2, but 1 sounds like it might be our best option.

[hweimer]

Well, in 4 you also have an easy-to-edit JSON file in the source directory. It just gets translated to a binary file during the compilation of the game. Will probably require some CMake trickery (which is not exactly my area of expertise), but should be doable.

[peyre]

Well, however best you can do it, it would be a good thing - the main thing is to make the files easily moddable.

[hweimer]

These suggestions could be helpful.

[hweimer]

In #749, I went for option 2. Please report any issues with noticeable delays when flying a mission.

[peyre]

That's fantastic, that you've made some progress on converting a few of our binary files to plaintext. I'd love to see some of our really early Issues resolved - in addition to the whole "ease of modification" aspect.

[hweimer]

Luckily, replacing fails.cdr by a JSON version turned out to be much easier than replacing the sequence files.

[peyre]

Great to hear.

[peyre]

I've just played most of a game. The change does seem to cause slight delays following missions, but no big deal.

However, when I was getting close to launching the Moon landing, the game crashed:

WARNING /home/leon/raceintospace/src/game/fs.cpp:279:try_find_file	can't find file `PPIX_08.ogg' in audio dir(s)
unhandled exception: vector::_M_range_check: __n (which is 2954) >= this->size() (which is 2954)
Aborted (core dumped)

ppix_08.ogg does exist in raceintospace/data/video/mission/, so I'm not sure what's going on.

[hweimer]

The warning is about a missing audio file during countdown and is unrelated to the crash. 2954 is the length of the vector holding the failure sequences, so it looks like that some sequence code couldn't be found and the index variable increased beyond the bounds of the vector. I will put in some additional checks that the game doesn't crash in such a situation.

[hweimer]

In #774, I'm now going from Option 2 for deserialization (see #3 (comment)) to Option 1 as the delays from deserialization are quite significant in AI-vs-AI simulations.

[hweimer]

With #823 being merged, the following binary files remain in use:

• baby(clif|norm|picx).cdr (Small images shown in mission control)
• crew.dat, user.dat (Roster information, but there is also a roster.json?)
• endgame.dat (Texts for the endgame screen)
• event.dat, news.dat (Event and news texts)
• mission.dat (Mission codes and mission step data)
• ntable.dat (Budget modification)
• portbut.but (AI thinking screen, but there are also PNG images under data/images?)
• p_rev.dat (Presidential review texts)
• (sov|usa)_port.dat (Spaceport images)
• spots.cdr (Spaceport animations)
• letter.dat (Font data for headlines?)
• vtable.dat (Coordinate data for VAB components)
• liftoff.abz (Component animation in the failure screen)

[peyre]

Also crew.dat, men.dat, and user.dat, for the 'naut roster.

[hweimer]

men.dat is just a copy either of crew.dat or user.dat, depending on the roster type you are playing. If I understand #4 correctly, roster.json and the related code are currently not being used at all. Is this correct?

I'd say that about half of the items (endgame.dat, event/news.dat, mission.dat, ntable.dat, portbut.but, p_rev.dat, and vtable.dat) are probably relatively easy to convert. Is there any of those that would be particularly useful?

[peyre]

I don't know about roster.json one way or another, sorry.

I look forward to seeing crew.dat and user.dat converted; that would allow some other things, like implementing the Service flag and adding a flag for Race (so our black and Asian astronauts don't look white).

Of the ones you suggest there, I expect event/news.dat would be more helpful.

[hweimer]

As a little christmas present, I have converted the roster files to JSON. I have left the old roster.json stuff in place for the time being; I will remove it if there are no issues with my implementation.

[peyre]

That is a nice present, thanks!

So, if I wanted to add the Service category, would it simply be a matter of adding a line for each astronaut saying

"Service": 1,

and so forth?