German version support

[laenion]

I recently got my hands onto a German version of the game, and of course I was eager to check whether I could use the game data with the open source version. Unfortunately it didn't work out: The game is crashing with a segmentation fault when trying to open the main menu after the intro movies:

graphio.cpp:293:17: runtime error: load of null pointer of type 'byte'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==13642==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000006c3f9e bp 0x000000000000 sp 0x7fff75da12b0 T0)
==13642==The signal is caused by a READ memory access.
==13642==Hint: address points to the zero page.
    #0 0x6c3f9d in PutBitmapNZ(int, int, void*, int, int) /home/ignaz/Downloads/sources/signus/signus/src/graphio.cpp:293
    #1 0x51cac0 in DrawMN /home/ignaz/Downloads/sources/signus/signus/src/mainmenu.cpp:51
    #2 0x51da7c in ProcessMenu /home/ignaz/Downloads/sources/signus/signus/src/mainmenu.cpp:106
    #3 0x51de55 in DoMainMenu() /home/ignaz/Downloads/sources/signus/signus/src/mainmenu.cpp:152
    #4 0x6a3a85 in signus_main() /home/ignaz/Downloads/sources/signus/signus/src/signus.cpp:1171
    #5 0x409e6b in signus_thread /home/ignaz/Downloads/sources/signus/signus/src/main.cpp:109
    #6 0x409e6b in main /home/ignaz/Downloads/sources/signus/signus/src/main.cpp:146
    #7 0x7f53dbf21e6a in __libc_start_main (/lib64/libc.so.6+0x23e6a)
    #8 0x40a429 in _start (/usr/local/bin/signus+0x40a429)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/ignaz/Downloads/sources/signus/signus/src/graphio.cpp:293 in PutBitmapNZ(int, int, void*, int, int)
==13642==ABORTING

What did I do? I've put all the .dat files from disc into a new directory called de, namely anims.dat, fonts.dat, graphics.dat, missions.dat, music.dat, screens.dat, sounds16.dat, sounds8.dat and texts.dat.

I assume the file format is just too old / incompatible with the current engine revision?

[nextghost]

Your assumption is correct. The original data files are not supported and compatibility was broken by original developers even before the sources were released under GPL. While it is possible to add support for the original data files later, my current priority is to make the game playable all the way to the end and improve the less-than-perfect English translation.

I can clean up and upload my Python scripts for extracting data from the original game files. But in order to merge the original German translation into this repository, we'll need permission from @vslavik, the rest of the original development team and possibly also the German distributor. Otherwise someone will have to make a new German translation from scratch.

[laenion]

I guess the scripts to extract the game data would be really helpful - that way one could just generate the files oneself independent of the license situation. (The German distributor has gone out of business a long time ago, so it may be difficult to find the current license holder...)

[vslavik]

We do not hold the rights for German localization, they were entirely with the distributor.

[nextghost]

In that case, data conversion script is the only thing we can add to this repo. @laenion I need you to look into the texts.dat file and tell me two things:

1. What's the character encoding?
2. There's an index of asset names at the end of each DAT file and the names have one-letter prefix (e.g. cwpnam9 in Czech version - the "c" is the prefix). Which prefix is used in the German version?

[laenion]

Sure!

What's the character encoding?

The texts.dat file is a bit strange, as the first paragraph seems to be in Czech, but I haven't found any encoding which would decode it correctly - so I assume that the paragraph is unused and its encoding was broken during the translation process.

All the other paragraphs are in German, which will correctly decode with ISO 8859-1 (or any other Latin-* encoding).

There's an index of asset names at the end of each DAT file and the names have one-letter prefix (e.g. cwpnam9 in Czech version - the "c" is the prefix). Which prefix is used in the German version?

Only a few files seem to have a prefix, but for those that do it's e (e.g. eunam54).

[nextghost]

The Czech paragraph is most likely encoded in KEYBCS2. I've pushed the extraction scripts to the tools directory. Run extract_langdata.py on anims.dat, graphics.dat, screens.dat, sounds16.dat and texts.dat. It'll create a directory tree in the current working directory with the root named lang. Then you'll need to do some manual fixes:

• convert *.wav in lang/speech/ to Ogg Vorbis
• edit all Makefile.am files in the directory tree and change the en segment in installation paths to de
• possibly also manually pick the correct language version in lang/graphics/bitmaps and delete the prefix character from filenames (check whether extrect_langdata.py tells you to do that or not)
• move the lang directory to signus-data/data/ and rename it to de
• register the new directory in configure.ac and signus-data/data/Makefile.am

Note that the main menu bitmaps and a few text files were copied from the English sources. That's because the files were either missing in the original data or they were not usable for technical reasons (main menu button layout is different in the GPL version).

[nextghost]

Also, all assets in texts.dat should have a prefix. If make starts complaining about missing textfiles, send me the output of extract.py texts.dat. That'll list all the assets in the dat file.

[laenion]

Thanks a lot for the commit and the detailed instructions, I have a working German version now (except for the main menu of course).

Just a few notes:

• udes55.txt doesn't seem to exist in the German version, so I deleted it from Makefile.am
• I also had to change the installation paths to de in the Makefile.am files in the de directory.
• Some mission briefings contain one or more 'þ' characters in the beginning. Again I wasn't able to find any encoding which would map that into a reasonable character. In the end I just used sed -i 's/þ//g' *.txt to remove them... (Something seems to be wrong with the encoding anyway, as 'Ä' appears both in the text, but is also used for the box around the error messages where it should be a '─' character.)

[nextghost]

Yes, you can safely delete the 'þ' characters, they were non-printable in the original fonts. The same codepoint (0xFE) was assigned to a filled square character in KEYBCS2, probably some sort of editor marker.

You can ignore the err*.txt files, they seem to be left over from early DOS version of the game and they're not used anywhere AFAICT. I'll probably delete them from this repo later.

Fixed udes55.txt in the extraction script. Just to be safe, it's better to copy the missing file from the English translation.

[nextghost]

I had to add a new text file wpnnm12.txt because two different weapons were incorrectly assigned the same name. When you update to the latest version, don't forget to copy the file to your German data directory and register it in Makefile.am. Otherwise the game might crash in mission 8 or later.

Apart from that, did you find any other issues with the extracted German data or can I close this ticket?

[laenion]

I haven't really played the game any further so far, but these were all problems I've found so far. If I should find anything else in the future then I'll just open a new ticket. Thanks a lot for all your work!

[nextghost]

Thanks, closing.