-
Notifications
You must be signed in to change notification settings - Fork 751
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Translation support #66
Comments
Little other question have possibility to play in 1080 Full Screen ? |
Hi @liberodark The game will upscale to your current resolution, so yes it will run at 1080p with black bars in the side and bilinear filtering (same as GOG.com's version). That said we are currently working on the renderer to improve the visual output, here amounts having it change the output resolution meaning you will be able to see more of the world at one time (no black bars etc). |
Great thank you for your work ! |
@AJenbo We could acquire gamedata from around the world and do some basic text analysis. The purpose of this exercise is to find out what encoding tables were used for the gamedata. Analogously, OpenMW and VCMI allow the user to switch between Windows encoding tables 1250 (Eastern European), 1251 (Cyrillic) and 1252 (Western European). I pitched them an idea way back ago that we could do away with manual selection. Morrowind's main data file has a header with a 256 byte comment. In the commercial releases, this comment is in the same language as the rest of the gamedata, so we can compare that 256 byte chunk against known chunks to determine the language and encoding table. This is a little resource hungry since each acknowledged language would take up 256 bytes in resource segment and the language check is O(n). But it takes away hassle from the end user. If Diablo doesn't use UTF-8, then this is an available solution to compare found gamedata against known gamedata - either by part or checksum and size. |
Diablo always uses ISO_8859-1, changing this would break multiplayer and save game compatability. The only exception is the PS1 release, but we are unable to use the data from this release as it is in a mostly unknown format (and has a lower resolution and frame rate). There are some errors in Diablo's fonts:
(this is based on the UI font) This has been done at least partially previously: Expanding to Windows-1252 adds the following symbols: |
This sounds just lovely save for one thing: we would have to find a way to give the lines context. I would suggest some sort of annex companion document included in the package a priori. Assuming gettext would always generate the lines under the same order, having a context file (like an excel sheet) would be very easy and it would help prospective translators a lot. |
Gettext support adding comments to the message in code and then including them in the exported .pot file for the translators. |
Could you describe in detail how would this work? Would it eventually be possible to export the .po files into an online translation environment such as Transifex? As a translator myself, there are two ways this could be done. Keeping translations in a free online translation platform, or hosting a translation kit including up-to-date translation memories, which would have to be maintained manually. My suggestion is the following: whether translations are to be included in the package or hosted online, do set targets for priority languages first (giving priority to French, Italian, German, Spanish is an industry standard) and ensure their translations are finished and polished before making a package out of them. For greek, hebrew, arabic, slavic and asian languages, first handle font support before allowing linguists to begin working on them. More importantly do not let incomplete translations make their way into the package. A few other things that would be nice tot have: Finally, if possible ensure every language has at least a qualified reviewer who is a native speaker and has at least some translation experience. |
These should answer most of your questions: Regarding targeting specific languages and having native speaking translators doing reviews; we are a hobby project and as such not part of the industry so there isn't much we can do beyond piquing others interest the same way as is the case for the code that has been written. |
It does, thank you for the info. In regards to language targeting: I would at least say include only fully translated languages in the package milestone builds (incomplete ones could still be in the nightlies or when downloading source for compilation). This is so translators are motivated to finish their work, as incomplete/unpolished translations are a familiar sight in many open source projects. |
Appreciate the input |
If Transifex is chosen in the end, count me in for the German translation. I've helped with translations of a couple of Open Source projects that I'm far less familiar with than with Diablo. Also I kind of always wanted to do this; it's almost a quarter of a century since I first loaded up Diablo.exe in an hex editor and begun translating the strings found there (which was actually really difficult due to length limitations and the fact that English usually is quite a bit shorter). I'd love to eventually do this properly! |
I addressed parts of this issue by patching the MPQ with alternative audio files from the playstation release of diablo, see https://github.com/john-tornblom/psx-tools/tree/audio-lang I got the audio going with diabloweb. However, these audio files are encoded in a different PCM format, which devilutionX (SDL) does not recognize. I assume its a simple fix though. Re-encoding the WAVs with sox or something might be quick and crude way forward... I am not 100% sure if I did the WAV mappings correctly, I used a pretty simple spectrum analysis approach to correlate audio files between the PSX and PC version, so there could be errors here. Also, there seems to be some kind of locale files on the PSX release, so at least extracting some text is feasible. I'm not sure how complete it is with respect to the PC version though... |
@john-tornblom make sure you didn't enable audio compression when generating the MPQ, it's a feature only avalible in later version (starcraft era) of storm (was included in some updates) and we do not support it in DevilutionX |
thanks @AJenbo, that did the trick! I've now gotten it to work with DevilutionX (v1.0.1). |
Extracting text from a PS1 CDROM is pretty easy if you just neglect the binary preamble of the locale files. I posted gettext msgids from MAINTXT.ENG on pastebin: https://pastebin.com/61GmL3BL If these msgids are similar to the strings used in devilutionX, translating to French, German, Swedish and Japanese should be quick and strait-forward. |
Looks like the dialog texts are missing from this. But yeah it should be easy to merge thease with the devilutionx PO files. If only some one knew how to link things on Windows :( |
I posted PO files here: https://gist.github.com/john-tornblom/50b9bd4ea3a3ffc36b94f387696a1f0d |
@AJenbo I would love to help out with Vietnamese translation. Ideally UTF-8 font is recommended however I am aware of the breaking changes with multi-player and save game data. Alternatively I think the fonts could be extended to Windows 1258. Thoughts? |
Diablo was build with Windows-1252, but in-game it is limited to ASCII. Moving to UTF-8 shouldn't really be an issue and is what we plan on doing. Moving to language specific code pages like Windows 1258 is problematic as it locks the binery to a specific language, and breaks chat between various clients. The biggest blocker atm is getting gettext building on Mac and Windows. Next step would be to create a TTF version of the diablo font and figure out how to do font substitution in order to support languages that needs glyphs not found in ASCII. The current progress can be found here: #533 |
Supertux seems to be using tinygettext. From a programmer point of view, its a simple API choice that has to be made. Using string keys with simple macro, e.g., From a translator point of view, we would like to have a fileformat with good tool support, and possibly the ability to convert translations to other file formats. |
For assets (images and audio) we would use mpq or folders with localized language files. So no changes would be needed there except loading an extra mpq file. |
Right, ofc. I'm not that well acquainted with the src, I just assumed assets in MPQs are accessed using enums, like the ones in https://github.com/diasurgical/devilutionX/blob/master/enums.h |
to hell with the mpqs! :P |
Are you working on create a new font? It is necessary to check whether Diablo II uses bitmap font. |
Since we are sticking with bitmap fonts we will keep the one from D1 and and simply add the languages specific fonts as we add translations. |
devilutionx-es.po.zip |
Thanks @AJenbo for do that! There is a problem with the translation that I don't know how it could be solved, and it may happen in other languages as well. |
Ok, we will look into it once translations have been integrated. |
zh_cn.po.zip |
Thanks |
I played around with clang python bindings and made a simple script that replace string literals that appear in the locale files on the PSX PAL version with macro calls to gettext. Perhaps this can help with initial translation effort? See https://github.com/john-tornblom/devilutionX/blob/883032eaf8679dc502e178a9d9a176a2168e6f12/patch_gettext.py I also tried to replace all strings directly to Swedish using a similar script. I commited the end result to john-tornblom@37346a0 It compiles, and menu system seems to work. Unfortunately, game crashes while loading the first level. |
If you can help figure out how to link with gettext or something equivalent for all platforms that would be HUGLY appreciated :D |
Alright, I'm not a windows user so I'm afraid I cannot help with that. What about the c++ alternatives like tinygettext? I could also contribute with a custom .po parser that instantiates a hashmap we can do lookups in, would that be sufficient? |
I did look in to tinygettext and indeed it would work if we can get it to work, I was struggling with it but that was before we went with C+17 so might be better now. A custom po/mo paser would also totally work. |
For text rendering we also need something that can efficiently look up what unicode block a char belong to so that it can find the correct font texture to use. |
OK, I'll have a look at alternatives to gettext next weekend and see what I can come up with, starting with tinygettext. |
Case you didn't alreayd see my attempt: #902 |
Did some experiments with parsing MO files, see the function mo_parse at https://gist.github.com/john-tornblom/0ba2f1d46a48a288145b9b5ed0fdb501#file-mo2po-c-L57 Do you think this would be sufficient (assuming a std::map instead of an array of string_map_t)? We still need gettext to compile MO files, but I guess that is not a problem? |
Looks good.
Yeah that is fine, the po-file is normally compiled in to a mo-file by the translation tool, like Poedit or similar. |
I pushed an initial version to john-tornblom@8bf129f I apologize for the mix of C and C++, and the formatting of language.cpp. At the time of writing, no strings are actually translated, they need to be passed through the translation macro, e.g., "Exit Diablo" --> _("Exit Diablo"). This works for some strings, but not all, e.g., those defined as global variables cannot be translated at this time since they are initialized before the language subsystem. Would be nice if this could be tested on more platforms, I only use Ubuntu. By default, en.mo will be used. This can be changed in diablo.ini, and must match the basename of the .mo file, e.g., "sv.mo" with the following ini settings: When the implementation is validated on more platforms, I can automate most of the "string" --> _("string") changes. |
The cheat is to still apply the macro to them, and also to the point where they are being presented. const char *test = _("Static text");
printf(_(test)); But refactoring would be better. |
That cheat only works if the static text is translated after LanguageInitialize() has been invoked. Global variables are initialized before main() is called, in which case the language subsystem has not loaded the preferred translation (given the current implementation). I know that in C, it is possible to define a "module constructor" that gets invoked before main(), so in principle it is possible to get your suggested trick working. But I'm not sure how that works with C++, since many of its language constructs rely on a runtime lib that must be initialized before being used. If this approach is taken, we need to be very careful with the order in which different "modules" are initialized. |
Ok, ill give you a more verbose example: // Fake get text function that we tell xgettext to look for.
#define _F(x) x
const char *test = _F("Static text");
int main(int argc, char **argv)
LanguageInitialize();
printf(_(test));
} It's done for a lot of applications and works fine there, despite not being ideal. |
Right, global variables can only store msgid, and must be passed through the _() macro each time it is being used, e.g., with strcpy, strlen etc. The _F() macro is just used to generate po files? |
I integrated with CMake to optionally compile the po files (run cmake with -DUSE_GETTEXT=1), and added a Swedish translation of the main menu. Seems to work fine on Ubuntu, should I submit a PR? |
Correct, it's non ideal, but works
Sounds good :) Yes please |
Basic translation support is now enabled in the application. If anyone wants to work on a language please download Poedit and start a new translation based on the deviutionx.pot file. Then update from the source and you should see a list of the latest texts that is available for translation. Once done submit a PR with the .po that you have created. Not that there is currently a limit with how DevbiutionX renders text which limits it to the following symbols: Thanks to @john-tornblom for working on the text parser. |
Just in case someone in here didn't notice yet, but we now have support for Latin, Cyrillic and Greek (no Greek translation yet) texts in game. We also expect to have support for Korean (55% translated), Japanese (still no translation), and Chinese (fully translated for both CN and TW) later this month. |
@uwodb We just finished the first rendering of the Korean font :) If you have time to help with finishing the translation (55% compleate) there is still 1-2 weeks before we do the release. |
Hi,
Have see patch in FR for diablo but on linux with devilutionx how to make that ?
Best Regards
The text was updated successfully, but these errors were encountered: