Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

COMMON: Standardize language names and codes #2440

Merged
merged 3 commits into from Nov 11, 2021
Merged

Conversation

@trembyle
Copy link
Contributor

@trembyle trembyle commented Sep 1, 2020

Please see the additional documentation in language.cpp. The goal is to use more consistent naming conventions based wherever possible on ISO standards.

Some existing entries needed to be changed in common and also in individual engines. I was able to build successfully and detect and run games in these languages. I'm looking for feedback on this. Is there anything else that I need to consider?

@sev-
Copy link
Member

@sev- sev- commented Sep 1, 2020

Let's imagine I already added Chinese DIG to ScummVM (I really did yesterday). It is stored in scummvm.ini with 'language=zh', which was ZH_TWN. Now, that language is parsed to ZH_ANY. But in SCUMM engine code there are no checks against ZH_ANY, and the game will stop working.

Thus, a separate step for converting scummvm.ini is required. We did that in the past when we made some targets obsolete. See engines/obsolete.cpp. In order to avoid multiple conversions, I would do the following: Check versioninfo field in the INI file, if it is 2.2.* or less, then iterate over all targets and convert language, then save the file. Saving file will update versioninfo, and you will avoid double conversion.

Loading

common/language.h Outdated Show resolved Hide resolved
Loading
{ "ru", "ru_RU", "Russian", RU_RUS },
{ "sr", "sr_RS", "Serbian", SR_SRB },
Copy link
Contributor

@lolbot-iichan lolbot-iichan Sep 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need "sr_RS@latin" here, since "sr_RS" seems to be cyrillic locale and the only currently supported game in Serbian is using latin script.

Loading

Copy link
Contributor Author

@trembyle trembyle Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The right way might be to include another parameter for codeset. We also don't differentiate between Traditional and Simplified Chinese either, but it looks like both are available in Wintermute.

This list has some more examples:
https://docs.oracle.com/cd/E23824_01/html/E26033/glset.html

Loading

Copy link
Contributor

@lolbot-iichan lolbot-iichan Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also don't differentiate between Traditional and Simplified Chinese either

In WME, "Chinese (China)" and "Chinese (Taiwan)" are used in "Simplified Chinese" and "Traditional Chinese" meaning respectfully. As far as I understand, those things are pretty much the same:

  • Simplified Characters were introduced by China and are also used at Malaysia and Singapore, China is much bigger then Malaysia and Singapore in both area and population.
  • Traditional Characters are still used at Taiwan, Hong Kong and Macau, Taiwan is much bigger then Hong Kong and Macau in both area and population.

I guess "Chinese (China)" and "Chinese (Taiwan)" can be renamed to "Chinese (Simplified)" and "Chinese (Traditional)", if needed.

ZH_ANY was added as a workaround for a WME game Reversion that at some moment contained 3 Chinese translation packages in a single game directory after an update: new Simplified and Traditional translations were added and old Chinese package was not deleted (and detected by detection.cpp). To fix this, we decided to mark all old Reversion Chinese games as ZH_ANY ("generic" Chinese) and use ZH_CHA/ZH_TWN for Simplified/Traditional.

Loading

@lolbot-iichan
Copy link
Contributor

@lolbot-iichan lolbot-iichan commented Sep 2, 2020

Thus, a separate step for converting scummvm.ini is required. We did that in the past when we made some targets obsolete. See engines/obsolete.cpp. In order to avoid multiple conversions, I would do the following: Check versioninfo field in the INI file, if it is 2.2.* or less, then iterate over all targets and convert language, then save the file. Saving file will update versioninfo, and you will avoid double conversion.

Besides ConfMan's language records and AdvancedDetector target names, there is one more scenario: engines/glk/quetzal.cpp is writing language codes to savenames. Not sure if it's fine or not.

Loading

@trembyle
Copy link
Contributor Author

@trembyle trembyle commented Sep 2, 2020

Besides ConfMan's language records and AdvancedDetector target names, there is one more scenario: engines/glk/quetzal.cpp is writing language codes to savenames. Not sure if it's fine or not.

Since this is the first public release to include GLK, users shouldn't expect saved games from prior development builds to work in stable 2.2.0.

Loading

@trembyle
Copy link
Contributor Author

@trembyle trembyle commented Sep 2, 2020

Thus, a separate step for converting scummvm.ini is required. We did that in the past when we made some targets obsolete. See engines/obsolete.cpp. In order to avoid multiple conversions, I would do the following: Check versioninfo field in the INI file, if it is 2.2.* or less, then iterate over all targets and convert language, then save the file. Saving file will update versioninfo, and you will avoid double conversion.

I think I got close here, but it's not quite working yet. I'm unable to detect some games with new codes. Can you see what I'm doing wrong?

Loading

@sev-
Copy link
Member

@sev- sev- commented Sep 2, 2020

What are "some" games?

Loading

@sev-
Copy link
Member

@sev- sev- commented Sep 2, 2020

Besides ConfMan's language records and AdvancedDetector target names, there is one more scenario: engines/glk/quetzal.cpp is writing language codes to savenames. Not sure if it's fine or not.

Since this is the first public release to include GLK, users shouldn't expect saved games from prior development builds to work in stable 2.2.0.

No, this change cannot go to 2.2.0, it is way too risky so close before the release.

Loading

@trembyle
Copy link
Contributor Author

@trembyle trembyle commented Sep 2, 2020

Besides ConfMan's language records and AdvancedDetector target names, there is one more scenario: engines/glk/quetzal.cpp is writing language codes to savenames. Not sure if it's fine or not.

GLK has only one game in an affected language (nissen). This is only available in Danish, and the two character code for that language is not changing.

Loading

@trembyle trembyle changed the title Standardize language names and codes COMMON: Standardize language names and codes Dec 1, 2020
@sev-
Copy link
Member

@sev- sev- commented Nov 1, 2021

I think, it is now a good time to merge this, but it got bitrot in the meantime. Could you please rebase?

Loading

@trembyle
Copy link
Contributor Author

@trembyle trembyle commented Nov 3, 2021

I think, it is now a good time to merge this, but it got bitrot in the meantime. Could you please rebase?

Yes, I will work on fixing the config file updates for older language codes first, then rebase.

After the merge, I'll review engine detection entries for any codes that are changing so that they meet the new standard.

Loading

@trembyle
Copy link
Contributor Author

@trembyle trembyle commented Nov 5, 2021

I'm getting closer on the functionality, but it's not there yet. I pushed just to show some progress. Only the bare minimum rebase has happened, just to get it running. More cleanup (splitting & squashing) of commit history is needed.

I'll keep working on this. Certainly I'm not expecting a review until it's complete, although if anyone is feeling charitable, pointers will be gladly accepted. In particular, I need to figure out where to put this new code. It's not getting called from GUI::LauncherDialog::loadGame as I expected.

Loading

@trembyle trembyle force-pushed the language branch 2 times, most recently from 2f9aff6 to afa9f56 Nov 6, 2021
trembyle added 3 commits Nov 6, 2021
* Add documentation to add or maintain language values
* Fix and reorder existing entries to match ISO standards
* Use ISO-3166 country codes and ISO-639 language codes
When an obsolete language code is encountered in a user's
config file, we translate this to the updated language ID.
@trembyle
Copy link
Contributor Author

@trembyle trembyle commented Nov 6, 2021

@sev- It should be working now.

Loading

@sev-
Copy link
Member

@sev- sev- commented Nov 11, 2021

Thank you!

Loading

@sev- sev- merged commit 5ef7793 into scummvm:master Nov 11, 2021
5 of 8 checks passed
Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants