Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New translation file loading #1795

Merged
merged 20 commits into from
Jul 23, 2018
Merged

New translation file loading #1795

merged 20 commits into from
Jul 23, 2018

Conversation

ferdnyc
Copy link
Contributor

@ferdnyc ferdnyc commented Jul 5, 2018

Update: This has now grown into a full replacement of the code to manage translation-file loading, including changes to the locations and indexing of the translation files themselves (though the individual files have not changed, other than being renamed and moved to a new directory). See comments and commit messages for further details.

While this is no longer marked WIP, one item remains unfinished:

  • Required: Updates to file paths in build scripts
  • Required: Code to gracefully handle missing translation resources (possible when OpenShot is packaged by Linux distros, where translation files are separated out into an optional package)
  • Optional: Command-line arguments to launch.py to list all supported languages available to OpenShot (--list-languages), launch OpenShot in a specific language (--lang <code>) overriding both the environment and the preferences file for the current session only.

Original PR

PR #1332 sought to fix loading of English translations in non-English locales (see #1256) by removing a hardcoded en_US bypass in the translation file loading. In doing so, it inadvertently broke loading of translation files that match the system locale.

This removes the entire "skip loading" clause, because we can never assume that any translation shouldn't be loaded. English translations may need to be loaded and applied, if OpenShot's language is set to English in non-English locales. In contrast, we still need to load our OpenShot translations for the local language even if the selected language is the same as the system locale.

Fixes #1521. (I believe. It's not something I can test, since I'm not in a non-English locale.)

Also, the log messages for $LANG and $LOCALE no longer fall back to default strings, so
they will properly show when the corresponding variables are unset or set to the empty string.

The log messages that output the LANG and LOCALE environment variables
were using `QLocale().system().name()` as a fallback if the variable
is not found. But we'd want to know if they're unset, so default to "".
PR OpenShot#1332 sought to fix loading of English translations in non-English
locales (see OpenShot#1256) by removing a hardcoded `en_US` bypass in the
translation file loading. In doing so, it inadvertently broke loading
of translation files that match the system locale.

This removes the entire "skip loading" clause, because we can never
assume that any translation shouldn't be loaded. English translations
may need to be loaded and applied, if OpenShot's language is set to
English in non-English locales. In contrast, we still need to load our
OpenShot translations for the local language even if the selected
language is the same as the system locale.
@peanutbutterandcrackers
Copy link
Contributor

So I have noticed something.

Currently (with v2.4.2), if I have set my OpenShot language to 'Default' but have anything other than en_US as my system's default language, OpenShot isn't respecting that but using en_US. But if I myself change the language to be anything particular, it respects that.

With this branch, if I set my OpenShot language to 'default' and have my default language (in the the system to be, say ne_NP, for instance) it respects the default language choice. +1 for this PR
However:
When my system language is set to something other than en_US and I tell OpenShot to use 'en_US' it doesn't do it. Only when the system language is set to 'en_US' am I able to have OpenShot use 'en_US'. However, 'en_CA', 'en_BR' all seem to work just fine. But then again, I haven't tested those 'en_*' variations with my system locale set to the same language...

Thought this would be necessary info. :)

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 8, 2018

That is interesting. That may mean that this code, from get_current_locale() also needs to come out:

	 
         # Don't try on default locale, since it fails to load what is the default language
         if 'en_US' in locale_name:
             continue

I thought "current locale" (in the function name) meant the system locale, but if it means the locale being loaded, then there's the same issue that I corrected with the previous change: Even though en_US is the "default language", we may still need to activate it if OpenShot is actually set to English in a non-English locale.

However, it's true that loading an en_US translation file isn't technically possible, simply because there isn't one. That's the sense in which it's the "default locale", as that comment says. The intent of the original code, AFAICT, was to avoid loading any translations when en_US was the interface language selected for OpenShot. It just didn't work right, and then the fix made things even worse. CORRECTLY preventing the loading of any translations might work, but since I think Qt also auto-applies its own translations for the system locale, even that may not suffice.

Perhaps what we need to do is special-case en_US to unload any other translations, when it's selected. Unloading translations is technically possible in Qt. Only problem is, you have to know what translator objects were previously loaded and pass in the ones you want to unload. I can't find any method to query the currently-installed translators, in order to remove them. I'll ask on the Qt forum.

ETA: This turned out to be a red herring, as I discovered that get_current_locale() was basically duplicating the entire logic for loading translation files, but was only ever called from the tracking-metrics code in order to embed a locale variable. So I ripped out the entire method, because that's crazy.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 8, 2018

The other option, in thinking about it, would be to just provide a translation file for en_US with OpenShot. It'd be simple to generate, since it would just involve translating every string into itself. But then we could actually load that file when en_US is selected.

Anyway, I've asked the Qt forum about querying installed translators.

This completely reworks the on-disk model for translation files,
doing away with the `lang_code/LC_MESSAGES/` path hierarchy entirely
in favor of a flat directory structure containing files named with
the pattern "OpenShot.lang_code.qm". This allows us to take advantage
of QTranslator's fallback code that will, for instance, automatically
load `OpenShot.fr.qm` when `OpenShot.fr_ZZ.qm` is not found.

The translation files are also indexed in a QResource file named
`openshot_lang.qrc`, which is compiled into `openshot_lang.py`. This
allows the translations to be loaded from the resource path ":/locale".

The translations are stored in the `language/` subdirectory, instead
of `locale/`, because `from locale import openshot_lang` conflicts
with the Python module named `locale`.
This modifies the translation-loading code to make use of the
":/locale" resource path for OpenShot translations, and greatly
simplifies the loading code by making use of Qt's automatic
fallback in `QTranslator.load()` instead of parsing the pieces
of the locale string directly.

The list of supported languages is now built using a QDirIterator
over the ":/locale" resource path.
Instead of all the complicated nonsense `get_current_locale()` was
performing, have it just return `info.CURRENT_LANGUAGE`. It was
only ever being called in one place, `src/classes/metrics.py`, and
doing a lot of work for nothing.

TBH, I'm not even sure this will work now (`info.CURRENT_LANGUAGE`
might not always be set when `get_current_locale()` is called.
If it breaks the metrics, we'll fix it some more-sane way than
basically **repeating** all of the translation-loading code just
to set a single metrics variable.
@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 9, 2018

OK, so I just blew up the world, translations-wise.

The set of commits I just pushed in here basically guts the entirety of the previous translation code, including moving all of the source files around for more efficient access. (They're now all stored in one directory, as src/language/OpenShot.LANG_CODE.qm, and indexed in a QResource file in that directory.) The loading of translations is now completely redone to take advantage of QTranslator's automatic fallback for language codes instead of parsing the codes manually.

As a result, you can now launch OpenShot with e.g. LANG=fr_ZZ.florb openshot-qt and the interface will still come up in French (if the language preference is set to Default) because Qt knows enough that if it fails to find translations for "fr_ZZ.florb", it automatically tries "fr_ZZ" and then "fr", stopping at the first one that loads. This applies to both its internal translations and OpenShot's translation files.

#WorksForMe but that's hardly a comprehensive evaluation, especially as I'm in the en_US locale. But I did try various combinations of system and application locale selections and it seemed to always do the right thing.

Testing is very much needed, so please have at it.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 9, 2018

(This does duplicate all of the translation files in the source, but that's meant to be temporary. Once we can coordinate with @jonoomph to update his process for updating and importing translations, we should be able to basically blow away the src/locale/ directory except for a couple of utility files that can move to src/language or wherever. Also, the build scripts will need to be updated before this can be merged.)

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 9, 2018

Also, I just realized I'll have to push another commit into this.

The rpmfusion packages for Fedora package all of the translation files in a separate RPM, openshot-lang, which is optional to the install of openshot itself. If the package isn't installed, /usr/lib/python3.6/site-packages/openshot_qt/locale/ is an empty directory. I'm guessing that's not uncommon.

So, I'll have to handle the case where the language/ dir is missing/empty and the code can't load its resource file, which currently will break it. But that's easy enough to fix.

@ferdnyc ferdnyc changed the title Don't skip loading translations in system locale WIP: New translation file loading Jul 9, 2018
ferdnyc added a commit to ferdnyc/openshot-qt that referenced this pull request Jul 10, 2018
This add standard Python argparse handling for command-line arguments to
launch.py, which now responds to `--help` with:

```console
src/launch.py --help
Loaded modules from current directory: /home/ferd/rpmbuild/REPOS/openshot-qt/src
usage: launch.py [-h] [--list-languages] [-V]

OpenShot version 2.4.2

optional arguments:
  -h, --help        show this help message and exit
  --list-languages
  -V, --version
```

There is also commented out code for another argument, `--lang`, to
set the interface language on the command line via language code (as
listed using `--list-languages`). That will require the changes from
from PR OpenShot#1795 before it can be enabled.
@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 10, 2018

@jonoomph @peanutbutterandcrackers @DylanC @N3WWN (et al):
While this is still a WIP, any time you have to take a look at it and any thoughts you have on the direction it's going would be very welcome. Main takeaways, with what I'll attempt to make brief explanations:

  1. The translation-loading logic in src/language.py is gutted and reworked
    QTranslator has an automatic fallback feature that we were not making use of at all. Short-short version, if you tell it to load the translation file OpenShot.en_RU.latin1.qm and it's not found, it's smart enough to fall back to OpenShot.en_RU.qm and then OpenShot.en.qm automatically, without having to parse codes. The new code takes advantage of that, both in the Qt and OpenShot translation directories.

  2. To make the previous work, all translation files are moved into the same directory.
    They're now named OpenShot.code.qm instead of code/LC_MESSAGES/OpenShot.qm

  3. The language files are indexed with a .qrc file, bytecompiled into a loadable Python module
    Just like images/openshot.qrc is compiled into images/openshot_rc.py and loaded with from images import openshot_rc, the translation files have language/openshot_lang.qrc, compiled into openshot_lang.py and loaded with from language import openshot_lang.

    (I had to change the name of the directory from locale to language because from locale import ... conflicts with a Python module.)

    All of the supported language files need to be listed in the resource file.

  4. The language path is referenced as a Qt resource, ":language/", just like ":images/"
    classes/info.py now uses a QDirIterator to walk that resource file listing when building info.SUPPORTED_LANGUAGES, and classes/language.py searches it when loading translations.

Those last two are potentially the most controversial/problematic ones, as they mean that new translations can no longer be added simply by dropping a file (or, a directory) into the right location and having it automatically picked up. The file also has to be added to the .qrc, which then needs to be bytecompiled into a new openshot_lang.py.

I decided to go the resource-file route after noting that __pycache__ was listed in info.SUPPORTED_LANGUAGES on my system — the directory-scanning was picking up every subfolder of src/locale, including ones that have nothing to do with translations. The resource file prevents that, and it also makes searching and loading them far more efficient.

However, if that's a problem, a supplemental path can be added where OpenShot searches for additional translation files, as a method of adding more/updated translations ad-hoc. The resource file is not required to make the rest of this work, so if anyone objects I can revert the translation-loading logic to scanning language/ as a directory, like before. We'd still retain the fallback logic and other benefits.

As I said in my last comment, I still have to add logic to make sure that a missing language directory and/or resource file doesn't break anything, since that will be a common situation with Linux distros. But I feel this is basically there, and in my testing it's handled every combination of LANG= setting, language selection in the interface, and etc., and OpenShot always displayed in the correct language. Thanks to Qt's fallback, you can run LANG=fr_ZZ.florb src/launch.py and OpenShot will still come up in French (as long as the language is set to Default in the preferences).

With the new command-line argument processing I've submitted in PR #1828, I plan to add a --lang command-line argument that will force OpenShot to use the specified language for the current session, overriding and ignoring both the environment and the preferences setting.

Just like `make` in `src/images` builds `openshot_rc.py` from
`openshot.qrc`, in `src/language` it will build `openshot_lang.py`
from `openshot_lang.qrc`
@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 13, 2018

@peanutbutterandcrackers inexplicably decided to send me results of testing (very much appreciated!) in a direct email. Since that information is vital to this PR, reproducing here:

When the default language of the system is set to anything else but 'en_US', your branch respects the default but won't respect it if the user selects 'en_US' as their language of choice. Like I reported earlier. Here are a few screenshots:

In all of the screenshots, except for one in which the language is set to 'en_US', you can see that OpenShot is respecting the language choice (ne_NP isn't translated well - almost no translations except for a single string; I might spend some time in translating to Nepali when I have the time - perhaps after a month or so). I set the default locale of the system (for the user) to be some East Asian Language for the testing. 'en_CA' and all works just fine.

Screenshots not included, as I'm happy to take anyone's word for what language OpenShot loads in. If someone tells me it didn't load in English when they selected English, then it didn't load in English.

I have requested log files from such a session, as those I do need to see.

Also, @peanutbutterandcrackers, am I correct in assuming that when OpenShot failed to display en_US as selected, the language that was used was the one corresponding to then-current system locale? So, the language preference en_US was identical to "Default"?

Testing now, I am seeing the same as you. Apologies, I honestly thought I'd tested for that condition before opening the PR. But, this is why feedback from other testers is so important.

Technical digression

The other thing I find odd is that even Qt translations are never reported as being loaded, for en_US — despite the fact that they're present in the /usr/share/qt5/translations/ directory on my system. For example, even though /usr/share/qt5/translations/qtbase_en.qm exists, I never see logs like:

    language:INFO Attempting to load qtbase_en_US in '/usr/share/qt5/translations'
    language:INFO Successfully loaded qtbase_en_US in '/usr/share/qt5/translations'

With French-Canadian, the following is shown due to the presence of qtbase_fr.qm:

    language:INFO Attempting to load qtbase_fr_CA in '/usr/share/qt5/translations'
    language:INFO Successfully loaded qtbase_fr_CA in '/usr/share/qt5/translations'

So it looks like Qt may be doing some special-casing of its own. I do note that qtbase_en.qm is only 16 bytes long, so most likely it contains either no data at all, or an "ignore me" type instruction. Maybe that's why it doesn't "successfully" load, even though it's present.

Still, I'll look into it. Maybe it would be useful to do something similar with an OpenShot.en_US.qm of our own.

This change _properly_ special-cases for `en_US` as the preference
locale, by completely **replacing** the list of possible translations
to load. This ensures that no language other than `en_US` is pulled
in by the QTranslator. Since we _know_ that we have full support
for `en_US` and always will, there's no need to keep a list of other
possible languages from the locale/environment to fall back on.

Add OpenShot's language preference to the locale-environment logging.
@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 13, 2018

I believe this is now fixed, @peanutbutterandcrackers if you'd be so kind as to re-test?

Launching with `launch.py --lang <code>` will override all environment
and preference language selections, and force OpenShot to use the
specified language code. (It must be one of the codes found in the
`info.SUPPORTED_LANGUAGES` list, as displayed using `--list-languages`.)
QDirIterator doesn't allow any control over ordering, so we use a
QDir to reference ":/locale", and then pull out the `entryList()`
with sorting set to `QDir.Name`.

This will leave `SUPPORTED_LANGUAGES` sorted English-alphabetically
by country code after `en_US`, e.g.:

```console
$ src/launch.py --list-lang
Loaded modules from current directory: /home/ferd/rpmbuild/REPOS/openshot-qt/src
Supported Languages:
         en_US  American English
           ady
            af  Afrikaans
            am  አማርኛ
            ar  العربية
           ast  Asturianu
            az  Azərbaycan
            be  Беларуская
           ber
            bg  Български
            bn  বাংলা
            br  Brezhoneg
            bs  Bosanski
            ca  Català
   ca@valencia  Català
           ckb  کوردیی ناوەندی
            cs  Čeština
            cy  Cymraeg
            da  Dansk
            de  Deutsch
            el  Ελληνικά
         en_AU  Australian English
         en_CA  Canadian English
         en_GB  British English
            eo
            es  Español De España
            et  Eesti
            eu  Euskara
            fa  فارسی
            fi  Suomi
           fil  Filipino
            fo  Føroyskt
            fr  Français
         fr_CA  Français Canadien
            fy  West-Frysk
           gaa
            gl  Galego
            he  עברית
            hi  हिन्दी
            hr  Hrvatski
            hu  Magyar
            hy  Հայերեն
            ia
            id  Indonesia
            is  Íslenska
            it  Italiano
            ja  日本語
            jv
            ka  ქართული
           kab  Taqbaylit
            kk  Қазақ Тілі
            kn  ಕನ್ನಡ
            ko  한국어
            ku
            ky  Кыргызча
            la
            lt  Lietuvių
            lv  Latviešu
            mk  Македонски
            ml  മലയാളം
            mn  Монгол
            ms  Bahasa Melayu
            my  မြန်မာ
           nap
            nb  Norsk Bokmål
            ne  नेपाली
            nl  Nederlands
            nn  Nynorsk
            oc
            pa  ਪੰਜਾਬੀ
            pl  Polski
            pt  Português
         pt_BR  Português
            ro  Română
            ru  Русский
            se  Davvisámegiella
           shn
            si  සිංහල
            sk  Slovenčina
            sl  Slovenščina
            sq  Shqip
            sr  Српски
      sr@latin  Српски
            sv  Svenska
            ta  தமிழ்
            te  తెలుగు
            th  ไทย
            tr  Türkçe
            tt
            ug  ئۇيغۇرچە
            uk  Українська
            uz  O‘Zbek
            vi  Tiếng Việt
         zh_CN  简体中文
         zh_HK  繁體中文
         zh_TW  繁體中文
```
@peanutbutterandcrackers
Copy link
Contributor

peanutbutterandcrackers commented Jul 13, 2018

Neat! It works great now! Just like it ought to. I even changed to two different system locales and it respected the choice for all languages this time around - even en_US. Awesome! 👍

Not attaching logs and stuff, then. 😄

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 13, 2018

Not attaching logs and stuff, then.

Nope, no need.

Neat! It works great now! Just like it ought to. I even changed to two different system locales and it respected the choice for all languages this time around - even en_US. Awesome!

Cool, glad to hear it! Thanks for looking at that.

I got more heavy-handed about just throwing away the system locale information in certain situations, including when the preference is set to en_US. (Or when any language is selected on the command line with --lang). Because there's no actual translation file to load when we want to use en_US for the interface, it can only be "enabled" successfully by eliminating all other potential languages, since the loader goes through that list and only halts if it succeeds in loading an OpenShot translation file — which it never will, for en_US.

The only other workable approach would be to actually distribute a "dummy" OpenShot.en_US.qm "translation" file for American English, which the translation system could then load when English is the selected language. But it makes more sense to just not even try, since we know the interface starts out already "translated" into en_US.

I'm going to go through and check over the remaining worries I have about how this will handle missing translation data files, to make sure that can't cause any problems. Once I've got those bases covered, I'll remove the WIP tag and present this as eligible for testing and merging.

Though I was really hoping to get a read on how @jonoomph feels about the pathname changes, move to a resource file, and etc. Because if nothing else, this will require changes to:

  1. The build server, so it packages the translation files with their new names and locations, and also so it includes the compiled resource file.
  2. The process for managing translations, since the compiled translation dictionaries have to be stored in a new location and using a completely different directory structure and filename convention.
  3. The process for updating translations, which will need to perform a pyrcc5 resource-compilation as its last step.

The things that might be problematic can still be changed, if I'm told they're problematic. But I'm loathe to just assume they aren't purely on the basis of no objections, when that lack of objections means it hasn't even been looked at. And I hate the idea that merging this would amount to pulling the rug out from under him the next time he looks at translations, unless he's been made aware of the changes.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 15, 2018

Maybe the solution here is to create an EN.po on Launchpad and have that built into a EN.qm file.

Oh, it's already solved, as of my last commits. No need for that. *I'm* still idly curious what Qt is doing with its own English "translations", but that's no longer important for OpenShot itself.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 15, 2018

I did some digging on this. I'm guessing that it uses contributor .po files/changes from Launchpad and builds those into the .qm files.

Yeah, that's as far as I'd gotten, too. But it raises more questions than it answers. Those files are the result of the translators' work, but how are the un-filled-out .po templates (I think that's a .pot file?) generated from the OpenShot source? How are the .po files compiled into .qms? Because, that's the part that requires modification as a result of this change.

My preference would be to have those .po files checked into the tree, and build them into .qm there. Because OpenShot-qt doesn't have a build process, I can understand the value of checking in the compiled files, but it doesn't seem to me like that's a reason not to have the tools for generating them and the .po inputs in the tree as well.

This was part of my "reduce logspam" initiative, but since the
file is changing so radically, I'd rather do this here instead.
The commit currently part of the logspam branch will no longer
apply after this is merged.
@mnwstl
Copy link

mnwstl commented Jul 15, 2018

OK! I will delete this comment and I will write in the correct location.
I've run some tests and I have some more information.

@peanutbutterandcrackers
Copy link
Contributor

@ferdnyc - Would you like me to guide @mnwstl into testing the changes from your branch or should I just ask him to attach the logs, sir?

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 16, 2018

Neither. @mnwstl please open an Issue for your problem, and we'll look into it.

(Logs won't be useful, trust me. Not without getting more information first. This branch will not change anything regarding the language labeling, though it may fix the "pt_BR comes up in English" thing if that's an instance of #1521, which I suspect it is. However, we should work to understand the other half of the issue first. And that should be done in the correct place, not on this PR.)

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 16, 2018

Actually it looks like this is already reported, in #1793. So, @mnwstl, if you feel that report covers the issues you're having (combined with #1521), rather than opening a new issue you can also just use the Subscribe button at #1793 to follow updates there, and/or add any additional info by leaving a comment there.

Much like `src/images/Makefile`, the `src/language/Makefile` rules will
now also trigger recompilation of `openshot_lang.py` whenever any of the
`*.qm` files is updated, not just when `openshot_lang.qrc` is edited.

Also checking in recompiled `openshot_lang.py`.
@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 22, 2018

So, I really think this branch is 100% ready, and would solve a lot of i18n problems for our non-English-speaking users. It awaits only @jonoomph 's attention to sort out the build-process side of things. @DylanC , @peanutbutterandcrackers , any hope of moving this forward?

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 22, 2018

For the record, the substantive work on this was completed 2018-07-14 @ 8:29am, since then I've just been buffing the fenders, polishing the windscreen, and detailing the interior, simply to keep busy on it and prevent myself forgetting it's here.

@DylanC
Copy link
Collaborator

DylanC commented Jul 23, 2018

@ferdnyc - I'm happy enough to merge it but I've been holding back due to the possible build system breakage. Maybe its best to just merge and get the build system fixed asap. At least our side of things will be completed. (and you can stop polishing the windscreen) 😆

@DylanC DylanC merged commit c7eecd3 into OpenShot:develop Jul 23, 2018
@DylanC
Copy link
Collaborator

DylanC commented Jul 23, 2018

@jonoomph - Can you update the required paths in the build scripts? Ask @ferdnyc if you need more details.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 23, 2018

Yeah, honestly I think probably the build system won't break... might not break... but what will definitely break is the translation support in all of the builds.

With this merged but the builds not updated, every OpenShot build that's churned out is going to be American English-only. Because even though they'll still contain the src/locale directory, those files are no longer used in the code. The src/language/ directory that the new code does load translations from, that directory isn't going to be included in the builds until the build process is updated.

(Once that happens, I can delete src/locale in a follow-up PR. At that point it'll no longer be needed. Deleting those files right now, I'm certain would break the builds, which expect to find them there even though they're no longer being used.)

@DylanC
Copy link
Collaborator

DylanC commented Jul 23, 2018

@ferdnyc - Oh that's alright, I think we can cope without the translations in the daily builds for a little while. I'm sure Jonathan will get back on this in a reasonable amount of time.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 23, 2018

Actually, I take it back. Looking at the build scripts I can find, it doesn't appear that specific directories are managed — everything inside src/ seems like it's just sucked up wholesale. So, things may just work as-is, though that's merely a guess.

The only exception is build-mac-dmg.sh, assuming that's even in use, which needed handling added for the language directory. I'm about to submit a PR to do that, though like I said I don't even know if that script is actually in use.

@ferdnyc ferdnyc deleted the translation branch July 26, 2018 06:00
@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 26, 2018

@DylanC @peanutbutterandcrackers

Well, good news on the daily builds front: I decided to pull down the most recent Linux AppImage, and near as I can tell it's built with my tutorial code, my new translation-loading code, and even my new command-line processing, all working as expected.

The language dir was automatically pulled into the build, I guess it just grabs every single file in the repo indiscriminately. (That'd never fly in Fedora packaging, but here its overzealousness works in our favor.)

Heck, the AppImage loader even passes the commandline flags along to OpenShot1 — you can run ./OpenShot-v2.4.2-dev1-1532505391-53c400da-554d9f2f-x86_64.AppImage --list-languages to get a list of the supported languages inside the AppImage2, and I successfully started OpenShot in French Canadian with ./OpenShot-v2.4.2-dev1-1532505391-53c400da-554d9f2f-x86_64.AppImage --lang fr_CA

Notes

  1. Though, irritatingly, the AppRun loader does still show the stupid "Do you want to install a desktop launcher?" question every. f'ing. time, even if all you want to do is get the command-line --help or get a list of the language files. That's super annoying, I would love to eliminate that dialog box somehow, or at least make it a one-time-only question that recorded your answer and left you alone after that point.
  2. I did notice that --list-languages crashed at the end of the list with a Python traceback, because I guess exit() is either disabled or overridden by the cx_Freeze environment. So, I'll have to figure out a Right Way™ to code around that for the cx_Freeze environment.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Jul 26, 2018

P.S> I guess the next step, in terms of the translations, is now to submit the PR that deletes src/locale... though I'd really like to get a go-ahead from @jonoomph that it's not still needed on the other end of the translations process, first. (Not that the files couldn't simply be retrieved from the commit history, if needed... but, still... I'd feel better knowing it wasn't going to break anyone's process.)

@jonoomph
Copy link
Member

This is now blocking the release of 2.4.3. I might need to revert this PR, if I can't find a sane way of updating and testing languages on all 3 OSes. I also don't understand the moving, and converting all *.qm files into the a python resource file, which seems to simply duplicate our translations files, and add new steps for creating releases? Just confused and trying to sort through this now that it's been merged.

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Sep 19, 2018

This is now blocking the release of 2.4.3. I might need to revert this PR, if I can't find a sane way of updating and testing languages on all 3 OSes.

That part should be unchanged, as I understand it, though you may have to take me through more of the process so I can understand how I've impacted it. I tried to address everything I could see in how translations were managed.

My more-recent PR #2103 (the one that did pull the trigger on deleting locale):

  1. moved the OpenShot.pot file into language/
  2. updated the generate_ and test_ Python scripts to run from that path instead (and appeared to function, though I didn't check in an updated OpenShot.pot because it would've changed all the paths from your filesystem to mine)
  3. Updated every use of locale I could find in the source tree to eliminate it (language had already been added where necessary), so that nothing depends on the locale/ dir.

The reason for the rename was three-fold:

  1. It allowed me to do this without disturbing locale/ until the new tree was committed and in use
  2. The new resource file is loaded in src/classes/language.py with from language import openshot_lang, and since locale is both a standard Python module and imported in that same file with an import locale on a previous line, from locale import openshot_lang wouldn't have worked.
  3. (And, also, more explaining the choice of name than a reason for it, it matches the name of the src/classes/language.py file that consumes its contents.)

I also don't understand the moving, and converting all *.qm files into the a python resource file, which seems to simply duplicate our translations files, and add new steps for creating releases?

It does, agreed, but that wasn't the motivator or the sole result.

The reason for the resource file and the consolidation of the .qm files in a single directory was that it allowed me to leverage a lot of Qt's automatic translation loading and fallback features, which previously were being duplicated in the code. That part's true independent of the resource file, Qt just expects to have translation files in a single directory named <namespace><lang>_(<country_code>?).qm, and for example when it's passed the language en_GB it will automatically attempt to load OpenShot.en_GB.qm, then OpenShot.en.qm (and other forms, IIRC) without us needing to do anything at all.

I actually wish I'd named the files like openshot_en_GB.qm which is the even more standard form, and would've allowed me to eliminate all of the special-case language discovery code in our Fedora packaging. It automatically scans for files of that format. Unfortunately, I wasn't aware of that until later.

The compiled resources can be accessed using a path of ":/locale/" which Qt understands as a non-filesystem path. That's supposedly much more efficient, and it allowed me to avoid worrying about any other contents of the directory, since it'll only look at the resources there. The resource file is entirely optional, and in its absence the code should fall back to directly accessing the locale/OpenShot*.qm files. (However, if it's present it will only use the resources compiled into it, so there's a tradeoff there.)

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Sep 19, 2018

Oh, and of course, the main motivation for doing all this is that translation-file loading was breaking in various corner cases, for various users, and every one of the various attempts to patch it back into correct functionality broke things for a different group of users. The too-much-manual-logic-not-enough-letting-Qt-do-its-job nature of the old code was, in my evaluation, the core of the problems. i18n is hard to do right, and just leaving it to Qt struck me as the better part of valor.

This PR, as far as I've been made aware, fixed nearly all of the issues for nearly all of the users, the only remaining one being a problem with the pr_BR / pr translations falling back to the wrong Portuguese in Brazil, for reasons that never quite got sorted out. (But, in the grand scheme of things, that's the lesser of two weevils, since Brazilians have a lot easier time understanding European Portuguese than Russians have understanding English, and OpenShot was loading in English for a lot of international users, or loading in local languages when they SELECTED English for a lot of others.)

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Sep 19, 2018

Building info.SUPPORTED_LANGUAGES quickly, as a filesystem operation in a single directory (or using the resource file, if present) is important because the command line features of src/launch.py execute before the GUI even starts. --language-list and --lang allow the user to (respectively) list all of the locales OpenShot has translations available for, and force a language selection in a last-resort manner that should always, always work, even in cases setting environment variables or application preferences doesn't have the expected effect. When debugging future i18n issues (like the pr_BR thing), I'm hopeful that may be useful.

$ openshot-qt --list-languages
Compiled translation resources missing!
Loading translations from: /usr/lib/python3.6/site-packages/openshot_qt/language
Loaded modules from installed directory: /usr/lib/python3.6/site-packages/openshot_qt
Compiled translation resources missing!
Loading translations from: /usr/lib/python3.6/site-packages/openshot_qt/language
Supported Languages:
         en_US  American English
           ady  
            af  Afrikaans
            am  አማርኛ
            ar  العربية
           ast  Asturianu
            az  Azərbaycan
            be  Беларуская
           ber  
            bg  Български
            bn  বাংলা
            br  Brezhoneg
            bs  Bosanski
            ca  Català
   ca@valencia  Català
           ckb  کوردیی ناوەندی
            cs  Čeština
            cy  Cymraeg
            da  Dansk
            de  Deutsch
            el  Ελληνικά
         en_AU  Australian English
         en_CA  Canadian English
         en_GB  British English
            eo  
            es  Español De España
            et  Eesti
            eu  Euskara
            fa  فارسی
            fi  Suomi
           fil  Filipino
            fo  Føroyskt
            fr  Français
         fr_CA  Français Canadien
            fy  West-Frysk
           gaa  
            gl  Galego
            he  עברית
            hi  हिन्दी
            hr  Hrvatski
            hu  Magyar
            hy  Հայերեն
            ia  
            id  Indonesia
            is  Íslenska
            it  Italiano
            ja  日本語
            jv  
            ka  ქართული
           kab  Taqbaylit
            kk  Қазақ Тілі
            kn  ಕನ್ನಡ
            ko  한국어
            ku  
            ky  Кыргызча
            la  
            lt  Lietuvių
            lv  Latviešu
            mk  Македонски
            ml  മലയാളം
            mn  Монгол
            ms  Bahasa Melayu
            my  မြန်မာ
           nap  
            nb  Norsk Bokmål
            ne  नेपाली
            nl  Nederlands
            nn  Nynorsk
            oc  
            pa  ਪੰਜਾਬੀ
            pl  Polski
            pt  Português
         pt_BR  Português
            ro  Română
            ru  Русский
            se  Davvisámegiella
           shn  
            si  සිංහල
            sk  Slovenčina
            sl  Slovenščina
            sq  Shqip
            sr  Српски
      sr@latin  Српски
            sv  Svenska
            ta  தமிழ்
            te  తెలుగు
            th  ไทย
            tr  Türkçe
            tt  
            ug  ئۇيغۇرچە
            uk  Українська
            uz  O‘Zbek
            vi  Tiếng Việt
         zh_CN  简体中文
         zh_HK  繁體中文
         zh_TW  繁體中文

(As you can see, my Fedora packaging doesn't yet include the .qrc file, that's something I have to correct when packaging the 2.4.3 release.)

@ferdnyc
Copy link
Contributor Author

ferdnyc commented Sep 19, 2018

Wait, why the hell am I defending code I wrote moths ago, begged you to look at, and heard NOTHING in response? Do what you want, leave me out of it.

@jonoomph
Copy link
Member

@ferdnyc I really appreciate your work and detailed posts! You are a most impressive coder and communicator! It's true I have my hands full and suck at timely reviews, but I'm working hard to improve. We've merged a ton more PRs in the past 2 months, and this one kind of surprised me with how much changed. Parts of this PR seem unneeded from the outside looking in, but I totally understand your logic and appreciate the response tonight.

@jonoomph
Copy link
Member

I do intend to keep this merged, but I will need to modify many of my release scripts and processes to correctly update the translations. In the end, I think this is much better than what we did previously... I just need to rework some infrastructure changes around it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Translation not working again
5 participants