Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

Before Release: add Release-only locales to list #7104

Closed
1 of 3 tasks
liuche opened this issue Dec 12, 2019 · 27 comments
Closed
1 of 3 tasks

Before Release: add Release-only locales to list #7104

liuche opened this issue Dec 12, 2019 · 27 comments
Labels
E2 Estimation Point: easy, half a day to 2 days eng:qa:verified QA Verified 🌐 L10N Localization, translation, strings, ..

Comments

@liuche
Copy link
Contributor

liuche commented Dec 12, 2019

Why/User Benefit/User Problem

Currently, we will bundle the same locales in Nightly and Preview, regardless of what status of translation they are in.

However, before we make our main release, we need to make sure only release-approved locales are included. They should be on Nightly and Beta, but not Production builds.

Acceptance Criteria (how do I know when I’m done?)

Before July 18:

  • Verify (or implement) the functionality to have separate locales in Nightly/Beta vs Production. See the locale list in l10n.toml (see checklist in Allow different locales for Nightly and Release #3088). i.e. make a release build that doesn't have all the locales, and verify only those locales show up in the build.

On July 20:

L10N strings uplift discussion
L10N locale doc

┆Issue is synchronized with this Jira Task

@liuche liuche added feature request 🌟 New functionality and improvements must 🌐 L10N Localization, translation, strings, .. and removed feature request 🌟 New functionality and improvements labels Dec 12, 2019
@liuche liuche added this to Prioritized Feature Backlog in Fenix Sprint Kanban Dec 12, 2019
@liuche liuche added the 🙅 waiting Issues that are blocked or has dependencies that are not ready label Dec 12, 2019
@vesta0 vesta0 added the E2 Estimation Point: easy, half a day to 2 days label Dec 17, 2019
@vesta0 vesta0 removed the must label Mar 31, 2020
@liuche
Copy link
Contributor Author

liuche commented Apr 7, 2020

@vesta0 will we want to filter out partially-translated locales after Beta Migration, or do we want to keep all the translated strings? Filtering will mean that users in that locale will see English-only, rather than a (low percentage) partial translation. We'll have to contact @Pike soon if we want to get that list.

@vesta0
Copy link
Collaborator

vesta0 commented Apr 17, 2020

@liuche do we how what percentage of strings have been translated? How far are we from Fennec parity in term of localization?

@liuche
Copy link
Contributor Author

liuche commented Apr 17, 2020

Here's the current progress on Pontoon (and I think Fenix is the main Android project): https://pontoon.mozilla.org/projects/android-l10n/

What do you think, @Pike (or Jeff Beatty but I don't know his github) - do we know enough about the localization languages progress compared to Fennec to make this release list?

@Pike
Copy link
Contributor

Pike commented Apr 17, 2020

I'll punt this to @gueroJeff

@gueroJeff
Copy link

gueroJeff commented Apr 20, 2020

Here's the current progress on Pontoon (and I think Fenix is the main Android project): https://pontoon.mozilla.org/projects/android-l10n/

This is the link I've been using: https://pontoon.mozilla.org/projects/android-l10n/tags/fenix/

What do you think, @Pike (or Jeff Beatty but I don't know his github) - do we know enough about the localization languages progress compared to Fennec to make this release list?

We're not going to reach parity, just by virtue of some locales shipping on Fennec chose not to participate in localizing Fenix. The metric we're using is average completion across all locales. Fennec was at 90% across all locales, Fenix is currently at 75% average.

To answer the earlier question, all products ship partial localizations. That's part of the risk that accompanies community localization. Fenix should follow that practice too.

@vesta0 vesta0 moved this from Prioritized Feature Backlog to Prioritized Eng Backlog in Fenix Sprint Kanban Apr 28, 2020
@liuche liuche moved this from Prioritized Eng Backlog to Inbox in Fenix Sprint Kanban Jun 3, 2020
@liuche
Copy link
Contributor Author

liuche commented Jun 5, 2020

@gueroJeff will review the locales and decide on what the cutoff for inclusion with the Release build. Once he gives us the criteria (and the locales?) we'll be able to update the script to include the release locales.

@gueroJeff
Copy link

I'm going to set the cutoff at 70% complete. There's some automation the Bedrock team implemented to determine how complete a localization is and then automatically push it to prod when it reaches a certain threshold. Similar automation here would be preferred, but if we're unable to get that for the launch, I can produce a manual list at l10n freeze.

@liuche liuche moved this from Inbox to Bugs before Release in Fenix Sprint Kanban Jun 10, 2020
@liuche liuche moved this from Bugs before Release to Sprint in Fenix Sprint Kanban Jun 22, 2020
@liuche liuche added 🙅 waiting Issues that are blocked or has dependencies that are not ready and removed 🙅 waiting Issues that are blocked or has dependencies that are not ready labels Jun 22, 2020
@liuche
Copy link
Contributor Author

liuche commented Jun 25, 2020

We'll need to uplift the new localizations up until July 18, which will be after code freeze (which is okay because it's new strings only). After we know the localization rate, we'll be able to determine the locales to include/filter.

However, right now, we can still figure out how to make this work for the current set of locales, and just update the specific set of locales once we know their completion rates.

@liuche liuche added 🙅 waiting Issues that are blocked or has dependencies that are not ready and removed 🙅 waiting Issues that are blocked or has dependencies that are not ready labels Jun 25, 2020
@liuche liuche added 79ReleaseBlocker and removed 🙅 waiting Issues that are blocked or has dependencies that are not ready labels Jul 13, 2020
@person808
Copy link
Contributor

person808 commented Jul 16, 2020

I downloaded an apk from here since it looks like we run a script in taskcluster to remove locales that aren't in the locale list. Then I manually went through the languages and checked off the locales I saw in the l10n config. Some locales were missing, maybe related to #12613?

The missing locales are kn, ms, ne-NP, nv, uz. Of these only kn appears to have more than 70% completion. ms, ne-NP, and nv have 0% completion so they are probably missing because the translation strings don't even exist. uz has 13% completion so I'm not sure why it doesn't show up (although we should remove it from the locale list anyway)

@Delphine
Copy link
Contributor

I currently don't see either Uzbek (uz) nor Kannada (kn) on the latest Nightly. They both have partial localizations, so this is maybe also related to #12613?

@person808
Copy link
Contributor

We don't have the strings for Uzbek or Kannada in the app yet. Looks like they're just missing due to lag in getting the translations pulled in. The TC script to strip locales works on my machine so it looks like the first part of this issue is verified.

person808 added a commit to person808/fenix that referenced this issue Jul 20, 2020
person808 added a commit to person808/fenix that referenced this issue Jul 20, 2020
person808 added a commit to person808/fenix that referenced this issue Jul 20, 2020
person808 added a commit to person808/fenix that referenced this issue Jul 20, 2020
person808 added a commit that referenced this issue Jul 20, 2020
@liuche liuche added the eng:qa:needed QA Needed label Jul 21, 2020
@project-bot project-bot bot moved this from Sprint to Ready for QA in Fenix Sprint Kanban Jul 21, 2020
@liuche
Copy link
Contributor Author

liuche commented Jul 21, 2020

This will be ready for QA once we cut the Release Candidate for the Firefox Production Migration. We will want to verify that in the language picker, only the approved 70%+ locales are included (see delphine's comment above for that list).

@sflorean
Copy link
Contributor

@liuche started testing on 79.0.0 AC 48.0.8 RC2 and a couple of locales are missing, here are some of them: an, bn, bs, cs, dsb, el, eo, et, fa, ff, gd, gn, he, hsb, hy-Am, id, is, kk, kn, lij, lt, ml, mr, my. On what RC should we test this?

@Delphine
Copy link
Contributor

Delphine commented Jul 28, 2020

So I've tested on this 79 build (that I got from https://firefox-ci-tc.services.mozilla.com/tasks/index/mobile.v2.fenix.fennec-production.2020.07.24.revision), and I'm seeing a few locales missing (not all the ones mentioned above):
bn, dsb, he, hsb, id
Has there been an issue cutting off locales for the release?

@person808
Copy link
Contributor

person808 commented Jul 28, 2020

I see dsb and hsb as Lower Sorbian and Upper Sorbian. bn is Bangla I think? I don't see he or id in that apk

@liuche
Copy link
Contributor Author

liuche commented Jul 29, 2020

@person808 any ideas on what might be happening here? are these getting filtered out by accident, or are they not being included/imported at all?

@person808
Copy link
Contributor

person808 commented Jul 29, 2020

Not sure. When I run the filtering script locally on the releases/v79.0.0 branch, the only locales that get removed are bg, in, iw, and uz. Taskcluster should run the scripts for fennec-beta and fennec-production (see kind.yml and look for filter-incomplete-translations)

Edit: Wait the values-id and values-he resource folders appear to be missing

@flodolo
Copy link

flodolo commented Jul 29, 2020

Not that I know a lot about Android, but
https://developer.android.com/reference/java/util/Locale.html

Locale's constructor has always converted three language codes to their earlier, obsoleted forms: he maps to iw, yi maps to ji, and id maps to in. This continues to be the case, in order to not break backwards compatibility.

@Pike
Copy link
Contributor

Pike commented Jul 29, 2020

Those are mapped in https://github.com/Pike/compare-locales/blob/e3eab8131efa2d2a2c7c81827b8ff9d7ac2f2d9a/compare_locales/paths/matcher.py#L13-L19 between project config and disk paths. If either he or id are not shipping, you need to prune the corresponding values-iw or values-in folder.

@sflorean
Copy link
Contributor

Testing results:
"an", Aragonese - displayed
"ar", Arabic - displayed
"ast", Asturian - displayed
"az", Azerbaijani - displayed
"be", Belarusian - displayed
"bn", Bangla - displayed
"br", Breton - displayed
"bs", Bosnian - displayed
"ca", Catalan - displayed
"cak", Cak - displayed
"co", Corsican - displayed
"cs", Cestina - displayed
"cy", Cymraeg - displayed
"da", Dansk - displayed
"de", Deutsch - displayed
"dsb", displayed
"el", Greek - displayed
"en-CA", - displayed
"en-GB", - displayed
"eo", Esperanto - displayed
"es", Espanol - displayed
"es-AR", Espaniol Argentina - diplayed
"es-CL", Espaniol Chile - displayed
"es-ES", Espaniol Espana - displayed
"es-MX", Espaniol Mexico - displayed
"et", Eesti - displayed
"eu", Basque/Euskara - displayed
"fa", Persian - displayed
"ff", Pullaar (fulah) - displayed
"fi", Suomi - displayed
"fr", Francais - displayed
"fy-NL", West-Frysk - displayed
"ga-IE", Gaeilge - displayed
"gd", Scottish Gaelic- displayed
"gn", Guarani - displayed
"gu-IN", Gujarati India - displayed
"he", Hebrew? not displayed
"hi-IN", Hindi India - displayed
"hr", Hrvatski - displayed
"hsb", displayed
"hu", Magyar - displayed
"hy-AM", Armenian - displayed
"id", Indonesian? not displayed
"is", Islenska - displayed
"it", Italian - displayed
"ja", Japanese - displayed
"ka", Georgian - displayed
"kab", Kabyle - displayed
"kk", Kazakh - displayed
"kn", Kannada - displayed
"ko", Korean - displayed
"lij", Ligurian - displayed
"lo", Lao - displayed
"lt", Lithuanian - displayed
"ml", Malayalam - displayed
"mr", Marathi - displayed
"my", Burmese - displayed
"nb-NO", Norsk bokmal - displayed
"nl", Nederlands - displayed
"nn-NO", Nynorsk - displayed
"oc", Occitan - displayed
"pa-IN", Punjabi India - displayed
"pl", Polski - displayed
"pt-BR", Portugues Brazil - displayed
"pt-PT", Portugues Portugal - displayed
"rm", Rumantsch - displayed
"ro", Romanian - displayed
"ru", Russian - displayed
"sk", Slovak - displayed
"sl", Slovenian - displayed
"sq", Shqip - displayed
"sr", Serbian - displayed
"su", Sundanase - displayed
"sv-SE", Svenska Sverige - displayed
"ta", Tamil - displayed
"te", Telugu - displayed
"th", Thai - displayed
"tr", Turkish - displayed
"trs", Trs - displayed
"uk", Ukrainian - displayed
"ur", Urdu - displayed
"vec", Venetian displayed
"vi", Vietnamese - displayed
"zh-CN", Chinese China - displayed
"zh-TW" Chinese Taiwan - displayed

Note:

  • en-US is not in the above list but English (United States) is listed in settings-language
  • we couldn't find "id" (Indonesian?) and "he" (Hebrew?) listed
  • only 84 languages are displayed, including en-US

@liuche @Delphine

@sflorean sflorean removed the eng:qa:needed QA Needed label Jul 29, 2020
@gueroJeff
Copy link

There are also inconsistencies in that list. Several display names are the locale code, not the language name. Several are not in English (e.g., "Polski") and the two Chinese should display as Simplified Chinese (zh-CN) and Traditional Chinese (zh-TW).

@Pike
Copy link
Contributor

Pike commented Jul 29, 2020

We don't have control over the natural names, those are coming from Android (and I think that's a feature).

For Hebrew and Indonesian, we should translate the codes when reading the release locales, probably. I'd not make the config list be different. Filed #13066 on that.

@person808
Copy link
Contributor

person808 commented Jul 29, 2020

Ah I see. Looks like l10n.toml contains "he" and "id" instead of "iw" and "in" so the folders are being deleted. Will file PRs to update the script so its aware of the mappings in the release branch and master

@person808
Copy link
Contributor

I'll close this since the filtering appears to work except for Hebrew and Indonesian. We can verify that they are present when we QA #13066

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
E2 Estimation Point: easy, half a day to 2 days eng:qa:verified QA Verified 🌐 L10N Localization, translation, strings, ..
Projects
Fenix Sprint Kanban
  
Sprint 20.11 Done
Development

No branches or pull requests

8 participants