Skip to content
Permalink
Browse files

bug #31411 [Intl] Fix root fallback locale (ro0NL)

This PR was merged into the 3.4 branch.

Discussion
----------

[Intl] Fix root fallback locale

| Q             | A
| ------------- | ---
| Branch?       | 3.4
| Bug fix?      | yes
| New feature?  | no
| BC breaks?    | no     <!-- see https://symfony.com/bc -->
| Deprecations? | no
| Tests pass?   | yes (including intl-data group)
| Fixed tickets | #...   <!-- #-prefixed issue number(s), if any -->
| License       | MIT
| Doc PR        | symfony/symfony-docs#... <!-- required for new features -->

We should never return "root" as a fallback locale for the "root" locale itself.

While at it, i realized the alias meta files are pointless :)

4.2) ro0NL@b9fc8b7
4.3) ro0NL@922a1eb

Commits
-------

11ff24a [Intl] Fix root fallback locale
  • Loading branch information...
fabpot committed May 8, 2019
2 parents 6c6f76f + 11ff24a commit 75d1dd45e5c9909ecc3d3a2db3af138c24b0ca12
Showing with 68 additions and 199 deletions.
  1. +17 −29 src/Symfony/Component/Intl/Data/Generator/LocaleDataGenerator.php
  2. +15 −6 src/Symfony/Component/Intl/Data/Generator/RegionDataGenerator.php
  3. +4 −4 src/Symfony/Component/Intl/Locale.php
  4. +8 −37 src/Symfony/Component/Intl/Resources/bin/update-data.php
  5. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/az_AZ.json
  6. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/bs_BA.json
  7. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/en_NH.json
  8. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/en_RH.json
  9. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/ff_CM.json
  10. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/ff_GN.json
  11. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/ff_MR.json
  12. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/ff_SN.json
  13. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/in.json
  14. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/in_ID.json
  15. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/iw.json
  16. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/iw_IL.json
  17. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/mo.json
  18. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/no.json
  19. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/no_NO.json
  20. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/no_NO_NY.json
  21. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/pa_IN.json
  22. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/pa_PK.json
  23. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sh.json
  24. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sh_BA.json
  25. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sh_CS.json
  26. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sh_YU.json
  27. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_BA.json
  28. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_CS.json
  29. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_Cyrl_CS.json
  30. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_Cyrl_YU.json
  31. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_Latn_CS.json
  32. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_Latn_YU.json
  33. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_ME.json
  34. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_RS.json
  35. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_XK.json
  36. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/sr_YU.json
  37. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/tl.json
  38. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/tl_PH.json
  39. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/uz_AF.json
  40. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/uz_UZ.json
  41. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/zh_CN.json
  42. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/zh_HK.json
  43. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/zh_MO.json
  44. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/zh_SG.json
  45. +0 −3 src/Symfony/Component/Intl/Resources/data/locales/zh_TW.json
  46. +24 −0 src/Symfony/Component/Intl/Tests/LocaleTest.php
@@ -14,9 +14,6 @@
use Symfony\Component\Filesystem\Filesystem;
use Symfony\Component\Intl\Data\Bundle\Compiler\BundleCompilerInterface;
use Symfony\Component\Intl\Data\Bundle\Reader\BundleEntryReaderInterface;
use Symfony\Component\Intl\Data\Provider\LanguageDataProvider;
use Symfony\Component\Intl\Data\Provider\RegionDataProvider;
use Symfony\Component\Intl\Data\Provider\ScriptDataProvider;
use Symfony\Component\Intl\Data\Util\LocaleScanner;
use Symfony\Component\Intl\Exception\MissingResourceException;
use Symfony\Component\Intl\Locale;
@@ -31,23 +28,11 @@
*/
class LocaleDataGenerator extends AbstractDataGenerator
{
private $languageDataProvider;
private $scriptDataProvider;
private $regionDataProvider;
private $locales;
private $localeAliases;
private $fallbackMapping;
private $fallbackCache = [];
public function __construct(BundleCompilerInterface $compiler, $dirName, LanguageDataProvider $languageDataProvider, ScriptDataProvider $scriptDataProvider, RegionDataProvider $regionDataProvider)
{
parent::__construct($compiler, $dirName);
$this->languageDataProvider = $languageDataProvider;
$this->scriptDataProvider = $scriptDataProvider;
$this->regionDataProvider = $regionDataProvider;
}
/**
* {@inheritdoc}
*/
@@ -66,8 +51,12 @@ protected function scanLocales(LocaleScanner $scanner, $sourceDir)
protected function compileTemporaryBundles(BundleCompilerInterface $compiler, $sourceDir, $tempDir)
{
$filesystem = new Filesystem();
$filesystem->mkdir($tempDir.'/lang');
$filesystem->mkdir([
$tempDir.'/lang',
$tempDir.'/region',
]);
$compiler->compile($sourceDir.'/lang', $tempDir.'/lang');
$compiler->compile($sourceDir.'/region', $tempDir.'/region');
}
/**
@@ -83,19 +72,14 @@ protected function preGenerate()
*/
protected function generateDataForLocale(BundleEntryReaderInterface $reader, $tempDir, $displayLocale)
{
// Generate aliases, needed to enable proper fallback from alias to its
// target
// Don't generate aliases, as they are resolved during runtime
if (isset($this->localeAliases[$displayLocale])) {
return ['%%ALIAS' => $this->localeAliases[$displayLocale]];
return;
}
// Generate locale names for all locales that have translations in
// at least the language or the region bundle
try {
$displayFormat = $reader->readEntry($tempDir.'/lang', $displayLocale, ['localeDisplayPattern']);
} catch (MissingResourceException $e) {
$displayFormat = $reader->readEntry($tempDir.'/lang', 'root', ['localeDisplayPattern']);
}
$displayFormat = $reader->readEntry($tempDir.'/lang', $displayLocale, ['localeDisplayPattern']);
$pattern = $displayFormat['pattern'] ?? '{0} ({1})';
$separator = $displayFormat['separator'] ?? '{0}, {1}';
$localeNames = [];
@@ -110,7 +94,7 @@ protected function generateDataForLocale(BundleEntryReaderInterface $reader, $te
// Each locale name has the form: "Language (Script, Region, Variant1, ...)
// Script, Region and Variants are optional. If none of them is
// available, the braces are not printed.
$localeNames[$locale] = $this->generateLocaleName($locale, $displayLocale, $pattern, $separator);
$localeNames[$locale] = $this->generateLocaleName($reader, $tempDir, $locale, $displayLocale, $pattern, $separator);
} catch (MissingResourceException $e) {
// Silently ignore incomplete locale names
// In this case one should configure at least one fallback locale that is complete (e.g. English) during
@@ -158,22 +142,26 @@ protected function generateDataForMeta(BundleEntryReaderInterface $reader, $temp
/**
* @return string
*/
private function generateLocaleName($locale, $displayLocale, $pattern, $separator)
private function generateLocaleName(BundleEntryReaderInterface $reader, $tempDir, $locale, $displayLocale, $pattern, $separator)
{
// Apply generic notation using square brackets as described per http://cldr.unicode.org/translation/language-names
$name = str_replace(['(', ')'], ['[', ']'], $this->languageDataProvider->getName(\Locale::getPrimaryLanguage($locale), $displayLocale));
$name = str_replace(['(', ')'], ['[', ']'], $reader->readEntry($tempDir.'/lang', $displayLocale, ['Languages', \Locale::getPrimaryLanguage($locale)]));
$extras = [];
// Discover the name of the script part of the locale
// i.e. in zh_Hans_MO, "Hans" is the script
if ($script = \Locale::getScript($locale)) {
$extras[] = str_replace(['(', ')'], ['[', ']'], $this->scriptDataProvider->getName($script, $displayLocale));
$extras[] = str_replace(['(', ')'], ['[', ']'], $reader->readEntry($tempDir.'/lang', $displayLocale, ['Scripts', $script]));
}
// Discover the name of the region part of the locale
// i.e. in de_AT, "AT" is the region
if ($region = \Locale::getRegion($locale)) {
$extras[] = str_replace(['(', ')'], ['[', ']'], $this->regionDataProvider->getName($region, $displayLocale));
if (!RegionDataGenerator::isValidCountryCode($region)) {
throw new MissingResourceException('Skipping "'.$locale.'" due an invalid country.');
}
$extras[] = str_replace(['(', ')'], ['[', ']'], $reader->readEntry($tempDir.'/region', $displayLocale, ['Countries', $region]));
}
if ($extras) {
@@ -48,6 +48,20 @@ class RegionDataGenerator extends AbstractDataGenerator
*/
private $regionCodes = [];
public static function isValidCountryCode($region)
{
if (isset(self::$blacklist[$region])) {
return false;
}
// WORLD/CONTINENT/SUBCONTINENT/GROUPING
if (ctype_digit($region) || \is_int($region)) {
return false;
}
return true;
}
/**
* {@inheritdoc}
*/
@@ -125,12 +139,7 @@ protected function generateRegionNames(ArrayAccessibleResourceBundle $localeBund
$regionNames = [];
foreach ($unfilteredRegionNames as $region => $regionName) {
if (isset(self::$blacklist[$region])) {
continue;
}
// WORLD/CONTINENT/SUBCONTINENT/GROUPING
if (ctype_digit($region) || \is_int($region)) {
if (!self::isValidCountryCode($region)) {
continue;
}
@@ -31,7 +31,7 @@ final class Locale extends \Locale
* The default fallback locale is used as fallback for locales that have no
* fallback otherwise.
*
* @param string $locale The default fallback locale
* @param string|null $locale The default fallback locale
*
* @see getFallback()
*/
@@ -43,7 +43,7 @@ public static function setDefaultFallback($locale)
/**
* Returns the default fallback locale.
*
* @return string The default fallback locale
* @return string|null The default fallback locale
*
* @see setDefaultFallback()
* @see getFallback()
@@ -70,7 +70,7 @@ public static function getFallback($locale)
if (\function_exists('locale_parse')) {
$localeSubTags = locale_parse($locale);
if (1 === \count($localeSubTags)) {
if (self::$defaultFallback === $localeSubTags['language']) {
if ('root' !== self::$defaultFallback && self::$defaultFallback === $localeSubTags['language']) {
return 'root';
}
@@ -98,7 +98,7 @@ public static function getFallback($locale)
return substr($locale, 0, $pos);
}
if (self::$defaultFallback === $locale) {
if ('root' !== self::$defaultFallback && self::$defaultFallback === $locale) {
return 'root';
}
@@ -11,18 +11,13 @@
use Symfony\Component\Filesystem\Filesystem;
use Symfony\Component\Intl\Data\Bundle\Compiler\GenrbCompiler;
use Symfony\Component\Intl\Data\Bundle\Reader\BundleEntryReader;
use Symfony\Component\Intl\Data\Bundle\Reader\JsonBundleReader;
use Symfony\Component\Intl\Data\Bundle\Writer\JsonBundleWriter;
use Symfony\Component\Intl\Data\Generator\CurrencyDataGenerator;
use Symfony\Component\Intl\Data\Generator\GeneratorConfig;
use Symfony\Component\Intl\Data\Generator\LanguageDataGenerator;
use Symfony\Component\Intl\Data\Generator\LocaleDataGenerator;
use Symfony\Component\Intl\Data\Generator\RegionDataGenerator;
use Symfony\Component\Intl\Data\Generator\ScriptDataGenerator;
use Symfony\Component\Intl\Data\Provider\LanguageDataProvider;
use Symfony\Component\Intl\Data\Provider\RegionDataProvider;
use Symfony\Component\Intl\Data\Provider\ScriptDataProvider;
use Symfony\Component\Intl\Intl;
use Symfony\Component\Intl\Locale;
use Symfony\Component\Intl\Util\GitRepository;
@@ -171,27 +166,13 @@
$compiler = new GenrbCompiler($genrb, $genrbEnv);
$config = new GeneratorConfig($sourceDir.'/data', $icuVersionInDownload);
$jsonDir = dirname(__DIR__).'/data';
$targetDirs = [$jsonDir];
$workingDirs = [$jsonDir];
$config->addBundleWriter($jsonDir, new JsonBundleWriter());
echo "Starting resource bundle compilation. This may take a while...\n";
$filesystem->remove($workingDirs);
foreach ($workingDirs as $targetDir) {
$filesystem->mkdir([
$targetDir.'/'.Intl::CURRENCY_DIR,
$targetDir.'/'.Intl::LANGUAGE_DIR,
$targetDir.'/'.Intl::LOCALE_DIR,
$targetDir.'/'.Intl::REGION_DIR,
$targetDir.'/'.Intl::SCRIPT_DIR,
]);
}
// We don't want to use fallback to English during generation
Locale::setDefaultFallback(null);
Locale::setDefaultFallback('root');
echo "Generating language data...\n";
@@ -215,14 +196,7 @@
echo "Generating locale data...\n";
$reader = new BundleEntryReader(new JsonBundleReader());
$generator = new LocaleDataGenerator(
$compiler,
Intl::LOCALE_DIR,
new LanguageDataProvider($jsonDir.'/'.Intl::LANGUAGE_DIR, $reader),
new ScriptDataProvider($jsonDir.'/'.Intl::SCRIPT_DIR, $reader),
new RegionDataProvider($jsonDir.'/'.Intl::REGION_DIR, $reader)
);
$generator = new LocaleDataGenerator($compiler, Intl::LOCALE_DIR);
$generator->generateData($config);
echo "Resource bundle compilation complete.\n";
@@ -238,18 +212,15 @@
GIT_INFO;
foreach ($targetDirs as $targetDir) {
$gitInfoFile = $targetDir.'/git-info.txt';
$gitInfoFile = $jsonDir.'/git-info.txt';
file_put_contents($gitInfoFile, $gitInfo);
file_put_contents($gitInfoFile, $gitInfo);
echo "Wrote $gitInfoFile.\n";
echo "Wrote $gitInfoFile.\n";
$versionFile = $targetDir.'/version.txt';
$versionFile = $jsonDir.'/version.txt';
file_put_contents($versionFile, "$icuVersionInDownload\n");
echo "Wrote $versionFile.\n";
}
file_put_contents($versionFile, "$icuVersionInDownload\n");
echo "Wrote $versionFile.\n";
echo "Done.\n";

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.
Oops, something went wrong.

0 comments on commit 75d1dd4

Please sign in to comment.
You can’t perform that action at this time.