Parsing of mixed-cased locale ID. #361

kdeldycke · 2016-03-02T11:17:14Z

Following #351, it seems that mixed-cased locale IDs are not properly checked as existing depending on the platform.

The issue lies in babel.localedata.exists() which rely on filesystem assets. As a result, mixed-cased locale IDs are properly parsed on OSX (case-insensitive filesystem) but not on Linux.

The first commit demonstrate the issue depending on the system.
The second commit ensure that locale IDs recognized by Babel are uniques, even if lower-cased.
The third commit fix to the issue.

kdeldycke · 2016-03-02T12:00:55Z

Looks like this issue relates to #283.

codecov-io · 2016-03-02T16:40:06Z

Current coverage is `89.68%`

Merging #361 into master will increase coverage by +0.02% as of bb16d60

@@            master    #361   diff @@
======================================
  Files           23      23       
  Stmts         3794    3801     +7
  Branches         0       0       
  Methods          0       0       
======================================
+ Hit           3402    3409     +7
  Partial          0       0       
  Missed         392     392

Review entire Coverage Diff as of bb16d60

Powered by Codecov. Updated on successful CI builds.

akx · 2016-03-03T16:29:42Z

babel/localedata.py

@@ -24,15 +25,22 @@
 _dirname = os.path.join(os.path.dirname(__file__), 'locale-data')


+def normalize_locale(name):
+    """Return a normalized locale ID or `None` if the ID is not recognized."""


Could you explain what "normalization" means in the docstring?

Sure. Done in https://github.com/kdeldycke/babel/commit/a5cb8317626b71d532f78c8d86c6b0e790ad4b7f .

akx · 2016-03-03T16:30:13Z

babel/localedata.py

-    if name in _cache:
-        return True
-    return os.path.exists(os.path.join(_dirname, '%s.dat' % name))
+    return True if normalize_locale(name) else False


return bool(normalize_locale(name)) :)

To be honest, I'd like to see this normalization thing as a "slow path", if the previous os.path.exists path fails.

Done: https://github.com/kdeldycke/babel/commit/1a6731e2b51747764199da40bb39fcb9354dfe58

akx · 2016-03-03T18:13:38Z

Nice, thanks @kdeldycke!

(CI: I'm not waiting for Appveyor, since most builds completed and some failed due to ephemeral errors.)

Parsing of mixed-cased locale ID.

kdeldycke · 2016-03-03T18:15:14Z

Thanks @akx for the quick merge! 😃

jtwang · 2016-03-03T18:20:53Z

babel/localedata.py

+        if name == locale_id.lower():
+            return locale_id
+
+
 def exists(name):


Wow, greedy method.
I'm really looking forward to #305 !

(just an observation, please ignore :)

jtwang · 2016-03-03T18:24:01Z

babel/localedata.py


    :param name: the locale identifier string
    """
    if name in _cache:
        return True
-    return os.path.exists(os.path.join(_dirname, '%s.dat' % name))
+    file_found = os.path.exists(os.path.join(_dirname, '%s.dat' % name))
+    return True if file_found else bool(normalize_locale(name))


Would iteration over in-mem objects would be faster than disk access? If so, please consider performing the in-memory check first. :)

(super duper minor optimization)

That'd complicate the whole deal unnecessarily, imho. It'd become something like

(1) name in cache?

(2) normalized name in cache?

(3) name in file system?

(4) normalized name in file system?

which would kinda interleave normalization with this "simple" existence checking...

jtwang · 2016-03-03T18:31:59Z

tests/test_localedata.py

+    all_ids = localedata.locale_identifiers()
+    assert len(all_ids) == len(set(all_ids))
+    # Check locale IDs don't collide after lower-case normalization.
+    lower_case_ids = list(map(methodcaller('lower'), all_ids))


IMO

lower_case_ids = [id.lower() for id in all_ids] is more readable, but don't care much.

Yeah, it would be, and more Pythonic, but that's kinda minor, and can be trivially corrected later if need be.

jtwang · 2016-03-03T18:35:41Z

tests/test_localedata.py

+
+
+def test_mixedcased_locale():
+    for l in localedata.locale_identifiers():


Is it necessary to iterate through all ids? Would it be sufficient to spot check some deliberately selected hardcoded values? Eg. 'Fi', 'FI_fi', 'fi_FI', 'fI_fI', 'ZH_hanT_HK'

That's another optimization that could be done (via pytest.parametrize), but I don't think our test suite takes too long to run as it is :)

True that. I just get the heebie jeebies when I see random.choice in unit tests, ha.

jtwang · 2016-03-03T19:00:36Z

Ah, crud, didn't notice this was already merged into master. Feel free to ignore all my comments, none of them are major. :)

akx · 2016-03-03T19:17:31Z

All comments were duly noted and processed, don't fret :D

Test parsing of mixed-cased locale ID.

Loading status checks…

6844ccc

gitmate-bot added the process/pending review label Mar 2, 2016

kdeldycke added 2 commits Mar 2, 2016

Test locale IDs uniqueness.

Loading status checks…

3450a67

Force normalization of locale ID.

Loading status checks…

78eea4c

akx reviewed Mar 3, 2016
View changes

kdeldycke added 2 commits Mar 3, 2016

Explicit normalize_locale() documentation.

Loading status checks…

a5cb831

Only attempt a normalization search if filesystem lookup fails.

Loading status checks…

1a6731e

akx added a commit that referenced this pull request Mar 3, 2016

Merge pull request #361 from kdeldycke/mixed_cased_locale_id

Loading status checks…

85239b2

Parsing of mixed-cased locale ID.

kdeldycke deleted the kdeldycke:mixed_cased_locale_id branch Mar 3, 2016

jtwang reviewed Mar 3, 2016
View changes

babel/localedata.py

if name == locale_id.lower():

return locale_id

def exists(name):

This comment has been minimized.

Sign in to view

jtwang Mar 3, 2016
Contributor

Wow, greedy method.
I'm really looking forward to #305 !

(just an observation, please ignore :)

jtwang reviewed Mar 3, 2016
View changes

pyup-bot mentioned this pull request Jan 31, 2017

Update babel to 2.3.4 mkurek/ralph#28

Open

pyup-bot mentioned this pull request Apr 11, 2017

Initial Update do3cc/blog#3

Open

pyup-bot mentioned this pull request Apr 27, 2017

Pin babel to latest version 2.4.0 osteele/assignment-dashboard#24

Open

pyup-bot mentioned this pull request May 12, 2017

Initial Update tantale/intranet#43

Closed

pyup-bot mentioned this pull request Jul 4, 2017

Initial Update pelson/pyggybank#2

Merged

This was referenced Aug 17, 2017

Pin babel to latest version 2.4.0 springermac/pyinstaller#67

Merged

Pin babel to latest version 2.5.0 osteele/assignment-dashboard#51

Open

This was referenced Aug 26, 2017

Pin babel to latest version 2.5.0 nicfit/nicfit.py#129

Merged

Initial Update oddbird/oddsite#127

Closed

This was referenced Sep 4, 2017

Scheduled weekly dependency update for week 36 oddbird/oddsite#128

Merged

Initial Update rooty/flask-test#2

Open

pyup-bot mentioned this pull request Sep 14, 2017

Pin babel to latest version 2.5.1 osteele/assignment-dashboard#56

Open

pyup-bot mentioned this pull request Oct 17, 2017

Initial Update tracim/tracim#373

Closed

pyup-bot mentioned this pull request Nov 3, 2017

Update babel to 2.5.1 tracim/tracim#424

Closed

vorpal-buildbot mentioned this pull request Jun 13, 2020

Initial Update PennyDreadfulMTG/Penny-Dreadful-Tools#7430

Merged

python-babel / babel

Parsing of mixed-cased locale ID. #361

Parsing of mixed-cased locale ID. #361

kdeldycke commented Mar 2, 2016

kdeldycke commented Mar 2, 2016

codecov-io commented Mar 2, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

akx commented Mar 3, 2016

kdeldycke commented Mar 3, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jtwang commented Mar 3, 2016

akx commented Mar 3, 2016



		def test_mixedcased_locale():
		for l in localedata.locale_identifiers():

python-babel / babel

Parsing of mixed-cased locale ID. #361

Parsing of mixed-cased locale ID. #361

Conversation

kdeldycke commented Mar 2, 2016

kdeldycke commented Mar 2, 2016

codecov-io commented Mar 2, 2016

Current coverage is 89.68%

This comment has been minimized.

akx Mar 3, 2016 Member

This comment has been minimized.

kdeldycke Mar 3, 2016 Author Contributor

This comment has been minimized.

akx Mar 3, 2016 Member

This comment has been minimized.

akx Mar 3, 2016 Member

This comment has been minimized.

kdeldycke Mar 3, 2016 Author Contributor

akx commented Mar 3, 2016

kdeldycke commented Mar 3, 2016

This comment has been minimized.

jtwang Mar 3, 2016 Contributor

This comment has been minimized.

jtwang Mar 3, 2016 Contributor

This comment has been minimized.

akx Mar 3, 2016 Member

This comment has been minimized.

jtwang Mar 3, 2016 Contributor

This comment has been minimized.

akx Mar 3, 2016 Member

This comment has been minimized.

jtwang Mar 3, 2016 Contributor

This comment has been minimized.

akx Mar 3, 2016 Member

This comment has been minimized.

jtwang Mar 3, 2016 Contributor

jtwang commented Mar 3, 2016

akx commented Mar 3, 2016

Current coverage is `89.68%`

akx Mar 3, 2016
Member

kdeldycke Mar 3, 2016
Author Contributor

akx Mar 3, 2016
Member

akx Mar 3, 2016
Member

kdeldycke Mar 3, 2016
Author Contributor

jtwang Mar 3, 2016
Contributor

jtwang Mar 3, 2016
Contributor

akx Mar 3, 2016
Member

jtwang Mar 3, 2016
Contributor

akx Mar 3, 2016
Member

jtwang Mar 3, 2016
Contributor

akx Mar 3, 2016
Member

jtwang Mar 3, 2016
Contributor