Parsing of mixed-cased locale ID. #361

kdeldycke · Mar 2, 2016

Following #351, it seems that mixed-cased locale IDs are not properly checked as existing depending on the platform.

The issue lies in babel.localedata.exists() which rely on filesystem assets. As a result, mixed-cased locale IDs are properly parsed on OSX (case-insensitive filesystem) but not on Linux.

The first commit demonstrate the issue depending on the system.
The second commit ensure that locale IDs recognized by Babel are uniques, even if lower-cased.
The third commit fix to the issue.

kdeldycke · Mar 2, 2016

Looks like this issue relates to #283.

codecov-io · Mar 2, 2016

Current coverage is `89.68%`

Merging #361 into master will increase coverage by +0.02% as of bb16d60

@@            master    #361   diff @@
======================================
  Files           23      23       
  Stmts         3794    3801     +7
  Branches         0       0       
  Methods          0       0       
======================================
+ Hit           3402    3409     +7
  Partial          0       0       
  Missed         392     392

Review entire Coverage Diff as of bb16d60

Powered by Codecov. Updated on successful CI builds.

akx · Mar 3, 2016

akx reviewed Mar 3, 2016

View changes

akx · Mar 3, 2016

babel/localedata.py

@@ -24,15 +25,22 @@
 _dirname = os.path.join(os.path.dirname(__file__), 'locale-data')


+def normalize_locale(name):
+    """Return a normalized locale ID or `None` if the ID is not recognized."""


Could you explain what "normalization" means in the docstring?

Sure. Done in kdeldycke@a5cb831 .

akx · Mar 3, 2016

akx reviewed Mar 3, 2016

View changes

akx · Mar 3, 2016

babel/localedata.py

-    if name in _cache:
-        return True
-    return os.path.exists(os.path.join(_dirname, '%s.dat' % name))
+    return True if normalize_locale(name) else False


return bool(normalize_locale(name)) :)

To be honest, I'd like to see this normalization thing as a "slow path", if the previous os.path.exists path fails.

Done: kdeldycke@1a6731e

akx · Mar 3, 2016

Nice, thanks @kdeldycke!

(CI: I'm not waiting for Appveyor, since most builds completed and some failed due to ephemeral errors.)

kdeldycke · Mar 3, 2016

Thanks @akx for the quick merge! 😃

jtwang · Mar 3, 2016

jtwang reviewed Mar 3, 2016

View changes

jtwang · Mar 3, 2016

babel/localedata.py

+        if name == locale_id.lower():
+            return locale_id
+
+
 def exists(name):


Wow, greedy method.
I'm really looking forward to #305 !

(just an observation, please ignore :)

jtwang · Mar 3, 2016

jtwang reviewed Mar 3, 2016

View changes

jtwang · Mar 3, 2016

babel/localedata.py


    :param name: the locale identifier string
    """
    if name in _cache:
        return True
-    return os.path.exists(os.path.join(_dirname, '%s.dat' % name))
+    file_found = os.path.exists(os.path.join(_dirname, '%s.dat' % name))
+    return True if file_found else bool(normalize_locale(name))


Would iteration over in-mem objects would be faster than disk access? If so, please consider performing the in-memory check first. :)

(super duper minor optimization)

That'd complicate the whole deal unnecessarily, imho. It'd become something like

(1) name in cache?

(2) normalized name in cache?

(3) name in file system?

(4) normalized name in file system?

which would kinda interleave normalization with this "simple" existence checking...

jtwang · Mar 3, 2016

jtwang reviewed Mar 3, 2016

View changes

jtwang · Mar 3, 2016

tests/test_localedata.py

+    all_ids = localedata.locale_identifiers()
+    assert len(all_ids) == len(set(all_ids))
+    # Check locale IDs don't collide after lower-case normalization.
+    lower_case_ids = list(map(methodcaller('lower'), all_ids))


IMO

lower_case_ids = [id.lower() for id in all_ids] is more readable, but don't care much.

Yeah, it would be, and more Pythonic, but that's kinda minor, and can be trivially corrected later if need be.

jtwang · Mar 3, 2016

jtwang reviewed Mar 3, 2016

View changes

jtwang · Mar 3, 2016

tests/test_localedata.py

+
+
+def test_mixedcased_locale():
+    for l in localedata.locale_identifiers():


Is it necessary to iterate through all ids? Would it be sufficient to spot check some deliberately selected hardcoded values? Eg. 'Fi', 'FI_fi', 'fi_FI', 'fI_fI', 'ZH_hanT_HK'

That's another optimization that could be done (via pytest.parametrize), but I don't think our test suite takes too long to run as it is :)

True that. I just get the heebie jeebies when I see random.choice in unit tests, ha.

jtwang · Mar 3, 2016

Ah, crud, didn't notice this was already merged into master. Feel free to ignore all my comments, none of them are major. :)

akx · Mar 3, 2016

All comments were duly noted and processed, don't fret :D

Test parsing of mixed-cased locale ID.

Loading status checks…

6844ccc

gitmate-bot added the process/pending review label Mar 2, 2016

kdeldycke added some commits Mar 2, 2016

Test locale IDs uniqueness.

Loading status checks…

3450a67

Force normalization of locale ID.

Loading status checks…

78eea4c

kdeldycke added some commits Mar 3, 2016

Explicit normalize_locale() documentation.

Loading status checks…

a5cb831

Only attempt a normalization search if filesystem lookup fails.

Loading status checks…

1a6731e

akx added a commit that referenced this pull request Mar 3, 2016

Merge pull request #361 from kdeldycke/mixed_cased_locale_id
Parsing of mixed-cased locale ID.

Loading status checks…

85239b2

kdeldycke deleted the kdeldycke:mixed_cased_locale_id branch Mar 3, 2016

pyup-bot referenced this pull request Jan 6, 2017
Merged
Update babel to 2.3.4 #13

pyup-bot referenced this pull request Jan 31, 2017
Open
Update babel to 2.3.4 #28

pyup-bot referenced this pull request Apr 11, 2017
Open
Initial Update #3

pyup-bot referenced this pull request Apr 27, 2017
Open
Pin babel to latest version 2.4.0 #24

pyup-bot referenced this pull request May 12, 2017
Closed
Initial Update #43

pyup-bot referenced this pull request Jul 4, 2017
Merged
Initial Update #2

This was referenced Aug 17, 2017

Merged

Pin babel to latest version 2.4.0 springermac/pyinstaller#67

Open

Pin babel to latest version 2.5.0 olin-computing/assignment-dashboard#51

This was referenced Aug 26, 2017

Merged

Pin babel to latest version 2.5.0 nicfit/nicfit.py#129

Closed

Initial Update oddbird/oddsite#127

This was referenced Sep 4, 2017

Merged

Scheduled weekly dependency update for week 36 oddbird/oddsite#128

Open

Initial Update rooty/flask-test#2

pyup-bot referenced this pull request Sep 14, 2017
Open
Pin babel to latest version 2.5.1 #56

This was referenced Oct 12, 2017

Merged

Initial Update fake-name/xA-Scraper#16

Closed

Initial Update tracim/tracim#373

pyup-bot referenced this pull request Nov 3, 2017
Closed
Update babel to 2.5.1 #424

pyup-bot referenced this pull request Feb 25, 2018
Merged
Pin babel to latest version 2.5.3 #39

python-babel/babel

Join GitHub today

Parsing of mixed-cased locale ID. #361

Conversation

kdeldycke commented Mar 2, 2016

gitmate-bot added the process/pending review label Mar 2, 2016

This comment has been minimized.

kdeldycke commented Mar 2, 2016

kdeldycke added some commits Mar 2, 2016

This comment has been minimized.

codecov-io commented Mar 2, 2016

Current coverage is 89.68%

akx reviewed Mar 3, 2016 View changes

This comment has been minimized.

This comment has been minimized.

akx reviewed Mar 3, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kdeldycke added some commits Mar 3, 2016

This comment has been minimized.

akx commented Mar 3, 2016

akx added a commit that referenced this pull request Mar 3, 2016

Hide details View details akx merged commit 85239b2 into python-babel:master Mar 3, 2016 3 of 6 checks passed

3 of 6 checks passed

This comment has been minimized.

kdeldycke commented Mar 3, 2016

kdeldycke deleted the kdeldycke:mixed_cased_locale_id branch Mar 3, 2016

jtwang reviewed Mar 3, 2016 View changes

This comment has been minimized.

jtwang reviewed Mar 3, 2016 View changes

This comment has been minimized.

This comment has been minimized.

jtwang reviewed Mar 3, 2016 View changes

This comment has been minimized.

This comment has been minimized.

jtwang reviewed Mar 3, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

jtwang commented Mar 3, 2016

This comment has been minimized.

akx commented Mar 3, 2016

pyup-bot referenced this pull request Jan 6, 2017

pyup-bot referenced this pull request Jan 31, 2017

This was referenced Feb 19, 2017

pyup-bot referenced this pull request Apr 11, 2017

pyup-bot referenced this pull request Apr 27, 2017

pyup-bot referenced this pull request May 12, 2017

pyup-bot referenced this pull request Jul 4, 2017

This was referenced Aug 17, 2017

This was referenced Aug 26, 2017

This was referenced Sep 4, 2017

pyup-bot referenced this pull request Sep 14, 2017

This was referenced Oct 12, 2017

pyup-bot referenced this pull request Nov 3, 2017

This was referenced Jan 10, 2018

pyup-bot referenced this pull request Feb 25, 2018

Current coverage is `89.68%`

akx reviewed Mar 3, 2016

View changes

akx reviewed Mar 3, 2016

View changes

akx merged commit `85239b2` into python-babel:master Mar 3, 2016
3 of 6 checks passed

jtwang reviewed Mar 3, 2016

View changes

jtwang reviewed Mar 3, 2016

View changes

jtwang reviewed Mar 3, 2016

View changes

jtwang reviewed Mar 3, 2016

View changes