New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4. #174

Closed
jun66j5 opened this Issue Jul 30, 2015 · 19 comments

Comments

Projects
None yet
@jun66j5
Contributor

jun66j5 commented Jul 30, 2015

Python 3.4.3 (default, Apr 13 2015, 22:30:47)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import babel
>>> babel.__version__
'2.0'
>>> from babel.core import Locale
>>> Locale.parse('zh_CN')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/py34/lib/python3.4/site-packages/babel/core.py", line 286, in parse
    language = get_global('language_aliases').get(language, language)
  File "/venv/py34/lib/python3.4/site-packages/babel/core.py", line 58, in get_global
    _global_data = pickle.load(fileobj)
TypeError: an integer is required (got type str)
>>>

Locale.parse('zh_CN') should work and load zh_CN localedata.

On Python 2.7, it works fine.

Python 2.7.3 (default, Jun 22 2015, 19:33:41)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import babel
>>> babel.__version__
'2.0'
>>> from babel.core import Locale
>>> Locale.parse('zh_CN')
Locale('zh', territory='CN', script='Hans')
>>>
@sils

This comment has been minimized.

Member

sils commented Jul 30, 2015

@jun66j5 thanks for your report!

Babel was barely maintained for some time and we're some new people trying to get it up to speed again. Because of this it might take some time until we're able to adress this bug.

@erickwilder

This comment has been minimized.

Contributor

erickwilder commented Jul 30, 2015

@jun66j5 is this working in the previous (1.x) release?

@erickwilder erickwilder added the bug label Jul 30, 2015

@jun66j5

This comment has been minimized.

Contributor

jun66j5 commented Jul 30, 2015

Babel 1.3 works fine with Python 3.3 and 3.4.

Python 3.3.6 (default, Jan 28 2015, 16:27:12)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import babel
>>> babel.__version__
'1.3'
>>> from babel.core import Locale
>>> Locale.parse('zh_CN')
Locale('zh', territory='CN', script='Hans')
@blisc

This comment has been minimized.

blisc commented Jul 30, 2015

I got the a similar error trying to compile Python documentation with Sphinx.
Downgrading to babel 1.3 seems to have resolved the issue on 3.4.3

@sidloki

This comment has been minimized.

sidloki commented Jul 31, 2015

Same error here. If I clone and checkout branch 2.0 and run:

/path/to/babel$ python3.4 setup.py import_cldr release sdist
/path/to/babel$ pip install dist/Babel-2.0.tar.gz

everything works fine (both python3.4 and python2.7). A package created with python2.7 is not working with python3.4.

erickwilder added a commit that referenced this issue Jul 31, 2015

@erickwilder

This comment has been minimized.

Contributor

erickwilder commented Jul 31, 2015

I can't reproduce the error. Both running directly at the Python interpreter and writing a new test case and running against tox/pytest results on the Locale object being created (and tests passing)
I'm on a MacOSX and Python 3.x versions installed with pyenv. Anyone with a Linux environment can test the issue174/locale-parse-py3x branch?

@erickwilder erickwilder self-assigned this Jul 31, 2015

@jun66j5

This comment has been minimized.

Contributor

jun66j5 commented Jul 31, 2015

babel/global.dat file in Babel-2.0.tar.gz on PyPI seems to be broken.
The global.dat cannot be loaded with Python 3.4 but it can be loaded with Python 2.6.

$ wget -q -P /tmp https://pypi.python.org/packages/source/B/Babel/Babel-2.0.tar.gz#md5=62917719897a81e22dcaa3b17eeb11d8
$ md5sum /tmp/Babel-2.0.tar.gz
62917719897a81e22dcaa3b17eeb11d8  /tmp/Babel-2.0.tar.gz
$ tar tzvf /tmp/Babel-2.0.tar.gz | grep /global.dat
-rw-r--r-- mitsuhiko/staff 116412 2015-07-27 19:35 Babel-2.0/babel/global.dat
$ tar xzf /tmp/Babel-2.0.tar.gz -C /tmp Babel-2.0/babel/global.dat
$ ls -l /tmp/Babel-2.0/babel/global.dat
-rw-r--r-- 1 jun66j5 jun66j5 116412 Jul 27 19:35 /tmp/Babel-2.0/babel/global.dat
$ python3.4
Python 3.4.3 (default, Apr 13 2015, 22:30:47)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('/tmp/Babel-2.0/babel/global.dat', 'rb') as f: data = pickle.load(f)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: an integer is required (got type str)
>>>
$ python2.6
Python 2.6.9 (default, Oct 22 2014, 19:47:46)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> with open('/tmp/Babel-2.0/babel/global.dat', 'rb') as f: data = pickle.load(f)
...
>>> data.keys()
['territory_aliases', 'territory_zones', 'zone_aliases', 'territory_currencies', 'windows_zone_mapping', 'script_aliases', '_version', 'variant_aliases', 'language_aliases', 'meta_zones', 'likely_subtags', 'zone_territories']
@sidloki

This comment has been minimized.

sidloki commented Jul 31, 2015

To reproduce the error:

# Python 2.7 environment
$ make clean
$ make import-cldr
$ py.test
# all tests passed

# python3.4 environment
$ make clean-pyc
$ py.test
# some tests failed with TypeError: an integer is required (got type str)

The problem is, that the global.dat created in a python2 environment cannot be used in a python3 environment.

If I create the global.dat in a python3 environment and use it under python2, I get the follwoing failures:

[doctest] babel.core.Locale.parse 
215 
216         >>> Locale.parse(l)
217         Locale('de', territory='DE')
218 
219         This also can perform resolving of likely subtags which it does
220         by default.  This is for instance useful to figure out the most
221         likely locale for a territory you can use ``'und'`` as the
222         language tag:
223 
224         >>> Locale.parse('und_AT')
Expected:
    Locale('de', territory='AT')
Got:
    Locale(u'de', territory=u'AT')

[doctest] babel.numbers.get_territory_currencies
076     of when the currency became active.  The longer the currency is being in
077     use the more to the left of the list it will be.
078 
079     The start date defaults to today.  If no end date is given it will be the
080     same as the start date.  Otherwise a range can be defined.  For instance
081     this can be used to find the currencies in use in Austria between 1995 and
082     2011:
083 
084     >>> from datetime import date
085     >>> get_territory_currencies('AT', date(1995, 1, 1), date(2011, 1, 1))
Expected:
    ['ATS', 'EUR']
Got:
    [u'ATS', u'EUR']
@shimizukawa

This comment has been minimized.

shimizukawa commented Aug 1, 2015

To reproduce the error:

C:\> python
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64 bit (AMD64)] on win32
>>> from babel.messages.pofile import read_po
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\py34\lib\site-packages\babel\messages\__init__.py", line 12, in <module>
    from babel.messages.catalog import *
  File "D:\py34\lib\site-packages\babel\messages\catalog.py", line 23, in <module>
    from babel.dates import format_datetime
  File "D:\py34\lib\site-packages\babel\dates.py", line 28, in <module>
    from babel.util import UTC, LOCALTZ
  File "D:\py34\lib\site-packages\babel\util.py", line 278, in <module>
    from babel import localtime
  File "D:\py34\lib\site-packages\babel\localtime\__init__.py", line 21, in <module>
    from babel.localtime._win32 import _get_localzone
  File "D:\py34\lib\site-packages\babel\localtime\_win32.py", line 18, in <module>
    tz_names = get_global('windows_zone_mapping')
  File "D:\py34\lib\site-packages\babel\core.py", line 58, in get_global
    _global_data = pickle.load(fileobj)
TypeError: an integer is required (got type str)
@mitsuhiko

This comment has been minimized.

Member

mitsuhiko commented Aug 1, 2015

This is a really annoying issue. Ideally we would stop using pickle and switch to something we can efficiently load from disk (msgpack as a start). Long term it would be great if we could mmap something, but unfortunately that might be pretty slow with the constant boxing :(

@icoxfog417

This comment has been minimized.

icoxfog417 commented Aug 2, 2015

As @jun66j5 says, global.dat on PyPI may be broken.
I installed from GitHub 2.0 branch as below instead of pip install babel, then the error didn't occur.

pip install git+https://github.com/mitsuhiko/babel.git@2.0

My environments is Windows 8.1 and Python 3.4.3.

@thuvh

This comment has been minimized.

thuvh commented Aug 4, 2015

I think this error happens in window environments

jun66j5 added a commit to jun66j5/babel that referenced this issue Aug 4, 2015

jun66j5 added a commit to jun66j5/babel that referenced this issue Aug 4, 2015

Removed uses of date instances from global.dat for workaround to avoi…
…d incompatible pickle format between python 2 and 3 (python-babel#174)
@erickwilder

This comment has been minimized.

Contributor

erickwilder commented Aug 4, 2015

@jun66j5
I saw that you're working on some changes to solve this issue. Any progress? Do you need some help?

@jun66j5

This comment has been minimized.

Contributor

jun66j5 commented Aug 5, 2015

Thanks, @erickwilder. Updated https://github.com/jun66j5/babel/commits/issue174/workaround.

Root cause is incompatible pickle data for datetime.date classes between Python 2 and 3. In that branch, the uses of the classes are removed and added unit tests for that.

Should I create a pull request with that branch?

@erickwilder

This comment has been minimized.

Contributor

erickwilder commented Aug 5, 2015

@jun66j5
Yes please. Submit a pull request so we can review.
We'll need to backport changes of that to the 2.x branch, so your call to use master or the maintenance branch as a target when opening.

jun66j5 added a commit to jun66j5/babel that referenced this issue Aug 5, 2015

Removed uses of datetime.date class from *.dat files (python-babel#174)
To avoid incompatible *.dat files between Python 2 and 3.

erickwilder added a commit to erickwilder/babel that referenced this issue Aug 5, 2015

Removed uses of datetime.date class from *.dat files (python-babel#174)
To avoid incompatible *.dat files between Python 2 and 3.
@ericholscher

This comment has been minimized.

ericholscher commented Aug 11, 2015

Ran into this one as well on Windows & 3.4: https://ci.appveyor.com/project/ericholscher/sphinx-autoapi/build/1.0.5

=================================== ERRORS ==================================== 
_________________ ERROR collecting tests/test_integration.py __________________ 
tests\test_integration.py:9: in <module> 
    from sphinx.application import Sphinx 
.tox\py34\lib\site-packages\sphinx\application.py:37: in <module> 
    from sphinx.builders import BUILTIN_BUILDERS 
.tox\py34\lib\site-packages\sphinx\builders\__init__.py:23: in <module> 
    from sphinx.util import i18n, path_stabilize 
.tox\py34\lib\site-packages\sphinx\util\i18n.py:15: in <module> 
    from babel.messages.pofile import read_po 
.tox\py34\lib\site-packages\babel\messages\__init__.py:12: in <module> 
    from babel.messages.catalog import * 
.tox\py34\lib\site-packages\babel\messages\catalog.py:23: in <module> 
    from babel.dates import format_datetime 
.tox\py34\lib\site-packages\babel\dates.py:28: in <module>
    from babel.util import UTC, LOCALTZ
.tox\py34\lib\site-packages\babel\util.py:278: in <module>
    from babel import localtime 
.tox\py34\lib\site-packages\babel\localtime\__init__.py:21: in <module> 
    from babel.localtime._win32 import _get_localzone 
.tox\py34\lib\site-packages\babel\localtime\_win32.py:18: in <module> 
    tz_names = get_global('windows_zone_mapping') 
.tox\py34\lib\site-packages\babel\core.py:58: in get_global 
    _global_data = pickle.load(fileobj) 
E   TypeError: an integer is required (got type str) 

erickwilder added a commit to erickwilder/babel that referenced this issue Aug 24, 2015

Removed uses of datetime.date class from *.dat files (python-babel#174)
To avoid incompatible *.dat files between Python 2 and 3.
Added unit tests for that *.dat files have only babel classes

 Author:    Jun Omae <jun66j5@gmail.com>
@vstinner

This comment has been minimized.

vstinner commented Sep 10, 2015

Cool! The fix (#188) was merged! Would it be possible to get a bugfix release? The latest Babel release on PyPI is incompatible with Python 3 :-(

@sils

This comment has been minimized.

Member

sils commented Sep 10, 2015

@Haypo we're still not having access to pypi to update that, it's a bit hard to reach the old maintainer. You can install Babel from git in the meantime: https://pip.pypa.io/en/latest/reference/pip_install.html#git

@erickwilder closing as it was fixed

@sils sils closed this Sep 10, 2015

shimizukawa added a commit to sphinx-doc/sphinx that referenced this issue Sep 13, 2015

Avoid "2.0" version of Babel because it doesn't work with Windows env…
…ironment. Closes #1976.

see also:
* python-babel/babel#174
* python-babel/babel#188

Version spec syntax "babel>=1.3,!=2.0" is following PEP440: https://www.python.org/dev/peps/pep-0440/#version-exclusion and it works with pip 6.0 or later (I didn't check before pip 6.0).

@pyup-bot pyup-bot referenced this issue Jul 4, 2017

Merged

Initial Update #2

@pyup-bot pyup-bot referenced this issue Oct 12, 2017

Merged

Initial Update #16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment