Problem installing 6.0 on newly installed Ubuntu machine

[gaqzi]

I just tried installing phonenumbers on a newly installed Ubuntu 12.04 installation.

I'm running these versions of things:
Python: 2.7.3
pip: 1.5.1
virtualenv: 1.11.1

To repeat:

$ mkvirtualenv test
$ pip install phonenumbers
[... long list of byte compiled files ...]
Cleaning up...
Command /home/deploy/.virtualenvs/test/bin/python -c "import setuptools, tokenize;__file__='/home/deploy/.virtualenvs/test/build/phonenumbers/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-11ulaL-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/deploy/.virtualenvs/test/include/site/python2.7 failed with error code -9 in /home/deploy/.virtualenvs/test/build/phonenumbers
Storing debug log for failure in /home/deploy/.pip/pip.log

I've no idea what error code -9 means, but it works if I install version 5.9.2 instead.

[daviddrysdale]

Hmm, works for me on a similar-but-not-identical system (Python 2.7.3/pip 1.0/virtualenv 1.7.1.2).

Are there any more details in the pip.log file mentioned?

[gaqzi]

I tried installing pip 1.0 as well but the same issue. I also tried it out with an older version of setuptools in case that was the issue, I'm currently running 2.1 in the VM, but switching to 1.4.2 which is the same version as on my dev computer didn't work either.

I'm not entirely sure what else to look at, for reference here is my pip freeze:

(test)deploy@opening-hours:~$ pip freeze
argparse==1.2.1
wsgiref==0.1.2

Here is the complete log when trying to install with setuptools 2.1: https://gist.github.com/gaqzi/9483407

[daviddrysdale]

Nothing helpful in the log file sadly (the stack is just pip code, not python-phonenumbers code).

Random thought: you're not low on disk space by any chance? The metadata (and therefore the package) gets bigger with every release...

[Katafalkas]

Experiencing the same problem. We run VMs on AWS.
Tried with latest ubuntu 12.04 and 13.10 image - same outcome.
Tried debugging the command:
/usr/bin/python -c "import setuptools;__file__='/srv/invitetravel_api_0/build/phonenumbers/build/phonenumbers/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"

>>> import setuptools
>>> __file__='/srv/foobar/build/phonenumbers/build/phonenumbers/setup.py'
>>> a = open(__file__)
>>> b = a.read()
>>> c = b.replace('\r\n', '\n')
>>> d = compile(c, __file__, 'exec')
>>> d
<code object <module> at 0x2b430b0, file "/srv/foobar/build/phonenumbers/build/phonenumbers/setup.py", line 18>
>>> exec(d)
usage: [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: --help [cmd1 cmd2 ...]
or: --help-commands
or: cmd --help
error: no commands supplied

[acatton]

I'm getting the exact same error message as well on my VMs. The compilation starts taking a lot of memory, it starts swapping. And then the process dies, and I get this error. (I believe it's pip itself generating the error because the sub-process died)

@daviddrysdale I think it's getting OOM killed, or malloc() fails. I don't know. Anyway, @daviddrysdale if you would like to reproduce this error, you should try to install python-phonenumbers on a vm with low memory.

[gaqzi]

@daviddrysdale: there's plenty of space, 6GB free.

On Tue, Mar 11, 2014 at 7:48 PM, David Drysdale notifications@github.comwrote:

Nothing helpful in the log file sadly (the stack is just pip code, not
python-phonenumbers code).

Random thought: you're not low on disk space by any chance? The metadata
(and therefore the package) gets bigger with every release...

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/33#issuecomment-37287109
.

"Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defense
against complexity."
-- David Gelernter, Machine Beauty:
Elegance and the Heart of Technology

[acatton]

@gaqzi Can you confirm what I just wrote. When your installing phonenumbers, can you monitor the memory of your server. Is the installation consuming a lot of memory?

[daviddrysdale]

Nothing significant changed in the code structure between 5.9.2 and 6.0.0, but the size of the geographic metadata increased substantially (phonenumbers/geodata/__init__.py grew from 3.2MB to 15.2MB).

Folks who can reproduce the problem: what happens if you unpack the 6.0.0 code and try to import phonenumbers.geodata locally? Does it succeed and generate an geodata/__init__.pyc file?

[gaqzi]

@acatton you're correct, it balloons like mad when installing. I tried it on my dev machine and I use around ~400MB during one brief phase of installation. I'll try on a VM as soon as I can as well.

[gaqzi]

@daviddrysdale: I've tried that out and just including phonenumbers.geodata consumes ~150MB or memory. During a peak of compilation it was taking somewhere around 700MB, but ended around ~150MB at the end of it.

[daviddrysdale]

Thanks for the updates; it sounds like I need to split the geodata out into a separate package, like the upstream Java project does.

[gavinwahl]

Just importing geodata takes about 2 seconds on my machine, even after the pyc is compiled. Have you considered shipping a compiled shelve database? Since GEOCODE_DATA is just a dictionary, shelve works quite well. I did some testing and opening and retrieving keys from the shelved data is instant.

By distributing the shelve file itself, you could avoid the memory problems, and speed up the initial import of phonenumbers.

[daviddrysdale]

I had a play with shelve, but it seems hard to get an underlying database that works everywhere -- I was hitting problems between bdb versions (1.85/v2 on Mac, v9 on Linux) and pickle versions (Python 2.x vs. 3.x). So I'll try something else.

[barrywhart]

What about SQLite? I found an example here of storing blobs in Python SQLite: http://eli.thegreenplace.net/2009/05/29/storing-blobs-in-a-sqlite-db-with-pythonpysqlite/

[artnez]

Can you guys test out #34? I need to give another glance over to make sure it's production ready, but the overall implementation is there. Tests passed and I was able to install in a low-memory VM where this problem surfaced.

[artnez]

To test it out, run this:

pip install git+git://github.com/path/python-phonenumbers.git@c1f99689a4eed256c1a678be3afec04bffbbb8c1#egg=phonenumbers

[chemary]

I have the same problem, I have tried 6.0.0, 5.9.2 and 5.9.1 and also the this last #34. All consume the 512MB of ram plus 512MB of swap of my VM until it dies.

[artnez]

@chemary You can somewhat bypass the problem by installing via pip using the -e flag. That will prevent pre-compiling the bytecodes for python/phonenumbers/geodata/__init__.py and install cleanly.

The downside is that the bytecode will get compiled on first load. If you're running a web server, this isn't a problem. Just import phonenumbers.geodata for every worker process (which will compile bytecodes) and you'll be OK.

I got #34 working on a VM with 128mb memory so I'm curious why it didn't work for you. How exactly did you install?

[chemary]

Hi @artnez, I installed with pip install phonenumbers
I have this environment and more than 400MB of free ram before installation:
Python 2.7.3rc2
virtualenv 1.9.1
Django==1.5.5

Installation by other ways means changing our deployment scripts, so I will try to validate phone numbers by simple regex for now.

[daviddrysdale]

Splitting the metadata into chunks seems sensible regardless, so I've merged that -- thanks @artnez .

I've also got an initial attempt to build a phonenumberslite version of the package, without any of the per-prefix metadata.

@chemary, could you try pip install phonenumberslite to see if the smaller package at PyPI works for you?

[chemary]

The package installs fine, I have tested it a little and seems to work fine.
Is this version production ready?
Thanks!

[daviddrysdale]

If it installs OK, it's probably fine -- it's basically the same package but missing a few large metadata directories.

(Obviously none of the phonenumbers.geocoder, phonenumbers.carrier or phonenumbers.timezone functionality is going to work in this version though.)

[daviddrysdale]

OK, I think this is done. To summarize:

• Future releases of phonenumbers will have the prefix metadata broken into chunks (thanks to @artnez).
• There will also be parallel releases of a phonenumberslite package, with the prefix metadata absent.

[artnez]

@daviddrysdale When will this hit pypi?

[daviddrysdale]

Ah, looks like I only pushed phonenumberslite-6.0.0a to PyPI. Just pushed a phonenumbers-6.0.0a to go with it.

[artnez]

Thanks man.

On Mar 26, 2014, at 2:32 AM, David Drysdale notifications@github.com wrote:

Ah, looks like I only pushed phonenumberslite-6.0.0a to PyPI. Just pushed a phonenumbers-6.0.0a to go with it.

—
Reply to this email directly or view it on GitHub.