Fix packager.py name normalization

[ghost]

After deployment we noticed that our application didn't work anymore:
[ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'sqlalchemy'

Our requirements.txt file has the following, so no problem there:

$ grep -i sqlalchemy requirements.txt 
SQLAlchemy==1.3.9

While debugging we noticed that the resulting deployment has sqlalchemy files in the wrong location:

$ unzip -l 5a76ade4f39e704f0925422520b454f2-python3.7.zip | grep -i 'sqlalchemy[^/]*/[^/]*$'
     4621  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/__init__.py
     5648  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/processors.py
     2377  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/schema.py
     3377  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/types.py
     2990  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/inspection.py
     6693  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/log.py
    16938  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/exc.py
    12721  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/interfaces.py
    51669  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/events.py
       11  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/top_level.txt
      104  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/WHEEL
     1100  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/LICENSE
    23329  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/RECORD
     7240  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/METADATA

It looks like packager.py does in fact have the code to move purelib back to the root directory -- function _install_purelib_and_platlib -- however, it is not working as intended.

While debugging further with print()s we noticed that wheel.data_dir variable is just wrong:

    @property
    def data_dir(self):
        # type: () -> str
        # The directory format is {distribution}-{version}.data
        return '%s-%s.data' % (self._name, self._version)

This yields sqlalchemy-1.3.9.data rather than SQLAlchemy-1.3.9.data and therefore the code in _install_purelib_and_platlib doesn't execute. The reason is the normalization function:

    def _normalize_name(self, name):
        # type: (str) -> str
        # Taken directly from PEP 503
        return re.sub(r"[-_.]+", "-", name).lower()

I don't think package names should be forced lower-case like that -- even though in theory they should be lower-case, in practice our requirements.txt does have quite a few upper-cased package names and identifiers:

$ grep '^[A-Z]' requirements.txt 
PyMySQL==0.9.3
SQLAlchemy==1.3.10
SQLAlchemy-FullText-Search==0.2.5
SQLAlchemy-Utils==0.34.2
WeasyPrint==50

I am creating a pull request to fix this issue, and after using it our deployment is able to successfully use SQLAlchemy package again.

$ unzip -l bf895adfe5c60f297701196b732057ca-python3.7.zip | grep -i 'sqlalchemy[^/]*/[^/]*$'
     4621  2019-10-29 22:03   sqlalchemy/__init__.py
     5648  2019-10-29 22:03   sqlalchemy/processors.py
     2377  2019-10-29 22:03   sqlalchemy/schema.py
     3377  2019-10-29 22:03   sqlalchemy/types.py
     2990  2019-10-29 22:03   sqlalchemy/inspection.py
     6693  2019-10-29 22:03   sqlalchemy/log.py
    16938  2019-10-29 22:03   sqlalchemy/exc.py
    12721  2019-10-29 22:03   sqlalchemy/interfaces.py
    51669  2019-10-29 22:03   sqlalchemy/events.py
       11  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/top_level.txt
      104  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/WHEEL
     1100  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/LICENSE
    23329  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/RECORD
     7240  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/METADATA

Thanks!

[stealthycoin]

What platform is this failing on?

[ghost]

@stealthycoin this is running newly installed Ubuntu 19.10 + standard PyCharm Community installation (on KVM, but that shouldn't matter, I think?). Adding further platform information below -- please let me know if you need any other info.

(venv) vencky@dev:~/PycharmProjects/lambda/backend$ arch
x86_64
(venv) vencky@dev:~/PycharmProjects/lambda/backend$ python --version
Python 3.7.3
(venv) vencky@dev:~/PycharmProjects/lambda/backend$ pip --version
pip 19.3.1 from /home/vencky/PycharmProjects/lambda/venv/lib/python3.7/site-packages/pip (python 3.7)
(venv) vencky@dev:~/PycharmProjects/lambda/backend$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:        19.10
Codename:       eoan
(venv) vencky@dev:~/PycharmProjects/lambda/backend$ python -c 'import platform; print(platform.platform())'
Linux-5.3.0-19-generic-x86_64-with-Ubuntu-19.10-eoan

Thanks!

[stealthycoin]

Ok i'll take another look. I couldn't repro on a handful of OSes but I didn't try Ubuntu.

[ghost]

Thanks @stealthycoin. To help with debugging, I'm attaching the PIP file that was generated with the exact command that Chalice executes during packaging:

$ pip wheel --no-deps --wheel-dir /tmp/tmpp_q63koa SQLAlchemy-1.3.10.tar.gz 
Processing ./SQLAlchemy-1.3.10.tar.gz
Building wheels for collected packages: SQLAlchemy
  Building wheel for SQLAlchemy (setup.py) ... done
  Created wheel for SQLAlchemy: filename=SQLAlchemy-1.3.10-cp37-cp37m-linux_x86_64.whl size=1154879 sha256=297a9c89c43c5974a8d29c1a6d18f00072d5b82a846f36141d2b2520b3926e10
  Stored in directory: /tmp/tmpp_q63koa
Successfully built SQLAlchemy

(I had to GZIP the WHL file for GitHub to allow the attachment.)

Note that on this platform SQLAlchemy files end up being in /SQLAlchemy-1.3.10.data/purelib/sqlalchemy -- that's when the Chalice packaging bug happens. I noticed that on the older Ubuntu version that we used Python files in the WHL package were in /sqlalchemy already and didn't need to be moved out of purelib (that's the case where you can't reproduce the bug since it's packaged already in the right place).

SQLAlchemy-1.3.10-cp37-cp37m-linux_x86_64.whl.gz

[ghost]

@stealthycoin did you have a chance to take a look at it? Thanks.

[marklynch]

@stealthycoin @johnvencky I ran into the same issue on amazonlinux but installing with this patch made it work.

[jamesls]

Trying to track down this issue. Here's the set of steps I tried. Are people still running into this? I'd like to repro this myself, but it seems to work as I expect when I ran on ubuntu. #1356 (comment)