Is it possible for env.py to be env.pyc?

[sqlalchemy-bot]

Migrated issue, originally created by Cody Scott (@Siecje)

I'm running into a problem with alembic. I don't have a env.py only a env.pyc and this line is failing. https://bitbucket.org/zzzeek/alembic/src/4effc8124fa2ad5aab42119e4e7aa1452982d257/alembic/script/base.py?at=master&fileviewer=file-view-default#base.py-425

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

did you set sourceless=True? http://alembic.zzzcomputing.com/en/latest/tutorial.html#editing-the-ini-file

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

no issue here. Please define "failing" with complete information, thanks.

[classic@photon2 alembic]$ PYTHONPATH=~/dev/sqlalchemy/lib/ python -m alembic.config current
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
e3f2bd11c8f9 (head)
[classic@photon2 alembic]$ rm foo/^C
[classic@photon2 alembic]$ mv foo/env.py foo/env.py.deleted
[classic@photon2 alembic]$ ls foo/
env.pyc  env.py.deleted  README  script.py.mako  versions
[classic@photon2 alembic]$ PYTHONPATH=~/dev/sqlalchemy/lib/ python -m alembic.config current
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
e3f2bd11c8f9 (head)

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

Yes sourceless was set to true, which allowed the versions to be loaded.

This is the traceback that I'm getting.

I'm trying to run jupyterhub which uses alembic.

    Traceback (most recent call last):
      File "/tmp/pip-build-1zvsjpso/jupyterhub/jupyterhub/app.py", line 1623, in launch_instance_async
      File "/boot/tp/python3/lib/python3.6/types.py", line 204, in __next__
        return next(self.__wrapped)
      File "/tmp/pip-build-1zvsjpso/jupyterhub/jupyterhub/app.py", line 1376, in initialize
      File "/tmp/pip-build-1zvsjpso/jupyterhub/jupyterhub/app.py", line 900, in init_db
      File "/tmp/pip-build-1zvsjpso/jupyterhub/jupyterhub/orm.py", line 530, in new_session_factory
      File "/tmp/pip-build-1zvsjpso/jupyterhub/jupyterhub/orm.py", line 475, in check_db_revision
      File "/boot/forge/pypi_jupyterhub/build/lib/python3.6/site-packages/alembic/command.py", line 496, in stamp
      File "/boot/forge/pypi_jupyterhub/build/lib/python3.6/site-packages/alembic/script/base.py", line 416, in run_env
      File "/boot/forge/pypi_jupyterhub/build/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 99, in load_python_file
    ImportError: Can't find Python file /boot/tp/python3/lib/python3.6/site-packages/jupyterhub/alembic/env.py

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

admin@hostname-not-set:/mnt/vda1/tp/python3/lib/python3.6/site-packages/jupyterhub/alembic$ ls 
README          env.pyc         script.py.mako  versions/
admin@hostname-not-set:/mnt/vda1/tp/python3/lib/python3.6/site-packages/jupyterhub/alembic$ pwd
/boot/tp/python3/lib/python3.6/site-packages/jupyterhub/alembic

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

I think I figured it out.
I'm using Python3.6 and PYTHONOPTIMIZE is set so

sys.flags.optimize is 1.

simple_pyc_file_from_path is adding a o to the path.

In Python3 there are no more .pyo files as per PEP 488.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

OK...but is your .pyc file in the local path where the .py would be? or is it in a __pycache__ ?

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

no more .pyo files as of python 3.5, not python 3

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

It is in the local path where the .py file would be.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

how are you generating that .pyc, since on Python 3.5+, if you do py_compile(path), the file goes into pycache with a special name? I'm just trying to understand how folks even make these sourceless installs. i have most of a fix for all of this but it still needs the version files, pyc or whatever, to be locally present in versions/ and not in a pycache.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

though I can make it work that way...

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

After they are compiled I move them up out of the __pycache__ folder and rename the file to just module.pyc, so that modules are where imports expect them.

Source

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

In versions/ I just have the .pyc files, no __pycache__, which works when I either restore env.py or copy env.pyc to env.pyo.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

https://gerrit.sqlalchemy.org/508

if you can test with this, that would help a lot theres a tar.gz at https://gerrit.sqlalchemy.org/changes/508/revisions/fd1dd170b1deb75b922a57405844e01409d0edf9/archive?format=tgz

[sqlalchemy-bot]

Cody Scott (@Siecje) wrote:

Wow that was a lot bigger than I thought. Yes that works!

Thanks for spending so much time on this.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

Support pep3149, latest import mechanics, fully

Reworked "sourceless" system to be fully capable of handling any
combination of: Python2/3x, pep3149 or not, PYTHONOPTIMIZE or not,
for locating and loading both env.py files as well as versioning files.
This includes: locating files inside of __pycache__ as well as listing
out version files that might be only in versions/__pycache__, deduplicating
version files that may be in versions/__pycache__ and versions/
at the same time, correctly looking for .pyc or .pyo files based on
if pep488 is present or not. The latest Python3x deprecation warnings
involving importlib are also corrected.

Change-Id: I2495e793c81846d3f05620dbececb18973dd8a8f
Fixes: #449

→ 9b5f12a

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• changed status to closed