load_module in compat.py doesn't handle .pyo

[sqlalchemy-bot]

Migrated issue, originally created by Greg Yang (@eulerreich)

I'm in a situation where I need to run alembic in a compiled package with all source files stripped away. Right now it's giving me a file not found error with the line

fp = open(path, 'rb')

where path is the full path of where the .py file is supposed to be; so the .pyo file is right in there but load_module doesn't see it.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

this is not a simple request. the .py files here are located using a directory listing approach. Logic would need to be added that first looks in place for .pyc or .pyo files, then the load module system needs to be modified.

Also, pep3147 might need to be taken into account; this is where the files would instead be in a __pycache__ directory and there's no simple way to locate these files. The good news is that per pep3147 it seems they will continue to support a .pyc or .pyo file sitting next to the .py (http://www.python.org/dev/peps/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports). However the system you're using to "strip the .py files away" may be too simplistic to work like this - in Python 3 the .pyc file won't just be sitting there normally.

Here's a partial diff to handle being able to load the .pyc/.pyo. It's quite complicated/hacky for the Py3k case:

--- a/alembic/compat.py
+++ b/alembic/compat.py
@@ -45,14 +45,31 @@ if py2k:
 
 if py33:
     from importlib import machinery
+    import os
     def load_module(module_id, path):
-        return machinery.SourceFileLoader(module_id, path).load_module()
+        dir_, fname = os.path.split(path)
+        modname, ext = os.path.splitext(fname)
+        finder = machinery.FileFinder(dir_,
+                        (machinery.SourceFileLoader, [".py"]),
+                        (machinery.SourcelessFileLoader, [".pyc", ".pyo"])
+                    )
+        loader, portion = finder.find_loader(modname)
+        # note we assume loader here, and not portion
+        if loader is None:
+            raise ImportError("Can't load file: %s" % path)
+        else:
+            # is this supported?  shrugs
+            loader.name = module_id
+        return loader.load_module(module_id)
 else:
     import imp
+    import os
     def load_module(module_id, path):
-        fp = open(path, 'rb')
+        dir_, fname = os.path.split(path)
+        modname, ext = os.path.splitext(fname)
+        fp, pathname, description = imp.find_module(modname, [dir_])
         try:
-            mod = imp.load_source(module_id, path, fp)
+            mod = imp.load_module(module_id, fp, path, description)
             if py2k:
                 source_encoding = parse_encoding(fp)
                 if source_encoding:

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

of course the above system assumes we're using the finder system, which is sort of the complicated part here. If we fed in completed .py .pyc .pyo paths we could perhaps just call directly to the appropriate loader function.

[sqlalchemy-bot]

Greg Yang (@eulerreich) wrote:

Thanks for the fast reply! I'm only working with py2.7 so that works for
me.

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

please try out 410898a; it includes a "sourceless" test and runs with py2k/3k with and without the -O flag

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• changed status to closed

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

oh, you know this isn't going to work for the env.py case, let me look at that.

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• changed status to reopened

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

OK, b3c4390 should do it.

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• changed status to closed

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

sorry, I have to alter this so that a flag is required in alembic.ini to enable. reading the .pyc files is very annoying and other users are reporting it as a bug.

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• changed status to reopened

[sqlalchemy-bot]

Michael Bayer (@zzzeek) wrote:

• Altered the support for "sourceless" migration files (e.g. only
  .pyc or .pyo present) so that the flag "sourceless=true" needs to
  be in alembic.ini for this behavior to take effect.
  fixes load_module in compat.py doesn't handle .pyo #163

→ 623c7e7

[sqlalchemy-bot]

Changes by Michael Bayer (@zzzeek):

• changed status to closed