Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load_module in compat.py doesn't handle .pyo #163

Closed
sqlalchemy-bot opened this issue Dec 31, 2013 · 13 comments
Closed

load_module in compat.py doesn't handle .pyo #163

sqlalchemy-bot opened this issue Dec 31, 2013 · 13 comments
Labels

Comments

@sqlalchemy-bot
Copy link

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Migrated issue, originally created by Greg Yang (@eulerreich)

I'm in a situation where I need to run alembic in a compiled package with all source files stripped away. Right now it's giving me a file not found error with the line

fp = open(path, 'rb')

where path is the full path of where the .py file is supposed to be; so the .pyo file is right in there but load_module doesn't see it.

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Michael Bayer (@zzzeek) wrote:

this is not a simple request. the .py files here are located using a directory listing approach. Logic would need to be added that first looks in place for .pyc or .pyo files, then the load module system needs to be modified.

Also, pep3147 might need to be taken into account; this is where the files would instead be in a __pycache__ directory and there's no simple way to locate these files. The good news is that per pep3147 it seems they will continue to support a .pyc or .pyo file sitting next to the .py (http://www.python.org/dev/peps/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports). However the system you're using to "strip the .py files away" may be too simplistic to work like this - in Python 3 the .pyc file won't just be sitting there normally.

Here's a partial diff to handle being able to load the .pyc/.pyo. It's quite complicated/hacky for the Py3k case:

--- a/alembic/compat.py
+++ b/alembic/compat.py
@@ -45,14 +45,31 @@ if py2k:
 
 if py33:
     from importlib import machinery
+    import os
     def load_module(module_id, path):
-        return machinery.SourceFileLoader(module_id, path).load_module()
+        dir_, fname = os.path.split(path)
+        modname, ext = os.path.splitext(fname)
+        finder = machinery.FileFinder(dir_,
+                        (machinery.SourceFileLoader, [".py"]),
+                        (machinery.SourcelessFileLoader, [".pyc", ".pyo"])
+                    )
+        loader, portion = finder.find_loader(modname)
+        # note we assume loader here, and not portion
+        if loader is None:
+            raise ImportError("Can't load file: %s" % path)
+        else:
+            # is this supported?  shrugs
+            loader.name = module_id
+        return loader.load_module(module_id)
 else:
     import imp
+    import os
     def load_module(module_id, path):
-        fp = open(path, 'rb')
+        dir_, fname = os.path.split(path)
+        modname, ext = os.path.splitext(fname)
+        fp, pathname, description = imp.find_module(modname, [dir_])
         try:
-            mod = imp.load_source(module_id, path, fp)
+            mod = imp.load_module(module_id, fp, path, description)
             if py2k:
                 source_encoding = parse_encoding(fp)
                 if source_encoding:

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Michael Bayer (@zzzeek) wrote:

of course the above system assumes we're using the finder system, which is sort of the complicated part here. If we fed in completed .py .pyc .pyo paths we could perhaps just call directly to the appropriate loader function.

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Greg Yang (@eulerreich) wrote:

Thanks for the fast reply! I'm only working with py2.7 so that works for
me.

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Michael Bayer (@zzzeek) wrote:

please try out 410898a; it includes a "sourceless" test and runs with py2k/3k with and without the -O flag

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Changes by Michael Bayer (@zzzeek):

  • changed status to closed
@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Michael Bayer (@zzzeek) wrote:

oh, you know this isn't going to work for the env.py case, let me look at that.

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Changes by Michael Bayer (@zzzeek):

  • changed status to reopened
@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Michael Bayer (@zzzeek) wrote:

OK, b3c4390 should do it.

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Dec 31, 2013

Changes by Michael Bayer (@zzzeek):

  • changed status to closed
@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Mar 14, 2014

Michael Bayer (@zzzeek) wrote:

sorry, I have to alter this so that a flag is required in alembic.ini to enable. reading the .pyc files is very annoying and other users are reporting it as a bug.

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Mar 14, 2014

Changes by Michael Bayer (@zzzeek):

  • changed status to reopened
@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Mar 14, 2014

Michael Bayer (@zzzeek) wrote:

  • Altered the support for "sourceless" migration files (e.g. only
    .pyc or .pyo present) so that the flag "sourceless=true" needs to
    be in alembic.ini for this behavior to take effect.
    fixes #163

623c7e7

@sqlalchemy-bot
Copy link
Author

@sqlalchemy-bot sqlalchemy-bot commented Mar 14, 2014

Changes by Michael Bayer (@zzzeek):

  • changed status to closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.