Bugfix: OS X ordinary users do not have permission to compile python stdlib's pyc, st_mtime do not change #5022

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
3 participants
Contributor

dongweiming commented Feb 4, 2014

When i use ipython autoreload on OS X 10.8.5, Sometimes can show the following error(In ubuntu it did not happen), For example:

In [1]: from lib2to3.refactor import get_all_fix_names

In [2]: lib2to3.fixes
[autoreload of lib2to3.fixer_util failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name token
]
[autoreload of lib2to3.pgen2.pgen failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name grammar
]
[autoreload of lib2to3.pgen2.driver failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name grammar
]
[autoreload of lib2to3.refactor failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name driver
]
[autoreload of lib2to3.btm_matcher failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name pytree
]
[autoreload of lib2to3.pgen2.parse failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name token
]
[autoreload of lib2to3.pgen2.tokenize failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name token
]
[autoreload of lib2to3.pgen2.grammar failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name token
]
[autoreload of lib2to3.btm_utils failed: Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/IPython/extensions/autoreload.py", line 233, in check
    superreload(m, reload, self.old_objects)
ImportError: cannot import name pytree
]
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-10-d6feeee22b95> in <module>()
----> 1 lib2to3.fixes

NameError: name 'lib2to3' is not defined

You can see some ImportErrors. At last, I found that because in osx, ordinary users call stdlib, no pyc compiled the module, Use sys.modules.get get more message:

# On OS X
<module 'lib2to3.pgen2.literals' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib2to3/pgen2/literals.py'>
# On Ubuntu
<module 'lib2to3.pgen2.literals' from '/usr/lib/python2.7/lib2to3/pgen2/literals.pyc'>

osx heen using .py:

ll /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib2to3/fixer_util.py*
-rw-r--r-- 1 root wheel 15K Feb  4 17:50 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib2to3/fixer_util.py
-rw-r--r-- 1 root wheel 16K Feb  4 16:25 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib2to3/fixer_util.pyc
-rw-r--r-- 1 root wheel 16K Jan  1  2012 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib2to3/fixer_util.pyo

So os.stat(py_filename).st_mtime > os.stat(pyc_filename).st_mtime, I added a hack:

When has get the py_filename/pyc_filename's st_mtime, try to compile this py_name, if raise IOError with Permission denied then continue

Owner

takluyver commented Feb 4, 2014

Thanks for investigating this so thoroughly, but I don't think this is quite the right fix: if your .py files have been changed more recently than the .pyc files, then there's something odd going on. And having libraries installed in non-user-writable locations is not specific to OSX. I think this is another pointer that autoreload shouldn't rely on examining .pyc files, which is issue #4127.

Contributor

dongweiming commented Feb 5, 2014

Sorry, take a bugfix and a feature together. Today i read the code, and do some try. I found i was wrong:

This ImportError bug because on OSX make module.dict.clear() ,deleted of these key can make module that contain absolute import reload ImportError

I see #4127, It needs to solve the cache a lot.

pre_run_code_hook make every line excute check(),
Whether it is not a call to reload the module, the module will traverse sys.modules, Most of them are unnecessary.

You can see [this commit)[https://github.com/dongweiming/ipython/commit/8bc38b699c5fe4ffc25d9c2e5099f1c032eb285e], But travis-ci is failure. I do not understand this test‘s principle

Owner

minrk commented Mar 7, 2014

@takluyver should this be closed now that #5044 is merged?

Owner

takluyver commented Mar 7, 2014

Yep, thanks.

takluyver closed this Mar 7, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment