importlib/_bootstrap.py write_bytecode raises an IOError if it can't open the .pyc file for writing

[davidmalcolm]

BPO	7187
Nosy	@brettcannon, @pitrou, @davidmalcolm
Files	python-3.1.1-importlib-fix-handling-of-readonly-pyc-files.patch: Patch to extend exception handling to also cover failure to open the .pyc for writing

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2009-11-07.23:57:58.479>
created_at = <Date 2009-10-22.20:37:06.079>
labels = ['type-bug', 'library']
title = "importlib/_bootstrap.py write_bytecode raises an IOError if it can't open the .pyc file for writing"
updated_at = <Date 2009-11-07.23:57:58.478>
user = 'https://github.com/davidmalcolm'

bugs.python.org fields:

activity = <Date 2009-11-07.23:57:58.478>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2009-11-07.23:57:58.479>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2009-10-22.20:37:06.079>
creator = 'dmalcolm'
dependencies = []
files = ['15185']
hgrepos = []
issue_num = 7187
keywords = ['patch']
message_count = 5.0
messages = ['94371', '94374', '94381', '94388', '95028']
nosy_count = 3.0
nosy_names = ['brett.cannon', 'pitrou', 'dmalcolm']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7187'
versions = ['Python 3.1']

[davidmalcolm]

I'm working on an RPM package of Python 3.1.1 for Fedora [1]. Within my RPMs are
precompiled .pyc and .pyo files that are not writable by users, and are hardlinked
together if they have equal content. For example:
$ ls -al /usr/lib/python3.1/tkinter/init.*
-rw-r--r--. 1 root root 157859 2009-10-22 15:43
/usr/lib/python3.1/tkinter/init.py
-rw-r--r--. 2 root root 226817 2009-10-22 15:42
/usr/lib/python3.1/tkinter/init.pyc
-rw-r--r--. 2 root root 226817 2009-10-22 15:42
/usr/lib/python3.1/tkinter/init.pyo

I see numerous IOError permission errors running the regression test suite, with
importlib/_bootstrap.py trying and failing to open the .pyc files for writing.
The IOError bubbles up each time and causes the test to fail.

Here's a minimal reproducer (assuming such an install as above)
>>> from tkinter import Tcl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 151, in decorated
    return fxn(self, module)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 399, in load_module
    return self._load_module(module)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 324, in _load_module
    code_object = self.get_code(module.__name__)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 443, in get_code
    self.write_bytecode(fullname, data)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 171, in inner
    return method(self, name, *args, **kwargs)
  File "/usr/lib/python3.1/importlib/_bootstrap.py", line 525, in write_bytecode
    file = _io.FileIO(bytecode_path, 'w')  # Assuming bytes.
IOError: [Errno 13] Permission denied: '/usr/lib/python3.1/tkinter/__init__.pyc'

Upon investigating Lib/importlib/_bootstrap.py, the logic for handling permission
errors in "write_bytecode" appears to be wrong:

• the initializer for _io.FileIO can raise an "IOError" if it can't open the
  .pyc file for writing
  • see Modules/_io/fileio.c:fileio_init, the exception is indeed raised if the
    call to:
    self->fd = open(name, flags, 0666);
    fails.

So it looks like the try/except clause needs to be extended to also cover opening
the file for writing, as well as writing the bytes to it.

I'm attaching a patch that makes this change.

With this change, the above failing import succeeds, and an invocation of
/usr/lib/python3.1/test/regrtest.py improves from:
276 tests OK.
39 tests failed:
test_cmd_line test_codeccallbacks test_codecencodings_cn
test_codecencodings_hk test_codecencodings_jp
test_codecencodings_kr test_codecencodings_tw test_cprofile
test_distutils test_docxmlrpc test_email test_heapq
test_htmlparser test_httpservers test_imp test_lib2to3
test_linecache test_modulefinder test_multiprocessing test_osx_env
test_plistlib test_pyclbr test_pydoc test_runpy test_socket
test_sqlite test_sundry test_tcl test_threading_local test_tk
test_ttk_guionly test_ttk_textonly test_uuid test_warnings
test_wsgiref test_xml_etree_c test_xmlrpc test_xmlrpc_net
test_zipfile
to:
304 tests OK.
10 tests failed:
test_email test_httpservers test_imp test_lib2to3 test_linecache
test_socket test_tk test_ttk_guionly test_ttk_textonly
test_zipfile

(I'm working on these other failures; some of them are due to errors in my
packaging)

FWIW the initial checkin of py3k/Lib/importlib/_bootstrap.py appears to have the
initialization of the FileIO outside the try/except block:
http://svn.python.org/view/python/branches/py3k/Lib/importlib/_bootstrap.py?
view=markup&pathrev=68698

though at that time it was _fileio._FileIO, rather than _io.FileIO. (did FileIO
only test for perms upon writing, and change behavior to test upon opening? or
has this case always led to this failure? or am I misreading this?)

Thanks
Dave

[1] https://bugzilla.redhat.com/show_bug.cgi?id=526126

[brettcannon]

Thanks for the report and potential fix, Dave. I'm swamped at the moment
but I will get to this.

[pitrou]

Wouldn't the problem disappear, though, if tkinter/init.py didn't
claim to be fresher than tkinter/init.pyc? Also, it would make the
pyc/pyo files actually useful, rather than having Python recompile the
source files again and again.
(see the modification dates in your listing)

[davidmalcolm]

Aha! Thanks pitrou, looks like I managed to mess up my packaging (I was
wondering why no no-one else had run into this, looks like no-one else
made this mistake!).

[brettcannon]

Committed to py3k in r76146 and r76147 for 3.1.

Thanks for the detailed report, Dave!