.pyc files created readonly if .py file is readonly, python won't overwrite

[pdsimanyi]

BPO	6074
Nosy	@warsaw, @amauryfa, @ncoghlan, @pitrou, @merwok, @briancurtin, @ericsnowcurrently
Files	showpycreadonly.sh: Script reproducing readonly pyc creation showpycreadonlysleep.sh: sleep version of showpycreadonly problem reproduction script issue6074.patch issue6074_3.3_eric.snow.diff: test and fix for 3.3

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2012-10-19.13:45:54.502>
created_at = <Date 2009-05-20.22:04:21.579>
labels = ['interpreter-core', 'type-bug', 'OS-windows']
title = ".pyc files created readonly if .py file is readonly, python won't overwrite"
updated_at = <Date 2013-01-14.21:11:03.739>
user = 'https://bugs.python.org/pdsimanyi'

bugs.python.org fields:

activity = <Date 2013-01-14.21:11:03.739>
actor = 'markon'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2012-10-19.13:45:54.502>
closer = 'ncoghlan'
components = ['Interpreter Core', 'Windows']
creation = <Date 2009-05-20.22:04:21.579>
creator = 'pdsimanyi'
dependencies = []
files = ['14025', '14083', '26912', '27028']
hgrepos = []
issue_num = 6074
keywords = ['patch']
message_count = 42.0
messages = ['88138', '88263', '88369', '88388', '88389', '88390', '88397', '88399', '88402', '88420', '90246', '111626', '112478', '115500', '115501', '115502', '115665', '168566', '168573', '168611', '168613', '168624', '168982', '169250', '169860', '170067', '170075', '173320', '173322', '173328', '173331', '173333', '173335', '173336', '173340', '173342', '173343', '173344', '173358', '173429', '173430', '173441']
nosy_count = 12.0
nosy_names = ['barry', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'eric.araujo', 'brian.curtin', 'pdsimanyi', 'mucisland', 'Steve.Thompson', 'python-dev', 'eric.snow', 'kevin.chen']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6074'
versions = ['Python 3.3', 'Python 3.4']

[pdsimanyi]

Some source code control tools, like Perforce, by default sync files readonly, and it's useful to
leave them readonly so that you can mark files changed by making them writeable even if they're not
checked out (e.g. working offline).

When python implicitly compiles a .py file that's readonly, it will create a .pyc file that's
readonly. This would be fine, except that when the .py file changes, python *silently* refuses to
overwrite a .pyc file that's readonly, even if the directory is completely writable by the user
executing Python.

The attached script shows this behavior. Running "python -v" will produce the following line
showing this:

  import b # from c:\Documents and Settings\psimanyi\tmp\b.py
  # can't create c:\Documents and Settings\psimanyi\tmp\b.pyc

And because I hoped this was a Windows-only problem, I tested on OSX (Leopard) and Ubuntu (9.10
alpha): it occurs on all three platforms.

Thanks!

--------------------------------

I fixed showpycreadonly.sh after attaching it, so to be sure you have the current version, it
follows:

rm -f [ab].py{,c}

echo 'import b' > a.py
echo 'print "b"' > b.py
ls -l b.py
python a.py
ls -l b.pyc
rm b.pyc
chmod -w b.py
python a.py
ls -l b.pyc
touch b.py
python -v a.py 2>&1 | grep b.py

[markon]

It seems Python is using a default mask to compile modules.
If you open a file b.py as "r", and you import it from a.py, you can see
that b.pyc is:
-rwxr-xr-x 1 marco marco 9 24 mag 16:17 a.py
-r--r--r-- 1 marco marco 10 24 mag 16:17 b.py
-rw-r--r-- 1 marco marco 122 24 mag 16:46 b.pyc

If you try to run a.py:
-rwxr-xr-x 1 marco marco 9 24 mag 16:17 a.py
-r-xr-xr-x 1 marco marco 10 24 mag 16:17 b.py

you'll get:
-rw-r--r-- 1 marco marco 122 24 mag 16:49 b.pyc

as you can see, there is a default "mode" to create compiled modules...

[markon]

Which version are you using?

I've seen the source code of import.c (2.5 and 2.6) and it seems that in
the 2.6 some bugfixes/features have been added.

However, I've just used the version 2.7 and it works fine (as 2.6, since
they have the same code (at least in import.c), until now).

bye

[pdsimanyi]

Oops, I was not accurate in reporting behavior: I now believe that this fails on Windows XP, but may work
fine on Linux. Below are my results, showing reproducible failure on XP. The most important thing about
this is: on XP, once the .pyc file is "broken", Python will not overwrite it, and it must be *manually*
removed. This is bad!

I will append Ubuntu results in a few minutes.

To reproduce the problem, I had to modify the script to include a "sleep", see attached
showpyreadonlysleep.sh (cygwin bash script).

Here's the results of running showpyreadonlysleep.sh, then manually reproducing the problem with
"touch/python":

$ ./showpycreadonlysleep.sh
umask = 0022
CYGWIN_NT-5.1 psimanyi_xp 1.5.25(0.156/4/2) 2008-06-12 19:34 i686 Cygwin
Python 2.6.2
Running python.exe from: /cygdrive/c/Python26/python
-rw-r--r-- 1 psimanyi mkgroup-l-d 10 May 26 12:43 b.py
b
-rwx------+ 1 psimanyi mkgroup-l-d 137 May 26 12:43 b.pyc
b
-r-x------+ 1 psimanyi mkgroup-l-d 137 May 26 12:43 b.pyc
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

And now, running manually:

$ touch b.py

psimanyi@psimanyi_xp ~
$ python -v a.py 2>&1 | grep 'b\.py'
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

psimanyi@psimanyi_xp ~
$ touch b.py

psimanyi@psimanyi_xp ~
$ python -v a.py 2>&1 | grep 'b\.py'
# c:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from c:\Documents and Settings\psimanyi\b.py
# can't create c:\Documents and Settings\psimanyi\b.pyc

Again, this shows that without removing the .pyc manually, the pyc will not be overwritten, and therefore
program startup will be slow!

[pdsimanyi]

Re: Ubuntu: I can't reproduce the problem. At this point, I consider this problem Windows XP-only.

Re: Windows & bash: I decided to verify that this problem occurs even if running outside of Bash in
the native (and wonderful) shell CMD.EXE. It *does* occur. Here's the state:

[1] Run my script in Bash, which fails as in my previous append from today.
[2] Start CMD.EXE. Then run a "touch b.py" in Bash, then verify using Bash "ls -l" that b.py is
protection 444 (readonly).
[3] In CMD.EXE, run "python -v a.py" (I don't know how to use something like grep in CMD.EXE, so I
just manually copy/pasted the "python -v" lines.) The following lines in CMD.EXE's output show that
the problem still occurs in CMD.EXE:

# C:\Documents and Settings\psimanyi\b.pyc has bad mtime
import b # from C:\Documents and Settings\psimanyi\b.py
# can't create C:\Documents and Settings\psimanyi\b.pyc
b
# clear __builtin__._

If you are unable to reproduce this on XP, I will be happy to help.

I would not be surprised if this had something to do with the granularity of mtime or some other
oddity of Windows, because: if I delete the sleep in the Bash script, then rerun the Bash script very
quickly using "control-P control-J" typed ahead, sometimes the Bash script fails and sometimes it
succeeds! (Nice work, Windows.)

Thanks much.

[pdsimanyi]

I may have a reason to analyze that would explain why this fails on
Windows: the non-Windows code path in import.c's open_exclusive() contians
an unlink(filename), but the Windows path does not.

I'm going to try to build Python on Windows and add an unlink for both
code paths.

More later.

[pdsimanyi]

I have a fix for this. The code is Windows-only. It has been verified to work for attached test script
showpycreadonlysleep.sh. It simply adds a "chmod" call. The issue is that the unlink() call silently fails
if the file is readonly, but unlink() succeeds if chmod() makes the file writable.

Out company would really appreciate having this fix integrated into 2.6 since we use the ActiveState 2.6
builds on Windows. I haven't test this on non-Windows platforms but it should not change the behavior on
non-Windows platforms if the #ifdef MS_WINDOWS is correct. The diff is below:

$ svn diff
Index: import.c

===================================================================

--- import.c    (revision 72946)
+++ import.c    (working copy)
@@ -840,6 +840,7 @@
 static FILE *
 open_exclusive(char *filename, mode_t mode)
 {
+
 #if defined(O_EXCL)&&defined(O_CREAT)&&defined(O_WRONLY)&&defined(O_TRUNC)
        /* Use O_EXCL to avoid a race condition when another process tries to
           write the same file.  When that happens, our open() call fails,
@@ -848,6 +849,9 @@
           writable, the file will never be written.  Oh well.
        */
        int fd;
+#ifdef MS_WINDOWS
+        (void) chmod(filename, 0600);
+#endif
        (void) unlink(filename);
        fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC
 #ifdef O_BINARY

It may be appropriate to document that the chmod() is only required on Windows, and therefore it is only
called on Windows to avoid slowing down non-Windows platforms.

[pdsimanyi]

I realized that my patch, with a chmod(..., 0600), may not work under
certain conditions where the original file is owned by a different user
but the directory is group- or world-writable. A more likely correct fix
would use chmod(..., 0666).

At any rate: someone should test the various combinations of whether this
user or another user owns the original .pyc file.

[markon]

TO pdsimanyi:

chmod it's not appropriate to change the permissions as you made: 0666
et voilà.

However, I don't understand if it can be a problem since you're using
Cygwin (and Bash under Windows) or not. I think this because Windows is
not POSIX compliant and under Windows the permissions are different than
the others under *NIX-like platforms (POSIX compliant).

Maybe in these days I can check the permissions on Windows.

[pdsimanyi]

Re: chmod 0666 inappropriate: I agree that leaving a file 0666 is inappropriate, however theoretically
the code should never leave the file 0666 since the unlink will delete the file.

The code was intended to show what the problem is by demonstrating a solution. Note that the line
immediately following the chmod 0666 is an unlink call, which will delete the file if the user had
sufficient permission to chmod the file to 0666.

I suppose the meta-level description of the problem is:
[1] Without the chmod 0666, the unlink will fail if the file is 0400 or similar.
[2] With the chmod 0666, the unlink succeeds, and the file will be recreated with new permissions (not
0666: I believe my tests show 0444) after the post-unlink file creation.
[3] Someone familiar with windows should analyze, propose a solution, test it under all relevant
scenarios of file/group ownership (and again the chmod is only required on Windows), and on
XP/Vista/Win7.

The only problem I can see with the chmod 0666 is if for some reason the file still exists with bad
permissions after the unlink. Since this is a low-overhead operation compared to the time to regenerate
the .pyc file, I think that it would be reasonable to have an #ifdef MS_WINDOWS, if the file exists
after the unlink(), print a big warning message that will always be visible even if -v is not used when
invoking Python.

This is getting complicated enough that you may want to see if you can get someone else to review it
for ideas about how to address the chmod/unlink code. I am certainly not a Windows expert hence I'm
making a best Unix-like attempt to analyze and test fixes.

Good luck and thanks.

[pdsimanyi]

I verified that launching the script from a CMD.EXE window instead of
from Cygwin also reproduces the "can't overwrite ZZZ.pyc". Here's a
sample of "python -v" showing this:

 # C:\depot\central\ntf\tools\bin\build1.pyc has bad mtime
 import build1 # from C:\depot\central\ntf\tools\bin\build1.py
 # can't create C:\depot\central\ntf\tools\bin\build1.pyc

If you would like some help, I can discuss this with Windows expert
programmers here and see whether they can suggest a solution you might
find acceptable. Thank you.

[mucisland]

Applies also to Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on win32.

[pitrou]

Apparently this may have become Windows-specific at some point. I can't reproduce under Linux with Python 2.6.5, 2.7 or 3.2.

I get a strange warning with -v under 2.7, though:

$ touch b.py && ~/cpython/27/python -v a.py 2>&1 | grep b.py
# /home/antoine/py3k/__svn__/b.pyc has bad mtime
import b # from /home/antoine/py3k/__svn__/b.py
# wrote /home/antoine/py3k/__svn__/b.pyc

("bad mtime"??)