ImportError when package is symlinked on Windows

[jaraco]

BPO	6727
Nosy	@amauryfa, @jaraco, @ncoghlan, @merwok, @bitdancer, @briancurtin, @asvetlov, @florentx
Files	test_import_symlink_package.py wstat-res.py f3c7f4243a04.diff 55d164f36389.diff 577b717055bc.diff 2b7bf4e5cb9f.diff 1cdb64480494.diff 4aaf78f0dd10.diff 4fdbc9f74235.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-04-16.18:10:23.945>
created_at = <Date 2009-08-18.15:13:02.752>
labels = ['interpreter-core', 'type-bug', 'OS-windows']
title = 'ImportError when package is symlinked on Windows'
updated_at = <Date 2014-04-16.18:10:23.943>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2014-04-16.18:10:23.943>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-16.18:10:23.945>
closer = 'r.david.murray'
components = ['Interpreter Core', 'Windows']
creation = <Date 2009-08-18.15:13:02.752>
creator = 'jaraco'
dependencies = []
files = ['21962', '22008', '24229', '24230', '24231', '24245', '24246', '24250', '24251']
hgrepos = ['103', '105']
issue_num = 6727
keywords = ['patch', 'needs review']
message_count = 60.0
messages = ['91699', '95737', '95748', '135711', '135712', '135713', '135714', '135717', '135718', '135719', '135720', '135741', '135743', '135744', '135745', '135761', '135766', '136100', '136112', '136184', '136186', '136188', '136332', '136372', '136602', '136604', '136605', '147797', '147798', '147799', '147800', '147801', '147803', '147805', '150986', '150987', '151205', '151206', '151207', '151208', '151295', '151394', '151403', '151470', '151776', '155159', '155160', '155161', '155162', '155167', '155169', '160607', '163026', '163029', '163032', '163041', '163247', '163248', '163281', '216536']
nosy_count = 12.0
nosy_names = ['amaury.forgeotdarc', 'jaraco', 'ncoghlan', 'eric.araujo', 'r.david.murray', 'brian.curtin', 'asvetlov', 'flox', 'santoso.wijaya', 'python-dev', 'wkornewald', 'alexreg']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6727'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[jaraco]

Python reports an import error, even when a valid package exists on
sys.path, if that package is a symlink directory and would otherwise be
importable.

The attached test script demonstrates the limitation on Python 2.6.2 and
3.1.0.

I suspect that the underlying import routines have some assumptions
about the file system that exclude symlinked directories.

[ncoghlan]

Given that you had to use ctypes to create the symlinks on Windows in
the first place, I'd say you're right :)

Probably the first step would be to make sure the nt OS and path modules
get Windows symlink support (although even that may not be enough, due
to the bootstrapping issues with file access and the import machinery).

[jaraco]

I agree that a proper solution might include full symlink support.

However, symlink support for windows (bpo-1578269) is taking some time
to develop and will likely never be available for Python 2.

My feeling is that a symlink naive implementation (such as Python is
now), should treat symlinks in a naive manner -- that is, treat them as
regular files and directories and let the OS perform the abstraction.

I hope to begin investigating this after I've closed out bpo-1578269,
although I'd be happy if someone else wanted to address this issue
beforehand.

If it's possible to address this in a straightforward way independent of
a symlink-aware Python interpreter, that would be desirable.

[wkornewald]

Symlinked packages used to work on Windows until recently, but a few days ago Microsoft published a few security patches and things stopped working. On my local machine I uninstalled all Visual C++ Runtime library patches (they were installed automatically on May 5th) and symlinks started working on my machine, again. On my friend's computer symlinked packages suddenly stopped working today, but he doesn't seem to have the Visual C++ Runtime libs, so we don't know how to work around this issue on his computer. Could someone please fix support for symlinked packages? It's an essential feature during development.

[briancurtin]

Are you able to narrow it down to which security update(s) caused the breakage?

[wkornewald]

I uninstalled these three security patches:

• KB2467173: http://support.microsoft.com/kb/2467173
• KB2467174: http://support.microsoft.com/kb/2467174
• KB2467175: http://support.microsoft.com/kb/2467175

Without those patches symlinked packages work for me. I didn't try to narrow it down to the exact patch. Instead I uninstalled all Visual C++ Redistributable packages in order to prevent those patches from ever being installed again. Should I risk breaking symlink support again to find the source of the problem?

[briancurtin]

Don't do anything you're not comfortable with. If you can get your system to whatever state it was in the past where things worked properly, feel free to dig into it. I will try to look into this situation and see if there's anything in these security updates that looks related.

[jaraco]

That's interesting. You're saying that you've been using symlinked packages for some time and that it works for you. I filed this bug because it's never worked for me. Can you describe a little bit about the environment in which you've encountered this? Specifically, which Python version, how were the symlinks created, and what is the sys.path like?

[jaraco]

And since you seem to have some systems that honor symlinked packages, can you run the attached test_import_symlink_package.py and report the results?

[briancurtin]

Out of the patches listed, http://www.microsoft.com/technet/security/advisory/2269637.mspx is the only part that appears to be related in any way, although it doesn't specify a whole lot on the surface. The title is "Insecure Library Loading Could Allow Remote Code Execution", so I'm guessing they could have made some changes under the hood that could affect us.

Before we go down that road, running the script Jason suggested would be helpful in proving what works and doesn't work depending on what patches are installed and what version is running.

[wkornewald]

I'm using Python 2.6.6 on Windows 7 Professional with the latest service pack. My system is pretty bare bones. Do you have Visual Studio or the Visual Studio Redistributables?

Here's the output of your script (the last exception seems to be caused by the .pyc file, but apart from that it seems to work):

C:\Users\wkornewald\Downloads\a>test_import_symlink_package.py
Problem does not exist here.
Traceback (most recent call last):
  File "C:\Users\wkornewald\Downloads\a\test_import_symlink_package.py", line 66, in <module>
    os.rmdir(tagged)
WindowsError: [Error 145] The directory is not empty: 'sample-tagged'

C:\Users\wkornewald\Downloads\a>dir sample-tagged
Volume in drive C has no label.
Volume Serial Number is 50C8-BE90

Directory of C:\Users\wkornewald\Downloads\a\sample-tagged

10.05.2011 19:04 <DIR> .
10.05.2011 19:04 <DIR> ..
10.05.2011 19:04 107 __init__.pyc
1 File(s) 107 bytes
2 Dir(s) 66.517.442.560 bytes free

[jaraco]

Thanks for that. The output is very telling.

First, it shows there's a bug in the test script where an exception occurs when it succeeds because a .pyc file is created and not properly cleaned up.

Second, it demonstrates that the bug as I reported it is not occurring in your environment.

This latter result tells me that I may be experiencing the same issue as you, but encountered it almost two years ago. This finding leads me to believe that the issue is less about the latest hotfix and more related to the VS redistributables. I almost certainly have some form of these redistributables installed as I usually have some form of Visual Studio installed.

The fact that it works in one environment and not another means I should be able to find the regression. I'll install a barebones Windows box and see how it behaves there.