pkgutil.walk_packages fails on write-only directory in sys.path

[QuikFoot]

BPO	7367
Nosy	@ned-deily

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2011-10-06.22:00:32.027>
created_at = <Date 2009-11-20.07:31:23.876>
labels = ['easy', 'type-bug', 'library']
title = 'pkgutil.walk_packages fails on write-only directory in sys.path'
updated_at = <Date 2011-10-07.19:03:17.034>
user = 'https://bugs.python.org/QuikFoot'

bugs.python.org fields:

activity = <Date 2011-10-07.19:03:17.034>
actor = 'python-dev'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2011-10-06.22:00:32.027>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2009-11-20.07:31:23.876>
creator = 'Quik_Foot'
dependencies = []
files = []
hgrepos = []
issue_num = 7367
keywords = ['easy']
message_count = 9.0
messages = ['95533', '95534', '95535', '95544', '95569', '95571', '145041', '145043', '145126']
nosy_count = 3.0
nosy_names = ['ned.deily', 'Quik_Foot', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7367'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[QuikFoot]

I spend the better part of a day figuring out this relatively simple
problem with the help files in IDLE with python 2.6 on a mac OSX

I came about the solution by following the code errors all over
up and down the program files.

Eventual I came about the public folder ex..
Users/ME/Public

I spent time looking through the properties and settings tabs editing my
firewall settings and file sharing options (I reverted them when i fixed
the problem)
but i ended up removing all file sharing settings anyway.

I simply right clicked on the public folder and my permission was set
to "Write only(Drop Box)"
a simple change to "Reed and Write" fixed the IDLE error.

Hope i just saved some one some time.

[ned-deily]

Can you be more specific about how to reproduce the problem you saw?
Also, which version of Python were you using? Please copy the information
from the the first line of IDLE's Python Shell window, like so:

Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin

[QuikFoot]

Python 2.6.4
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin

From the best I can tell the error came about
when I updated my firewall/shared folder settings.
(I lost a computer on my home network due to my partner leaving me :( )
so I did a complete revamp of my network to close up any holes
and Bam no IDLE.

The solution as I said came about from simply changing the user access
on the public folder in User/ME/

I initially came across the error when attempting to get a modules list
and was returned a list of errors. One of them was the [errno 13].

Hope that clears things.

[bitdancer]

A more specific issue title will also help others hitting the same
problem...hopefully I've guessed correctly as to what it should be.

[ned-deily]

I initially came across the error when attempting to get a modules list
and was returned a list of errors. One of them was the [errno 13].

OK, now that makes more sense!

The problem is that somehow you had a write-only directory ("~/Public/Drop
Box") on your python module search path. If you type the following commands
in IDLE or in the interpreter:

import sys
sys.path

you will likely see the "Drop Box" directory in the list of paths (assuming
things haven't changed).

When you run the "modules" help command, the python help module, pydoc, uses
the pkgutil module to search the entire list of directories in sys.path and
attempts to import each package it finds in each directory to print the
module's help information (btw, there are no separate help files).

While the pkgutil.walk_packages function tries to handle most errors, it
isn't prepared to handle the case of a write-only directory on sys.path and
that's the root cause of the error of the error you saw. The following test
case demonstrates the problem:

import os, pkgutil, sys

dirname_wo = os.tempnam()
os.mkdir(dirname_wo, 0222)  # write permission only
try:
    sys.path.insert(0, dirname_wo)
    def onerror(pkgname):
        print('onerror called for package %s' % pkgname)
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
        print modname
finally:
    os.rmdir(dirname_wo)

The long-term solution is to have pkgutil.walk_packages protect itself
against os.listdir errors. A patch is needed for that.

In your case, though, you should check for and remove whatever is adding
"Drop Box" to your Python sys.path. Perhaps you have an out-of-date export
of PYTHONPATH in a shell profile (.bash_profile, etc)? Or something left
over in a site-packages pth file (perhaps by trying to install a package
from the "Drop Box" directory)? By changing the permissions as you did, you
worked around the problem but that's not the right solution and you could be
compromising the security of your drop box file.

[ned-deily]

More precisely: you should check for and remove whatever is adding
"Drop Box" or a directory within it to your Python sys.path.

[python-dev]

New changeset 096b010ae90b by Ned Deily in branch '2.7':
Issue bpo-7367: Add test case to test_pkgutil for walking path with
http://hg.python.org/cpython/rev/096b010ae90b

New changeset 1449095397ae by Ned Deily in branch '2.7':
Issue bpo-7367: Fix pkgutil.walk_paths to skip directories whose
http://hg.python.org/cpython/rev/1449095397ae

New changeset a1e6633ef3f1 by Ned Deily in branch '3.2':
Issue bpo-7367: Add test case to test_pkgutil for walking path with
http://hg.python.org/cpython/rev/a1e6633ef3f1

New changeset 77bac85f610a by Ned Deily in branch '3.2':
Issue bpo-7367: Fix pkgutil.walk_paths to skip directories whose
http://hg.python.org/cpython/rev/77bac85f610a

New changeset 5a4018570a59 by Ned Deily in branch '3.2':
Issue bpo-7367: add NEWS item.
http://hg.python.org/cpython/rev/5a4018570a59

New changeset 0408001e4765 by Ned Deily in branch 'default':
Issue bpo-7367: merge from 3.2
http://hg.python.org/cpython/rev/0408001e4765

[ned-deily]

The applied changesets correct pkgutil's walk_packages for "classic" imports to ignore unreadable directories the same way that the interpreter's import does. With this fix to pkgutil, pydoc -k also no longer fails in this case. Applied in 2.7 (for 2.7.3), 3.2 (for 3.2.3), and default (for 3.3.0).

[python-dev]

New changeset 17b5afd321a8 by Ned Deily in branch '2.7':
Issue bpo-7367: Ensure test directory always gets removed.
http://hg.python.org/cpython/rev/17b5afd321a8

New changeset ff72f76dcf43 by Ned Deily in branch '3.2':
Issue bpo-7367: Ensure test directory always gets removed.
http://hg.python.org/cpython/rev/ff72f76dcf43