New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Buffer overflow when listing deeply nested directory #45320
Comments
This code: import os
import os.path
TARGET='C:/code/python/foo'
base = TARGET
for x in range(200):
subdirs = os.listdir(base)
base = os.path.join(base, subdirs[0])
print base Produces a TypeError (buffer overflow) when run on a to deeply nested directory for windows to handle: .. more output here..
C:code/python/foo\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.p
ng\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png\foo bar.png
Traceback (most recent call last):
File "killdir.py", line 6, in <module>
subdirs = os.listdir(base)
TypeError: listdir() argument 1 must be (buffer overflow), not str |
Worked as expected for me on Mac OS X 10.4.10 running from
where the range command is analogous to Python's range
The for loop barfed after making directory 0205. In Python I then executed these statements: import os.path
base = "/Users/skip/tmp/deep"
for x in range(210):
subdirs = os.listdir(base)
base = os.path.join(base, subdirs[0])
print base This went until it got to dir 0200 where it raised an
which stands to reason since base was 1025 characters long Skip |
To rephrase Skip's comment: Can you please report what operating system and Python version you are using? |
Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 |
Can you please explain what specifically you consider a bug here? I can see that the error message is confusing, so it could be improved. However, there is nothing we can do to make the error go away. The Microsoft C library simply does not support file names longer than MAX_PATH; you have to use Unicode file names to go beyond this limit. |
Yes, it is the error message and the exception that is the problem. First, it shouldn't raise TypeError (which indicates a programming error), it should raise either IOError, OSError or WindowsError. Second, the exception message is whacky: "listdir() argument 1 must be (buffer overflow), not str" I realize that it is probably impossible to detect this specific error condition but I still want something more explanatory than what it currently is. |
Can somebody please look at this bug? It still appears in SCons 2.2.0 on Windows 7 when it tries to do a os.listdir on C:\Program Files\Microsoft Visual Studio 9.0\VC\ATLMFC\INCLUDE;C:\Program Files\Microsoft Visual Studio 9.0\VC\INCLUDE;C:\Program Files\Microsoft SDKs\Windows\v6.0A\include;\build\libevent-2.0.20-stable-debug\include;\build\libevent-2.0.20-stable-debug\WIN32-Code;\build\gtest-1.4.0\include; |
Can you please report what Python version you are using? |
This is on Python 2.7.3 on Win7 32-bit, sorry. |
This issue is related to parsing of "et#" format which is used only in listdir() and _getfullpathname() under Windows. PyArg_ParseTuple() throws TypeError exception for multiple conversion errors (in this case it is an overflow of a static buffer). There are several ways to solve this issue:
Martin, what is your decision? |
See also bpo-4071. |
I suggest we close this as "won't fix" since I don't see how we can justify spending time working around a known limitation of Windows. |
Benjamin, what of the proposed options do you prefer? |
In Windows 7, FindFirstFileA uses a per-thread static buffer to decode the input bytes path to Unicode. This buffer limits the length to 259 characters (MAX_PATH - 1), even if a "\\?\" device path is used. Windows 8+ uses a dynamic buffer, but I don't see the point of switching to a dynamic buffer on our side given Windows 7 is still so widely used and the documentation still requires Unicode for long "\\?\" paths. Ideally, I think 2.7 should raise the same exception as 3.5 does in this case 1. For example: >>> os.listdir(long_bytes_path)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: listdir: path too long for Windows |
Python 2 is EOL. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: