test_nt_helpers fails with case difference in drive letter

[tjguk]

BPO	34195
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba
PRs	bpo-34195: Fix case-sensitive comparison in test_nt_helpers #8448

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tjguk'
closed_at = <Date 2018-07-25.19:12:46.957>
created_at = <Date 2018-07-23.07:41:35.932>
labels = ['type-bug', '3.8', 'OS-windows']
title = 'test_nt_helpers fails with case difference in drive letter'
updated_at = <Date 2018-07-25.19:12:46.956>
user = 'https://github.com/tjguk'

bugs.python.org fields:

activity = <Date 2018-07-25.19:12:46.956>
actor = 'tim.golden'
assignee = 'tim.golden'
closed = True
closed_date = <Date 2018-07-25.19:12:46.957>
closer = 'tim.golden'
components = ['Windows']
creation = <Date 2018-07-23.07:41:35.932>
creator = 'tim.golden'
dependencies = []
files = []
hgrepos = []
issue_num = 34195
keywords = ['patch']
message_count = 9.0
messages = ['322185', '322187', '322189', '322192', '322203', '322211', '322219', '322352', '322388']
nosy_count = 5.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = ['8448']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34195'
versions = ['Python 3.8']

[tjguk]

From a fresh build on Win10 with VS2017:

python -munittest -v test.test_ntpath.TestNtpath.test_nt_helpers

gives the following error:

======================================================================
FAIL: test_nt_helpers (test.test_ntpath.TestNtpath)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "c:\work-in-progress\python\cpython\lib\test\test_ntpath.py", line 432, in test_nt_helpers
    self.assertEqual(drive, nt._getvolumepathname(sys.executable))
AssertionError: 'c:\\' != 'C:\\'
- c:\
? ^
+ C:\
? ^

Ad hoc, it appears that:

sys.executable gives a lower-case path while nt._getvolumepathname gives an upper-case drive letter.

While the test could be trivially fixed, it seems worth investigating a little further to see what's happening inside nt._getvolumepathname

[tjguk]

import nt, sys; assert sys.executable.startswith(nt._getvolumepathname(sys.executable))

This code fails only when run from the python.bat as created by pcbuild\build.bat. The obvious difference is that the batch file sets PYTHONHOME which, presumably, is used to form sys.executable (haven't checked the startup code yet).

The docs for GetVolumePathName [*] don't specify that the drive letter of the path returned will be upper-case, but it doesn't seem unlikely.

So... it looks as though the test is unduly sensitive to case-differences in the face of something like PYTHONHOME which affects the way in which sys.executable is formed.

Phew! I'll put a test patch together later...

[*] https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getvolumepathnamew

[eryksun]

os__getvolumepathname_impl in Modules/posixmodule.c doesn't directly modify the case. So that leaves WinAPI GetVolumePathNameW as the culprit, but, according to the following test, apparently not in Windows 10.0.17134:

    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    path = (ctypes.c_wchar * 4)()
    kernel32.GetVolumePathNameW(r'c:\windows', path, 4)
    
    >>> path.value
    'c:\\'

On a related note, nowadays we need to be careful to only use a case-insensitive comparison for drive letters and other device names. On account of WSL, recent versions of NTFS support flagging directories as case sensitive 1. For example, a volume may be mounted using a junction in a case-sensitive directory.

[tjguk]

Thanks, @eryksun. Whatever the reason, it's consistently failing in the way I describe. A case-insensitive test is obviously good for that and for the other reasons you give, so I'll patch the test anyway.

[tjguk]

@eryksun almost idly I ran your ctypes code in the built interpreter. As written, it produces a lower-case c:\\ as yours did.

But...

Running Debug|Win32 interpreter...
Python 3.8.0a0 (heads/master:7a3056f, Jul 23 2018, 08:23:33) [MSC v.1912 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes, sys
>>>
>>> sys.executable
'c:\\work-in-progress\\python\\cpython\\PCbuild\\win32\\python_d.exe'
>>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
>>> path = (ctypes.c_wchar * 4)()
>>> kernel32.GetVolumePathNameW(sys.executable, path, 4)
1
>>> path.value
'C:\\'
>>>

[tjguk]

I think I've got down to the determining factor. For info:

PYTHONHOME has nothing to do with it: the same thing happens if I cd into PCBuild\win32 and run python_d.exe directly

For historical reasons the directory in which I'm building (c:\work-in-progress) is actually a junction to c:\users\<me>\work-in-progress. There is some commentary in the API docs about traversing junctions, so presumably that's special-cased in some way which results in an uppercase drive letter.

If I rebuild in, eg, c:\temp which is a normal directory, I don't see the uppercase conversion. So, while I still need to fix the underlying test to be case-insensitive, it looks like the Mystery of the Uppercase Drive Letter is at least mostly solved.

[eryksun]

the directory in which I'm building (c:\work-in-progress) is actually
a junction to c:\users\<me>\work-in-progress

That makes sense. GetVolumePathName traverses backwards from the final component. If it reaches a reparse point (other than a junction that targets a volume name of the form "\??\Volume{GUID}\"), it has to start over from the reparsed final path obtained from GetFinalPathNameByHandle. Subsequently it's traversing back from "C:\Users\<me>\work-in-progress" down to upper-case "C:\".

[tjguk]

New changeset ff64add by Tim Golden in branch 'master':
bpo-34195: Fix case-sensitive comparison in test_nt_helpers (GH-8448)
ff64add

[tjguk]

Test fixed to ignore case and volume differences between paths