obsolete macpath module dangerously broken and should be removed

[ned-deily]

BPO	9850
Nosy	@ronaldoussoren, @vstinner, @ned-deily, @merwok, @serhiy-storchaka, @yan12125, @DimitrisJim, @scottmacpherson, @Mariatta
PRs	bpo-9850: Deprecate the macpath module #1540 bpo-9850: Document macpath deprecation in Misc/NEWS #1590
Files	macpath_join_fix.patch: fix for macpath.join('', *comps) macpath_join_fix_with_test.patch: fix for macpath.join('', *comps) with a test 3.6-deprecate-macpath.patch 3.7-remove-macpath.patch 3.7-remove-macpath_ver2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-16.20:49:57.526>
created_at = <Date 2010-09-14.05:14:10.511>
labels = ['OS-mac', 'easy', '3.7']
title = 'obsolete macpath module dangerously broken and should be removed'
updated_at = <Date 2018-12-12.15:31:19.204>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2018-12-12.15:31:19.204>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-16.20:49:57.526>
closer = 'vstinner'
components = ['macOS']
creation = <Date 2010-09-14.05:14:10.511>
creator = 'ned.deily'
dependencies = []
files = ['22929', '22943', '44930', '44931', '44932']
hgrepos = []
issue_num = 9850
keywords = ['patch', 'easy']
message_count = 33.0
messages = ['116367', '116387', '116388', '116389', '116517', '116973', '142310', '142365', '142397', '217238', '217901', '227687', '227689', '247128', '247242', '277844', '277892', '277893', '278102', '293443', '293458', '293692', '293693', '293694', '293695', '293696', '293697', '293698', '293700', '293704', '293706', '293785', '331701']
nosy_count = 12.0
nosy_names = ['ronaldoussoren', 'vstinner', 'ned.deily', 'eric.araujo', 'jesstess', 'chortos', 'python-dev', 'serhiy.storchaka', 'yan12125', 'Jim Fasarakis-Hilliard', 'macpherson.scott', 'Mariatta']
pr_nums = ['1540', '1590']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue9850'
versions = ['Python 3.7']

[ned-deily]

The macpath module in the standard library purports to supply "the Mac OS 9 (and earlier) implementation of the os.path module. It can be used to manipulate old-style Macintosh pathnames on Mac OS X (or any other platform). The following functions are available in this module: normcase(), normpath(), isabs(), join(), split(), isdir(), isfile(), walk(), exists(). For other functions available in os.path dummy counterparts are available."

Old-style Mac pathnames are not further documented - in fact, the above quote is the entire external documentation for the module - but they are ones using colon separators, like ("MacHD:Users:nad:macpath_test:file"). These style path names were initially supported on Mac OS X by compatibility APIs for programs ported from classic Mac OS but those interfaces have long been deprecated in OS X and in most cases are not available in 64-bit execution mode.

Even if one did have a need to use the obsolete old-style paths, the macpath module is currently practically unusable for anything other than simple character manipulations of the path. Nearly all of the functions that actually call OS routines are broken in one or more ways.

Those that do make file system calls are calling APIs that are expecting normal posix-style ('/' separated paths) incorrectly with old-style (":) paths (ispath, isdir, lexists, etc) which means they only give correct results in the trivial case of unqualified file or directory names, i.e. those in the working directory.

The islink() function confusingly is testing whether a path points to a Finder Alias file (not a symlink). Unfortunately, the Carbon API used for that test does not exist in a 64-bit version and all of the Python Carbon modules were removed in Python 3.

$ arch -i386 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
1
$ arch -x86_64 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
False
$ python3.2 -c 'import macpath; print(macpath.islink("alias_to_file"))'
False

The underlying import errors are being masked by "try"'s:

$ arch -i386 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
$ arch -x86_64 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ResolveAliasFile'
$ python3.2 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named Carbon.File

The realpath function is also broken in 2.7, calling Carbon.File.FSResolveAliasFile with a ":" separated path when it expects a "/" path, and is totally broken in 3.x (no Carbon modules).

While macpath *could* be repaired by proper documentation, fixing the mishmash of path types internally, and supplying C wrappers and/or alternative 64-bit APIs, it does not seem worth the effort as these style paths are seldom encountered in modern code. Considering how broken the module currently is and given that it probably was simply overlooked in the Python 2 to 3 transition, I think a case could be made for immediate removal prior to Python 3.2 release and even for a 3.1.x maintenance release. Short of that, it should be cleared documented as deprecated in 3.2, 3.1, and 2.7 along with warnings about broken functionality along with added DeprecationWarnings to the module.

I can write up patches to do that depending on what the consensus is.

[ronaldoussoren]

I'm -0 on removing the module wholesale because parts of OSX still use MacOS9 style paths (in particular some Carbon APIs and AppleScript).

I'm +1 on removing all functions that aren't path manipulation code, that is 'macpath.join' would stay while 'macpath.islink' would go, and fixing the documentation.

And to be complete: I'm -1 on fixing macpath.islink and other functions that access a life filesystem because the macpath module is only meant to be used for that on MacOS9 and that platform is no longer supported. Path manipulation code is still useful.

BTW. The module cannot be removed from 2.7 and 3.1 due to API stability requirements.

[vstinner]

The solution may be different depending on Python version. I propose to keep macpath in Python 2.7, just because it's too late to change such thing in Python2. But we may mark macpath as deprecated, eg. "macpath will be removed in Python 3.2" (see above). I suppose that the same treatement can be used on Python 3.1 (maybe with a different warning type/message).

But for Python 3.2, I suggest to drop macpath. Said differently: drop Mac OS 9 support. If you really need Mac OS 9 support, continue to use Python 2.7. If I understood correctly, macpath is not used for os.path in Python 3.2, and it is broken (at least on 64 bits build).

But well, I am not a Mac programmer, so I don't know if Mac OS 9 is still commonly installed or not.

[ronaldoussoren]

MacOS9 is already unsupported, except for macpath all traces of OS9 support are gone with the possible exception of distutils (I removed traces of OS9 support code in 3.2 and those got restored when Tarek replaced distutils by the version from the 3.1 branch, I don't remember if I redid my OS9-removal work).

As I wrote earlier the path manipulation code (macpath.join, macpath.split etc.) are still useful because OS9-style paths are still used in some parts of OSX.

[ned-deily]

Patches in progress.

[merwok]

I agree with Ronald’s proposal.

(Re “I don't remember if I redid my OS9-removal work”: Looks like you didn’t.)

[chortos]

I fully agree with Ronald’s proposal. And for a start, here is a trivial patch that fixes macpath.join('', ...) [at the moment it just returns its last argument]. By the way, this fix is probably eligible for inclusion in Python 2.7 too.

I also have ‘implement macpath.relpath’ in my to-do list, although in practice this means nothing.

[merwok]

Thanks for the patch. Is the function already tested or does someone need to add a test?

[chortos]

It is already tested but here is a new version of the patch that expands the existing test to cover the situation that was broken (plus another one).

[jesstess]

Thanks for writing up this issue, ned.deily, and thanks for providing a patch, chortos.

I couldn't find documentation clearly specifying what the correct behavior of macpath.join should be (i.e. what are the exact rules for leading and trailing colons), but comparing the old and updated behavior against the new tests, this patch does make the function's behavior more consistent.

• The patch passes make patchcheck.
• The full test suite passes with this patch.

=> commit review

[ned-deily]

The patch appears fine but it really doesn't have anything to do with the gist of this issue. The problem remains that much of macpath is fundamentally broken. In the intervening years since the issue was opened, I contend that any need for OS 9 style paths (aka HFS paths) has diminished even further so I would still support starting in Python 3.5 the deprecation and removal process for this module. But, as long as it remains, to address the current problems at a minimum a patch would be needed to remove or raise exceptions for all of the macpath functions that make file system calls and to update the documentation.

[python-dev]

New changeset 2ae2ca9d2b66 by Serhiy Storchaka in branch '2.7':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/2ae2ca9d2b66

New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/54987723de99

New changeset e29866cb6b98 by Serhiy Storchaka in branch 'default':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/e29866cb6b98

[serhiy-storchaka]

Committed macpath_join_fix_with_test.patch with some additional tests. Thank you for your contribution Oleg.

Needed a patch for deprecating all module or some functions.

[ronaldoussoren]

I think it is by now safe to remove macpath, AFAIK there is no real use-case anymore for having classic MacOS9 paths on any recentish version of OSX.]

I'm setting the version to 3.6 because it is too late to do this for Python 3.5, but it can be done for 3.6.

[serhiy-storchaka]

In third-party code the path manipulating functions can be used in unexpected manner than doesn't relate to paths on MacOS. Also "import macpath" can be left even with the use of macpath was eliminated or are dead code. There is a chance to break third-party code when just remove macpath without deprecation period.

[ned-deily]

It's a bit late in the 3.6 cycle to be removing macpath, but it's not too late to mark it in 3.6 as deprecated and to be removed in 3.7. If someone wants to write two patches, one for 3.6 to add a deprecation warning to the code and to the docs, the other for 3.7 to actually remove all traces of macpath, that would be great!

[yan12125]

Here are my tries. There are more MacOS-specific module names in Tools/freeze/freeze.py. They are unrelated to macpath so I leave them to the next bug.

[yan12125]

Oops forgot to commit deleted files in the 3.7 patch.

[Mariatta]

I'm just reading PEP-4 now for procedure of declaring a module deprecated.
Should PEP-4 page be updated with the proposal / info for macpath deprecation?

https://www.python.org/dev/peps/pep-0004/

Thanks.

[DimitrisJim]

Bump, any update on what to do with this issue?

[vstinner]

Bump, any update on what to do with this issue?

I updated and completed Chi Hsuan Yen's 3.6-deprecate-macpath.patch to create a pull request. I suggest to start by deprecating macpath in 3.7, and remove it from Python 3.8.

This module is here for years. macOS 9 is now dead, all macOS users migrated to OS X nowadays. I don't think that there is an urgency to *remove* macpath.

The title says "obsolete macpath module dangerously broken": is "dangerous" still up to date after the following fix?

New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/54987723de99

If yes, we may also warn users in the macpath documentation that the module is dangerous, especially in the documentation of Python < 3.7.

[vstinner]

New changeset 89a1c93 by Victor Stinner in branch 'master':
bpo-9850: Deprecate the macpath module (bpo-1540)
89a1c93

[vstinner]

Bump, any update on what to do with this issue?

I just deprecated the macpath module in Python 3.7. It will be removed from Python 3.8.

[serhiy-storchaka]

You can use test.support.import_module(deprecated=True) for importing deprecated module in tests.

[serhiy-storchaka]

And please add a Misc/NEWS entry.

[DimitrisJim]

Great! Also, as Mariatta mentioned in a previous message, shouldn't an entry in PEP-4 be made?

[vstinner]

Serhiy Storchaka: "And please add a Misc/NEWS entry."

I created #1590

Serhiy: "You can use test.support.import_module(deprecated=True) for importing deprecated module in tests."

I looked at this function, but I don't want to skip the test if the module cannot be imported.

Jim Fasarakis-Hilliard: "Great! Also, as Mariatta mentioned in a previous message, shouldn't an entry in PEP-4 be made?"

I created python/peps#257

I didn't know the PEP-4.

[vstinner]

New changeset d812eb7 by Victor Stinner in branch 'master':
bpo-9850: Document macpath deprecation in Misc/NEWS (bpo-1590)
d812eb7

[serhiy-storchaka]

I looked at this function, but I don't want to skip the test if the module cannot be imported.

You can pass required_on=('',).

[vstinner]

> I looked at this function, but I don't want to skip the test if the module cannot be imported.

You can pass required_on=('',).

('',) looks mysterious to me. I prefer my current code.

If you consider that an helper is required, I would prefer a more explicit option on import_module(), like always_raise=True or something like that?

[serhiy-storchaka]

No, the current code LGTM.

[vstinner]

Ok, I don't see any open task anymore, so I close this issue.

The macpath may or may not be removed in Python 3.8, so not before 2018-01-29. IMHO it's ok to remove the module, macpath was used for MacOS 9, but all macOS users are now running macOS X. Otherwise, you can simply use an older Python version (like the future Python 3.7 ;-)). We can discuss the effective macpath removal later, when master will become Python 3.8.

[vstinner]

I created bpo-35471: "Remove macpath module".