Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

obsolete macpath module dangerously broken and should be removed #54059

Closed
ned-deily opened this issue Sep 14, 2010 · 33 comments
Closed

obsolete macpath module dangerously broken and should be removed #54059

ned-deily opened this issue Sep 14, 2010 · 33 comments

Comments

@ned-deily
Copy link
Member

@ned-deily ned-deily commented Sep 14, 2010

BPO 9850
Nosy @ronaldoussoren, @vstinner, @ned-deily, @merwok, @serhiy-storchaka, @yan12125, @DimitrisJim, @scottmacpherson, @Mariatta
PRs
  • #1540
  • #1590
  • Files
  • macpath_join_fix.patch: fix for macpath.join('', *comps)
  • macpath_join_fix_with_test.patch: fix for macpath.join('', *comps) with a test
  • 3.6-deprecate-macpath.patch
  • 3.7-remove-macpath.patch
  • 3.7-remove-macpath_ver2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-05-16.20:49:57.526>
    created_at = <Date 2010-09-14.05:14:10.511>
    labels = ['OS-mac', 'easy', '3.7']
    title = 'obsolete macpath module dangerously broken and should be removed'
    updated_at = <Date 2018-12-12.15:31:19.204>
    user = 'https://github.com/ned-deily'

    bugs.python.org fields:

    activity = <Date 2018-12-12.15:31:19.204>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-05-16.20:49:57.526>
    closer = 'vstinner'
    components = ['macOS']
    creation = <Date 2010-09-14.05:14:10.511>
    creator = 'ned.deily'
    dependencies = []
    files = ['22929', '22943', '44930', '44931', '44932']
    hgrepos = []
    issue_num = 9850
    keywords = ['patch', 'easy']
    message_count = 33.0
    messages = ['116367', '116387', '116388', '116389', '116517', '116973', '142310', '142365', '142397', '217238', '217901', '227687', '227689', '247128', '247242', '277844', '277892', '277893', '278102', '293443', '293458', '293692', '293693', '293694', '293695', '293696', '293697', '293698', '293700', '293704', '293706', '293785', '331701']
    nosy_count = 12.0
    nosy_names = ['ronaldoussoren', 'vstinner', 'ned.deily', 'eric.araujo', 'jesstess', 'chortos', 'python-dev', 'serhiy.storchaka', 'yan12125', 'Jim Fasarakis-Hilliard', 'macpherson.scott', 'Mariatta']
    pr_nums = ['1540', '1590']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue9850'
    versions = ['Python 3.7']

    @ned-deily ned-deily added the type-bug label Sep 14, 2010
    @ned-deily
    Copy link
    Member Author

    @ned-deily ned-deily commented Sep 14, 2010

    The macpath module in the standard library purports to supply "the Mac OS 9 (and earlier) implementation of the os.path module. It can be used to manipulate old-style Macintosh pathnames on Mac OS X (or any other platform). The following functions are available in this module: normcase(), normpath(), isabs(), join(), split(), isdir(), isfile(), walk(), exists(). For other functions available in os.path dummy counterparts are available."

    Old-style Mac pathnames are not further documented - in fact, the above quote is the entire external documentation for the module - but they are ones using colon separators, like ("MacHD:Users:nad:macpath_test:file"). These style path names were initially supported on Mac OS X by compatibility APIs for programs ported from classic Mac OS but those interfaces have long been deprecated in OS X and in most cases are not available in 64-bit execution mode.

    Even if one did have a need to use the obsolete old-style paths, the macpath module is currently practically unusable for anything other than simple character manipulations of the path. Nearly all of the functions that actually call OS routines are broken in one or more ways.

    Those that do make file system calls are calling APIs that are expecting normal posix-style ('/' separated paths) incorrectly with old-style (":) paths (ispath, isdir, lexists, etc) which means they only give correct results in the trivial case of unqualified file or directory names, i.e. those in the working directory.

    The islink() function confusingly is testing whether a path points to a Finder Alias file (not a symlink). Unfortunately, the Carbon API used for that test does not exist in a 64-bit version and all of the Python Carbon modules were removed in Python 3.

    $ arch -i386 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
    1
    $ arch -x86_64 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
    False
    $ python3.2 -c 'import macpath; print(macpath.islink("alias_to_file"))'
    False

    The underlying import errors are being masked by "try"'s:

    $ arch -i386 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
    $ arch -x86_64 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    AttributeError: 'module' object has no attribute 'ResolveAliasFile'
    $ python3.2 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named Carbon.File

    The realpath function is also broken in 2.7, calling Carbon.File.FSResolveAliasFile with a ":" separated path when it expects a "/" path, and is totally broken in 3.x (no Carbon modules).

    While macpath *could* be repaired by proper documentation, fixing the mishmash of path types internally, and supplying C wrappers and/or alternative 64-bit APIs, it does not seem worth the effort as these style paths are seldom encountered in modern code. Considering how broken the module currently is and given that it probably was simply overlooked in the Python 2 to 3 transition, I think a case could be made for immediate removal prior to Python 3.2 release and even for a 3.1.x maintenance release. Short of that, it should be cleared documented as deprecated in 3.2, 3.1, and 2.7 along with warnings about broken functionality along with added DeprecationWarnings to the module.

    I can write up patches to do that depending on what the consensus is.

    @ronaldoussoren
    Copy link
    Contributor

    @ronaldoussoren ronaldoussoren commented Sep 14, 2010

    I'm -0 on removing the module wholesale because parts of OSX still use MacOS9 style paths (in particular some Carbon APIs and AppleScript).

    I'm +1 on removing all functions that aren't path manipulation code, that is 'macpath.join' would stay while 'macpath.islink' would go, and fixing the documentation.

    And to be complete: I'm -1 on fixing macpath.islink and other functions that access a life filesystem because the macpath module is only meant to be used for that on MacOS9 and that platform is no longer supported. Path manipulation code is still useful.

    BTW. The module cannot be removed from 2.7 and 3.1 due to API stability requirements.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Sep 14, 2010

    The solution may be different depending on Python version. I propose to keep macpath in Python 2.7, just because it's too late to change such thing in Python2. But we may mark macpath as deprecated, eg. "macpath will be removed in Python 3.2" (see above). I suppose that the same treatement can be used on Python 3.1 (maybe with a different warning type/message).

    But for Python 3.2, I suggest to drop macpath. Said differently: drop Mac OS 9 support. If you really need Mac OS 9 support, continue to use Python 2.7. If I understood correctly, macpath is not used for os.path in Python 3.2, and it is broken (at least on 64 bits build).

    But well, I am not a Mac programmer, so I don't know if Mac OS 9 is still commonly installed or not.

    @ronaldoussoren
    Copy link
    Contributor

    @ronaldoussoren ronaldoussoren commented Sep 14, 2010

    MacOS9 is already unsupported, except for macpath all traces of OS9 support are gone with the possible exception of distutils (I removed traces of OS9 support code in 3.2 and those got restored when Tarek replaced distutils by the version from the 3.1 branch, I don't remember if I redid my OS9-removal work).

    As I wrote earlier the path manipulation code (macpath.join, macpath.split etc.) are still useful because OS9-style paths are still used in some parts of OSX.

    @ned-deily
    Copy link
    Member Author

    @ned-deily ned-deily commented Sep 16, 2010

    Patches in progress.

    @merwok
    Copy link
    Member

    @merwok merwok commented Sep 20, 2010

    I agree with Ronald’s proposal.

    (Re “I don't remember if I redid my OS9-removal work”: Looks like you didn’t.)

    @chortos
    Copy link
    Mannequin

    @chortos chortos mannequin commented Aug 18, 2011

    I fully agree with Ronald’s proposal. And for a start, here is a trivial patch that fixes macpath.join('', ...) [at the moment it just returns its last argument]. By the way, this fix is probably eligible for inclusion in Python 2.7 too.

    I also have ‘implement macpath.relpath’ in my to-do list, although in practice this means nothing.

    @merwok
    Copy link
    Member

    @merwok merwok commented Aug 18, 2011

    Thanks for the patch. Is the function already tested or does someone need to add a test?

    @chortos
    Copy link
    Mannequin

    @chortos chortos mannequin commented Aug 18, 2011

    It is already tested but here is a new version of the patch that expands the existing test to cover the situation that was broken (plus another one).

    @jesstess
    Copy link
    Member

    @jesstess jesstess commented Apr 27, 2014

    Thanks for writing up this issue, ned.deily, and thanks for providing a patch, chortos.

    I couldn't find documentation clearly specifying what the correct behavior of macpath.join should be (i.e. what are the exact rules for leading and trailing colons), but comparing the old and updated behavior against the new tests, this patch does make the function's behavior more consistent.

    • The patch passes make patchcheck.
    • The full test suite passes with this patch.

    => commit review

    @ned-deily
    Copy link
    Member Author

    @ned-deily ned-deily commented May 5, 2014

    The patch appears fine but it really doesn't have anything to do with the gist of this issue. The problem remains that much of macpath is fundamentally broken. In the intervening years since the issue was opened, I contend that any need for OS 9 style paths (aka HFS paths) has diminished even further so I would still support starting in Python 3.5 the deprecation and removal process for this module. But, as long as it remains, to address the current problems at a minimum a patch would be needed to remove or raise exceptions for all of the macpath functions that make file system calls and to update the documentation.

    @python-dev
    Copy link
    Mannequin

    @python-dev python-dev mannequin commented Sep 27, 2014

    New changeset 2ae2ca9d2b66 by Serhiy Storchaka in branch '2.7':
    Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
    https://hg.python.org/cpython/rev/2ae2ca9d2b66

    New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
    Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
    https://hg.python.org/cpython/rev/54987723de99

    New changeset e29866cb6b98 by Serhiy Storchaka in branch 'default':
    Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
    https://hg.python.org/cpython/rev/e29866cb6b98

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented Sep 27, 2014

    Committed macpath_join_fix_with_test.patch with some additional tests. Thank you for your contribution Oleg.

    Needed a patch for deprecating all module or some functions.

    @serhiy-storchaka serhiy-storchaka added type-feature and removed type-bug labels Sep 27, 2014
    @ronaldoussoren
    Copy link
    Contributor

    @ronaldoussoren ronaldoussoren commented Jul 22, 2015

    I think it is by now safe to remove macpath, AFAIK there is no real use-case anymore for having classic MacOS9 paths on any recentish version of OSX.]

    I'm setting the version to 3.6 because it is too late to do this for Python 3.5, but it can be done for 3.6.

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented Jul 24, 2015

    In third-party code the path manipulating functions can be used in unexpected manner than doesn't relate to paths on MacOS. Also "import macpath" can be left even with the use of macpath was eliminated or are dead code. There is a chance to break third-party code when just remove macpath without deprecation period.

    @ned-deily
    Copy link
    Member Author

    @ned-deily ned-deily commented Oct 2, 2016

    It's a bit late in the 3.6 cycle to be removing macpath, but it's not too late to mark it in 3.6 as deprecated and to be removed in 3.7. If someone wants to write two patches, one for 3.6 to add a deprecation warning to the code and to the docs, the other for 3.7 to actually remove all traces of macpath, that would be great!

    @ned-deily ned-deily added the 3.7 label Oct 2, 2016
    @ned-deily ned-deily removed the type-feature label Oct 2, 2016
    @yan12125
    Copy link
    Mannequin

    @yan12125 yan12125 mannequin commented Oct 2, 2016

    Here are my tries. There are more MacOS-specific module names in Tools/freeze/freeze.py. They are unrelated to macpath so I leave them to the next bug.

    @yan12125
    Copy link
    Mannequin

    @yan12125 yan12125 mannequin commented Oct 2, 2016

    Oops forgot to commit deleted files in the 3.7 patch.

    @Mariatta
    Copy link
    Sponsor Member

    @Mariatta Mariatta commented Oct 5, 2016

    I'm just reading PEP-4 now for procedure of declaring a module deprecated.
    Should PEP-4 page be updated with the proposal / info for macpath deprecation?

    https://www.python.org/dev/peps/pep-0004/

    Thanks.

    @DimitrisJim
    Copy link
    Mannequin

    @DimitrisJim DimitrisJim mannequin commented May 10, 2017

    Bump, any update on what to do with this issue?

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 10, 2017

    Bump, any update on what to do with this issue?

    I updated and completed Chi Hsuan Yen's 3.6-deprecate-macpath.patch to create a pull request. I suggest to start by deprecating macpath in 3.7, and remove it from Python 3.8.

    This module is here for years. macOS 9 is now dead, all macOS users migrated to OS X nowadays. I don't think that there is an urgency to *remove* macpath.

    The title says "obsolete macpath module dangerously broken": is "dangerous" still up to date after the following fix?

    New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
    Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
    https://hg.python.org/cpython/rev/54987723de99

    If yes, we may also warn users in the macpath documentation that the module is dangerous, especially in the documentation of Python < 3.7.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 15, 2017

    New changeset 89a1c93 by Victor Stinner in branch 'master':
    bpo-9850: Deprecate the macpath module (bpo-1540)
    89a1c93

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 15, 2017

    Bump, any update on what to do with this issue?

    I just deprecated the macpath module in Python 3.7. It will be removed from Python 3.8.

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented May 15, 2017

    You can use test.support.import_module(deprecated=True) for importing deprecated module in tests.

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented May 15, 2017

    And please add a Misc/NEWS entry.

    @DimitrisJim
    Copy link
    Mannequin

    @DimitrisJim DimitrisJim mannequin commented May 15, 2017

    Great! Also, as Mariatta mentioned in a previous message, shouldn't an entry in PEP-4 be made?

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 15, 2017

    Serhiy Storchaka: "And please add a Misc/NEWS entry."

    I created #1590

    Serhiy: "You can use test.support.import_module(deprecated=True) for importing deprecated module in tests."

    I looked at this function, but I don't want to skip the test if the module cannot be imported.

    Jim Fasarakis-Hilliard: "Great! Also, as Mariatta mentioned in a previous message, shouldn't an entry in PEP-4 be made?"

    I created python/peps#257

    I didn't know the PEP-4.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 15, 2017

    New changeset d812eb7 by Victor Stinner in branch 'master':
    bpo-9850: Document macpath deprecation in Misc/NEWS (bpo-1590)
    d812eb7

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented May 15, 2017

    I looked at this function, but I don't want to skip the test if the module cannot be imported.

    You can pass required_on=('',).

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 15, 2017

    > I looked at this function, but I don't want to skip the test if the module cannot be imported.

    You can pass required_on=('',).

    ('',) looks mysterious to me. I prefer my current code.

    If you consider that an helper is required, I would prefer a more explicit option on import_module(), like always_raise=True or something like that?

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented May 15, 2017

    No, the current code LGTM.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented May 16, 2017

    Ok, I don't see any open task anymore, so I close this issue.

    The macpath may or may not be removed in Python 3.8, so not before 2018-01-29. IMHO it's ok to remove the module, macpath was used for MacOS 9, but all macOS users are now running macOS X. Otherwise, you can simply use an older Python version (like the future Python 3.7 ;-)). We can discuss the effective macpath removal later, when master will become Python 3.8.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Dec 12, 2018

    I created bpo-35471: "Remove macpath module".

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    No branches or pull requests

    8 participants