Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update zipimport to support importlib.invalidate_caches() #58883

Closed
brettcannon opened this issue Apr 26, 2012 · 12 comments
Closed

Update zipimport to support importlib.invalidate_caches() #58883

brettcannon opened this issue Apr 26, 2012 · 12 comments
Labels
3.10 easy stdlib type-feature

Comments

@brettcannon
Copy link
Member

@brettcannon brettcannon commented Apr 26, 2012

BPO 14678
Nosy @brettcannon, @gpshead, @ncoghlan, @ericvsmith, @merwok, @ericsnowcurrently, @serhiy-storchaka, @crwilcox, @miss-islington, @desmondcheongzx
PRs
  • #24159
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-03-08.20:06:43.292>
    created_at = <Date 2012-04-26.17:41:49.735>
    labels = ['easy', 'type-feature', 'library', '3.10']
    title = 'Update zipimport to support importlib.invalidate_caches()'
    updated_at = <Date 2021-03-08.20:06:46.608>
    user = 'https://github.com/brettcannon'

    bugs.python.org fields:

    activity = <Date 2021-03-08.20:06:46.608>
    actor = 'brett.cannon'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-03-08.20:06:43.292>
    closer = 'brett.cannon'
    components = ['Library (Lib)']
    creation = <Date 2012-04-26.17:41:49.735>
    creator = 'brett.cannon'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 14678
    keywords = ['patch', 'easy']
    message_count = 12.0
    messages = ['159404', '159406', '159948', '159979', '160022', '197551', '325715', '361711', '362842', '362889', '362905', '388294']
    nosy_count = 11.0
    nosy_names = ['brett.cannon', 'gregory.p.smith', 'ncoghlan', 'eric.smith', 'eric.araujo', 'eric.snow', 'serhiy.storchaka', 'superluser', 'crwilcox', 'miss-islington', 'desmondcheongzx']
    pr_nums = ['24159']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue14678'
    versions = ['Python 3.10']

    @brettcannon
    Copy link
    Member Author

    @brettcannon brettcannon commented Apr 26, 2012

    zipimport's finders that get cached in sys.path_importer_cache should probably be updated to support importlib.invalidate_caches() (else the module should get re-implemented in pure Python in importlib and then be tossed).

    @brettcannon brettcannon added type-bug stdlib labels Apr 26, 2012
    @merwok
    Copy link
    Member

    @merwok merwok commented Apr 26, 2012

    zipimport in Python sounds good to me.

    @brettcannon
    Copy link
    Member Author

    @brettcannon brettcannon commented May 4, 2012

    I should mention I have a version from importers that is probably out-of-date: http://code.google.com/p/importers/source/browse/importers/zip.py

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented May 5, 2012

    Moving zipimporter to Python code is harder than it sounds: we don't want to break the ability to ship the standard library itself inside a zipfile.

    If you try to move zipimporter to pure Python, you could easily end up with a *very* ugly bootstrapping problem, on par with that already encountered when hacking on importlib._bootstrap.

    In fact, the path of least resistance here might actually be to implement zipimporter directly *in* importlib._bootstrap.

    @brettcannon
    Copy link
    Member Author

    @brettcannon brettcannon commented May 5, 2012

    The real problem becomes the issue of what zipfile depends on, which complicates bootstrapping. I mean things could go as far as to write a script that takes in anchor points in the stdlib and freezes all code they depend on, but that seems like potential overkill and executable bloat.

    But something needs to happen as zipimporter has major shortcomings thanks to to its re-implementation of zip handling (e.g. no ZIP64 support, etc.). Plus no one ever wants to touch that code.

    @gpshead
    Copy link
    Member

    @gpshead gpshead commented Sep 13, 2013

    Brett wrote a pure python zipimporter in http://bugs.python.org/issue17630 :)

    FWIW, the zipimport.c implementation (in 2.7) causes us serious pain when we've got the stdlib in a .zip file and need to update that while there are running python processes (which have cached the central directory of that .zip file but do not also keep a fd to the zip file open or even attempt to stat the file to see if it has changed [racy but it'd still be better than the nothing that is done now]).

    @serhiy-storchaka
    Copy link
    Member

    @serhiy-storchaka serhiy-storchaka commented Sep 19, 2018

    zipimport have been rewritten in pure Python (bpo-25711).

    @serhiy-storchaka serhiy-storchaka added 3.8 type-feature and removed type-bug labels Sep 19, 2018
    @crwilcox
    Copy link
    Mannequin

    @crwilcox crwilcox mannequin commented Feb 10, 2020

    What work remains to be done for this issue?

    @brettcannon
    Copy link
    Member Author

    @brettcannon brettcannon commented Feb 27, 2020

    @crwilcox Don't know what's left. That's probably part of the work that's left. ;) This issue is so old that it's possibly out-of-date since zipimport got rewritten in pure Python.

    @crwilcox
    Copy link
    Mannequin

    @crwilcox crwilcox mannequin commented Feb 28, 2020

    That is my thinking as well after rooting around a bit. I unfortunately am not knowledgable enough here to be 100% sure it is complete.

    @brettcannon
    Copy link
    Member Author

    @brettcannon brettcannon commented Feb 28, 2020

    It looks like it hasn't been done, else https://github.com/python/cpython/blob/master/Lib/zipimport.py would have some definition of a invalidate_caches() method. And specifically the finder lacks that method so it can be called by

    class PathFinder:
    """Meta path finder for sys.path and package __path__ attributes."""
    @classmethod
    def invalidate_caches(cls):
    """Call the invalidate_caches() method on all path entry finders
    stored in sys.path_importer_caches (where implemented)."""
    for name, finder in list(sys.path_importer_cache.items()):
    if finder is None:
    del sys.path_importer_cache[name]
    elif hasattr(finder, 'invalidate_caches'):
    finder.invalidate_caches()
    .

    @miss-islington
    Copy link
    Contributor

    @miss-islington miss-islington commented Mar 8, 2021

    New changeset 3abf6f0 by Desmond Cheong in branch 'master':
    bpo-14678: Update zipimport to support importlib.invalidate_caches() (GH-24159)
    3abf6f0

    @brettcannon brettcannon added 3.10 and removed 3.8 labels Mar 8, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 easy stdlib type-feature
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants