Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

importlib.invalidate_caches() doesn't clear all caches #77350

Closed
gvanrossum opened this issue Mar 28, 2018 · 9 comments
Closed

importlib.invalidate_caches() doesn't clear all caches #77350

gvanrossum opened this issue Mar 28, 2018 · 9 comments
Assignees

Comments

@gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Mar 28, 2018

BPO 33169
Nosy @gvanrossum, @brettcannon, @ncoghlan, @ned-deily, @ericsnowcurrently
PRs
  • #6402
  • #6403
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/brettcannon'
    closed_at = <Date 2018-04-07.00:02:53.795>
    created_at = <Date 2018-03-28.14:30:18.602>
    labels = ['3.7', '3.8']
    title = "importlib.invalidate_caches() doesn't clear all caches"
    updated_at = <Date 2018-04-07.00:02:53.710>
    user = 'https://github.com/gvanrossum'

    bugs.python.org fields:

    activity = <Date 2018-04-07.00:02:53.710>
    actor = 'brett.cannon'
    assignee = 'brett.cannon'
    closed = True
    closed_date = <Date 2018-04-07.00:02:53.795>
    closer = 'brett.cannon'
    components = []
    creation = <Date 2018-03-28.14:30:18.602>
    creator = 'gvanrossum'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 33169
    keywords = ['patch']
    message_count = 9.0
    messages = ['314595', '314672', '314677', '314684', '314685', '314745', '315042', '315044', '315045']
    nosy_count = 5.0
    nosy_names = ['gvanrossum', 'brett.cannon', 'ncoghlan', 'ned.deily', 'eric.snow']
    pr_nums = ['6402', '6403']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue33169'
    versions = ['Python 3.7', 'Python 3.8']

    @gvanrossum
    Copy link
    Member Author

    @gvanrossum gvanrossum commented Mar 28, 2018

    See python/mypy#4811. To summarize, importlib.invalidate_caches() doesn't clear the negative cache in sys.path_importer_cache.

    Could be related to https://bugs.python.org/issue30891?

    @brettcannon
    Copy link
    Member

    @brettcannon brettcannon commented Mar 29, 2018

    Correct because as documented it calls invalidate_caches() on meta-path finders and the one that handles sys.path calls invalidate_caches() on the finders found in sys.path_importer_cache() (as documented by https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder.invalidate_caches). Since None isn't a finder it doesn't do anything there as there is no method to call. So the "full of lies" comment by the OP on the mypy repo is over-the-top/incorrect.

    Now tweaking

    @classmethod
    def invalidate_caches(cls):
    """Call the invalidate_caches() method on all path entry finders
    stored in sys.path_importer_caches (where implemented)."""
    for finder in sys.path_importer_cache.values():
    if hasattr(finder, 'invalidate_caches'):
    finder.invalidate_caches()
    to delete entries in sys.path_importer_cache() that are set to None is straight-forward, but I don't think we can backport since it would be going against what the documentation says it does.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Mar 30, 2018

    I agree that we can't change 3.6 at this late stage, but the change is borderline enough on the "new feature"/"bug fix" scale that I think it's worth asking Ned if he's comfortable with us changing it for 3.7.0b4.

    Ned: the change we'd like to make is to update PathFinder.invalidate_caches to delete all entries in sys.path_importer_cache that have a value of None.

    @ned-deily
    Copy link
    Member

    @ned-deily ned-deily commented Mar 30, 2018

    The current behavior is causing problems for at least some use cases so it does seems to be closer to a bug than a feature. If there is agreement that it is desirable to make the change, I think it's better to do it now in 3.7 than later.

    @gvanrossum
    Copy link
    Member Author

    @gvanrossum gvanrossum commented Mar 30, 2018

    SGTM.

    On Fri, Mar 30, 2018, 06:41 Ned Deily <report@bugs.python.org> wrote:

    Ned Deily <nad@python.org> added the comment:

    The current behavior is causing problems for at least some use cases so it
    does seems to be closer to a bug than a feature. If there is agreement
    that it is desirable to make the change, I think it's better to do it now
    in 3.7 than later.

    ----------


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue33169\>


    @brettcannon
    Copy link
    Member

    @brettcannon brettcannon commented Mar 31, 2018

    OK, I will update importlib.machinery.PathFinder.invalidate_caches() to also drop None from sys.path_importer_cache (objects that are not None and do not define invalidate_caches() will be left alone just like they are today). I should hopefully be able to get to this before b4.

    @brettcannon brettcannon self-assigned this Mar 31, 2018
    @brettcannon
    Copy link
    Member

    @brettcannon brettcannon commented Apr 6, 2018

    New changeset 9e2be60 by Brett Cannon in branch 'master':
    bpo-33169: Remove values of None from sys.path_importer_cache when invalidating caches (GH-6402)
    9e2be60

    @brettcannon
    Copy link
    Member

    @brettcannon brettcannon commented Apr 7, 2018

    New changeset a09bb87 by Brett Cannon in branch '3.7':
    [3.7] bpo-33169: Remove values of None from sys.path_importer_cache when invalidating caches (GH-6402) (GH-6403)
    a09bb87

    @brettcannon
    Copy link
    Member

    @brettcannon brettcannon commented Apr 7, 2018

    All done in 3.7 and master.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants