Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assert statements missed when loaded by zipimporter #72318

Closed
jaraco opened this issue Sep 13, 2016 · 10 comments
Closed

assert statements missed when loaded by zipimporter #72318

jaraco opened this issue Sep 13, 2016 · 10 comments
Labels
3.7 only security fixes release-blocker stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@jaraco
Copy link
Member

jaraco commented Sep 13, 2016

BPO 28131
Nosy @Yhg1s, @brettcannon, @gpshead, @jaraco, @larryhastings, @benjaminp, @ned-deily, @berkerpeksag, @zooba, @zhangyangyu
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
  • Files
  • issue28131.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2016-09-14.05:14:50.275>
    created_at = <Date 2016-09-13.16:19:15.769>
    labels = ['3.7', 'type-bug', 'library', 'release-blocker']
    title = 'assert statements missed when loaded by zipimporter'
    updated_at = <Date 2017-03-31.16:36:13.360>
    user = 'https://github.com/jaraco'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:13.360>
    actor = 'dstufft'
    assignee = 'none'
    closed = True
    closed_date = <Date 2016-09-14.05:14:50.275>
    closer = 'berker.peksag'
    components = ['Library (Lib)']
    creation = <Date 2016-09-13.16:19:15.769>
    creator = 'jaraco'
    dependencies = []
    files = ['44647']
    hgrepos = []
    issue_num = 28131
    keywords = ['patch']
    message_count = 10.0
    messages = ['276296', '276311', '276359', '276360', '276369', '276371', '276372', '276376', '276377', '276378']
    nosy_count = 11.0
    nosy_names = ['twouters', 'brett.cannon', 'gregory.p.smith', 'jaraco', 'larry', 'benjamin.peterson', 'ned.deily', 'python-dev', 'berker.peksag', 'steve.dower', 'xiang.zhang']
    pr_nums = ['552']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue28131'
    versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

    @jaraco
    Copy link
    Member Author

    jaraco commented Sep 13, 2016

    Grabbing the recently released Python 3.6.0b1, I tried running one of my test suites, but found that some assertions were failing to assert when the package was loaded as a zip file (such as with pytest-runner installed dependencies). I distilled the issue to this:

    $ cat > mod.py
    def test(val):
            assert(val)
            print(val)
    $ zip mod.zip mod.py
    updating: mod.py (deflated 20%)
    $ rm mod.py
    $ python
    Python 3.6.0b1 (v3.6.0b1:5b0ca4ed5e2f, Sep 12 2016, 09:24:46) 
    [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.path.append('mod.zip')
    >>> import mod
    >>> mod.test(False)
    False
    >>> mod.__loader__
    <zipimporter object "mod.zip">
    >>> sys.flags.optimize
    0

    I would have expected the call to mod.test to have raised an AssertionError, and on Python 3.5 it does.

    I searched the what's new and didn't see anything advertising this change, so I suspect it's an unintentional regression.

    I'm including Brett for his familiarity with importlib, but welcome re-assignment.

    If I can do more to help, let me know.

    @jaraco jaraco added 3.7 only security fixes release-blocker stdlib Python modules in the Lib dir labels Sep 13, 2016
    @brettcannon
    Copy link
    Member

    I've added Greg and Thomas in case they have any ideas as they have looked at zipimport more recently than I have.

    @gpshead
    Copy link
    Member

    gpshead commented Sep 13, 2016

    This shouldn't be happening and makes no sense. It looks like the assert statement was removed at import code compilation time given the pdb trace with it from a zip file vs with it outside of a zip file:

    >>> pdb.run('mod.test(False)')
    > <string>(1)<module>()
    (Pdb) n
    False
    --Return--
    > <string>(1)<module>()->None
    (Pdb) n
    >>> pdb.run('mod.test(False)')
    > <string>(1)<module>()->None
    (Pdb) s
    --Call--
    > oss/cpython/build/mod.zip/mod.py(1)test()
    -> def test(val):
    (Pdb) s
    > oss/cpython/build/mod.zip/mod.py(3)test()
    -> print(val)
    (Pdb) s
    False
    --Return--
    > oss/cpython/build/mod.zip/mod.py(3)test()->None
    -> print(val)
    (Pdb) s
    --Return--
    > <string>(1)<module>()->None
    (Pdb) s
    > oss/cpython/default/Lib/bdb.py(435)run()
    -> self.quitting = True
    (Pdb) s
    >>> 

    vs no zip file:

    >>> pdb.run('mod.test(False)')
    > <string>(1)<module>()
    (Pdb) s
    --Call--
    > oss/cpython/build/mod.py(1)test()
    -> def test(val):
    (Pdb) s
    > oss/cpython/build/mod.py(2)test()
    -> assert(val)
    (Pdb) s
    AssertionError
    > oss/cpython/build/mod.py(2)test()
    -> assert(val)
    (Pdb) s
    --Return--
    > oss/cpython/build/mod.py(2)test()->None
    -> assert(val)
    (Pdb) s
    AssertionError
    > <string>(1)<module>()
    (Pdb) s
    --Return--
    > <string>(1)<module>()->None
    (Pdb) s
    AssertionError
    > oss/cpython/default/Lib/bdb.py(431)run()
    -> exec(cmd, globals, locals)
    (Pdb) s
    > oss/cpython/default/Lib/bdb.py(432)run()
    -> except BdbQuit:
    (Pdb) s
    > oss/cpython/default/Lib/bdb.py(435)run()
    -> self.quitting = True
    (Pdb) s
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "oss/cpython/default/Lib/pdb.py", line 1568, in run
        Pdb().run(statement, globals, locals)
      File "oss/cpython/default/Lib/bdb.py", line 431, in run
        exec(cmd, globals, locals)
      File "<string>", line 1, in <module>
      File "oss/cpython/build/mod.py", line 2, in test
        assert(val)
    AssertionError

    @gpshead
    Copy link
    Member

    gpshead commented Sep 13, 2016

    from the zip:

    >>> dis.dis(mod.test)
      3           0 LOAD_GLOBAL              0 (print)
                  2 LOAD_FAST                0 (val)
                  4 CALL_FUNCTION            1
                  6 POP_TOP
                  8 LOAD_CONST               0 (None)
                 10 RETURN_VALUE
    
    from the filesystem:
    >>> dis.dis(mod.test)
      2           0 LOAD_FAST                0 (val)
                  2 POP_JUMP_IF_TRUE         8
                  4 LOAD_GLOBAL              0 (AssertionError)
                  6 RAISE_VARARGS            1

    3 >> 8 LOAD_GLOBAL 1 (print)
    10 LOAD_FAST 0 (val)
    12 CALL_FUNCTION 1
    14 POP_TOP
    16 LOAD_CONST 0 (None)
    18 RETURN_VALUE

    @zhangyangyu
    Copy link
    Member

    This is introduced in 663a62bcf9c9. The compile optimize flag is opened in the change.

    @zooba
    Copy link
    Member

    zooba commented Sep 14, 2016

    Bah, that's meant to be a -1 in that change, not 1. I must have typo'd when moving from the default branch to put it in 3.5.

    Anyone's welcome to fix it, or I'll get it when I have time for CPython work.

    @berkerpeksag
    Copy link
    Member

    Here is a patch with a test case.

    @berkerpeksag berkerpeksag added the type-bug An unexpected behavior, bug, or error label Sep 14, 2016
    @benjaminp
    Copy link
    Contributor

    lgtm

    @1762cc99-3127-4a62-9baf-30c3d0f51ef7
    Copy link
    Mannequin

    python-dev mannequin commented Sep 14, 2016

    New changeset 7bec326972f5 by Berker Peksag in branch '3.5':
    Issue bpo-28131: Fix a regression in zipimport's compile_source()
    https://hg.python.org/cpython/rev/7bec326972f5

    New changeset 7a6c0c4e6072 by Berker Peksag in branch '3.6':
    Issue bpo-28131: Merge from 3.5
    https://hg.python.org/cpython/rev/7a6c0c4e6072

    New changeset 873760b02024 by Berker Peksag in branch 'default':
    Issue bpo-28131: Merge from 3.6
    https://hg.python.org/cpython/rev/873760b02024

    @berkerpeksag
    Copy link
    Member

    Thanks for the report and for the reviews!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 only security fixes release-blocker stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants