Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zlib.error: Error -2 while flushing: inconsistent stream state #74203

Closed
JeremyHeiner mannequin opened this issue Apr 7, 2017 · 6 comments
Closed

zlib.error: Error -2 while flushing: inconsistent stream state #74203

JeremyHeiner mannequin opened this issue Apr 7, 2017 · 6 comments
Assignees
Labels
3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@JeremyHeiner
Copy link
Mannequin

JeremyHeiner mannequin commented Apr 7, 2017

BPO 30017
Nosy @takluyver, @vadmium, @serhiy-storchaka, @zhangyangyu, @gotyaoi
PRs
  • bpo-30017: Allowed calling the close() method of the zip entry writer object #1041
  • [3.6] bpo-30017: Allowed calling the close() method of the zip entry writer object (GH-1041) #1092
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2017-04-16.09:05:54.988>
    created_at = <Date 2017-04-07.14:21:06.591>
    labels = ['3.7', 'type-bug', 'library']
    title = 'zlib.error: Error -2 while flushing: inconsistent stream state'
    updated_at = <Date 2017-04-16.09:05:54.987>
    user = 'https://bugs.python.org/JeremyHeiner'

    bugs.python.org fields:

    activity = <Date 2017-04-16.09:05:54.987>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2017-04-16.09:05:54.988>
    closer = 'serhiy.storchaka'
    components = ['Library (Lib)']
    creation = <Date 2017-04-07.14:21:06.591>
    creator = 'Jeremy Heiner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 30017
    keywords = []
    message_count = 6.0
    messages = ['291275', '291309', '291310', '291319', '291550', '291747']
    nosy_count = 6.0
    nosy_names = ['takluyver', 'martin.panter', 'serhiy.storchaka', 'xiang.zhang', 'Ellison Marks', 'Jeremy Heiner']
    pr_nums = ['1041', '1092']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue30017'
    versions = ['Python 3.6', 'Python 3.7']

    @JeremyHeiner
    Copy link
    Mannequin Author

    JeremyHeiner mannequin commented Apr 7, 2017

    I had some statements inside a with statement to write data to an entry in a
    ZipFile. It worked great. I added a second with statement containing almost
    exactly the same statements. That still worked great.

    I refactored those common statements into a function and called that function
    from the two with statements... and got an exception:

    zlib.error: Error -2 while flushing: inconsistent stream state

    I can't figure out why it matters whether the writing happens in the with or
    in the function called by the with, but here's a trimmed-down version of the
    code that demonstrates the problem:

    --------------------------------------------------------------------------------
    #!/usr/bin/env python

    import io, pprint, zipfile
    from zipfile import ZIP_DEFLATED
    
    def printLiteral( data, out ) :
            encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
            pprint.pprint( data, stream=encoder )
    
    data = { 'not' : 'much', 'just' : 'some K \N{RIGHTWARDS WHITE ARROW} V pairs' }
    
    with zipfile.ZipFile( 'zzz.zip', mode='w', compression=ZIP_DEFLATED ) as myzip :
    
        with myzip.open( 'this one works', 'w' ) as out :
            encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
            pprint.pprint( data, stream=encoder )
    
        with myzip.open( 'this one fails', 'w' ) as out :
            printLiteral( data, out )
    
            print( 'printed but entry still open' )
        print( 'entry has been closed but not file' )
    print( 'zip file has been closed' )

    And here's the output on my Arch Linux 64bit with package python 3.6.0-2...
    A co-worker sees the same behavior on MacOS 10.11.6 Python 3.6.1 :

    --------------------------------------------------------------------------------

    printed but entry still open
    Traceback (most recent call last):
      File "zzz.py", line 21, in <module>
        print( 'printed but entry still open' )
      File "/usr/lib/python3.6/zipfile.py", line 995, in close
        buf = self._compressor.flush()
    zlib.error: Error -2 while flushing: inconsistent stream state

    I tried debugging this in PyDev but got lost. Turning off the compression makes
    the exception go away.

    @JeremyHeiner JeremyHeiner mannequin added the type-bug An unexpected behavior, bug, or error label Apr 7, 2017
    @JeremyHeiner JeremyHeiner mannequin changed the title zlib zlib.error: Error -2 while flushing: inconsistent stream state Apr 7, 2017
    @gotyaoi
    Copy link
    Mannequin

    gotyaoi mannequin commented Apr 8, 2017

    At a guess, when encoder goes out of scope, it closes the underlying file object. Then, on exiting the with block, the zipfile tries to take some action with the closed file and errors out?

    @vadmium
    Copy link
    Member

    vadmium commented Apr 8, 2017

    It looks like the zip entry writer object may not expect its “close” method to be called multiple times. Other file objects tend to allow this with no ill effect. Adding Serhiy and Thomas who implemented the writer object in bpo-26039.

    The first time it is called will be when the context manager (“with” statement) exits.

    The second time will be when the TextIOWrapper is garbage-collected. This is documented behaviour <https://docs.python.org/3.6/library/io.html#io.IOBase.\_\_del__\>, but I think it is a design mistake of the IOBase hierarchy; see bpo-19829.

    A reasonable workaround would be to call encoder.detach() after you have finished writing the zip entry.

    @serhiy-storchaka
    Copy link
    Member

    I agree, that close() should be an idempotent operation. Proposed patch fixes this.

    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir 3.7 (EOL) end of life labels Apr 8, 2017
    @serhiy-storchaka serhiy-storchaka self-assigned this Apr 8, 2017
    @serhiy-storchaka
    Copy link
    Member

    New changeset 4c0d9ea by Serhiy Storchaka in branch 'master':
    bpo-30017: Allowed calling the close() method of the zip entry writer object (bpo-1041)
    4c0d9ea

    @serhiy-storchaka
    Copy link
    Member

    New changeset 8e5b52a by Serhiy Storchaka in branch '3.6':
    bpo-30017: Allowed calling the close() method of the zip entry writer object (bpo-1041) (bpo-1092)
    8e5b52a

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants