zipfile: Seeking encrypted file breaks after seeking backwards

[danifus]

BPO	38334
Nosy	@serhiy-storchaka, @miss-islington, @danifus
PRs	bpo-38334: Fix seeking bug for encrypted zipfiles #16529 bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. #16937 [3.8] bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937) #16951 [3.7] bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937) #16952

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-27.10:19:02.322>
created_at = <Date 2019-10-01.08:17:22.027>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'zipfile: Seeking encrypted file breaks after seeking backwards'
updated_at = <Date 2019-10-29.00:13:04.390>
user = 'https://github.com/danifus'

bugs.python.org fields:

activity = <Date 2019-10-29.00:13:04.390>
actor = 'dhillier'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-27.10:19:02.322>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2019-10-01.08:17:22.027>
creator = 'dhillier'
dependencies = []
files = []
hgrepos = []
issue_num = 38334
keywords = ['patch']
message_count = 10.0
messages = ['353646', '355393', '355433', '355436', '355446', '355453', '355455', '355456', '355461', '355604']
nosy_count = 3.0
nosy_names = ['serhiy.storchaka', 'miss-islington', 'dhillier']
pr_nums = ['16529', '16937', '16951', '16952']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38334'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[danifus]

Seeking back beyond the decrypted / unzipped buffer doesn't reset the decrypter's crc key values. All data read after seeking back beyond the buffer is garbled.

[danifus]

Hi,

I have another patch I would like to contribute to the zipfile module but would like to request a review of this one to minimise conflicts with later patches. If anyone is able to review the patch, I would really appreciate it :)

Also, with regards to selecting Python versions for this issue, is there any rule of thumb I should be following?

Thanks,
Dan

[serhiy-storchaka]

PR 16937 is simpler. It does not change the decrypter.

[danifus]

Thanks for looking at the PR.

I got carried away refactoring the decrypter for a future scenario where there could be different decrypters (possibly using certificates too) :) Your PR is much simpler.

Would you also be able to take a look at some other PRs I've submitted for zipfile. They are both pretty small changes:

https://bugs.python.org/issue36993
https://bugs.python.org/issue37523

Thanks again,
Dan

[danifus]

I also think that the read_init method in my PR is a useful refactor as it locates all the state that needs to be (re)set when starting a read into the same location.

At the moment this state is set in 1) __init__ and 2) the seek method when seeking back beyond what is in the buffer.

[serhiy-storchaka]

New changeset 5c32af7 by Serhiy Storchaka in branch 'master':
bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)
5c32af7

[miss-islington]

New changeset 76fbdaa by Miss Skeleton (bot) in branch '3.8':
bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)
76fbdaa

[miss-islington]

New changeset ed2db31 by Miss Skeleton (bot) in branch '3.7':
bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)
ed2db31

[serhiy-storchaka]

I got carried away refactoring the decrypter for a future scenario where there could be different decrypters (possibly using certificates too) :)

The decrypter was implemented with a generator for performance. The performance of decrypting is not critical, but this got us 2x speed up without complicating the code. See bpo-10030. If the support of other decrypters be added, they will likely be implemented in C in any case.

I also think that the read_init method in my PR is a useful refactor as it locates all the state that needs to be (re)set when starting a read into the same location.

This is a different issue.

Would you also be able to take a look at some other PRs I've submitted for zipfile.

Thank you, they look good to me. I left just few comments.

As for the original issue, I have doubts that backward seeking in a compressed file is a good idea. But this feature was added, and it worked with encrypted files if seek in the range of the buffer, so not working out of the range of the buffer is a bug which we should fix.

[danifus]

Thanks for your help! Good point, I'll create a new change for the refactoring.