Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-32583: Fix possible crashing in builtin Unicode decoders #5325

Merged
merged 2 commits into from
Jan 31, 2018
Merged

bpo-32583: Fix possible crashing in builtin Unicode decoders #5325

merged 2 commits into from
Jan 31, 2018

Conversation

zhangyangyu
Copy link
Member

@zhangyangyu zhangyangyu commented Jan 25, 2018 •
edited by bedevere-bot
Loading

@zhangyangyu zhangyangyu added type-bug An unexpected behavior, bug, or error needs backport to 3.6 labels Jan 25, 2018
@zhangyangyu zhangyangyu changed the title bpo-32583: Fix possible crashing in decode error handler bpo-32583: Fix possible crashing in builtin Unicode decoders Jan 26, 2018
@zhangyangyu
bpo-32583: Fix possible crashing in builtin Unicode decoders
1c7c83a 
When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash.
@zhangyangyu zhangyangyu merged commit 2c7fd46 into python:master Jan 31, 2018
@bedevere-bot
Copy link

@zhangyangyu: Please replace # with GH- in the commit message next time. Thanks!

@miss-islington
Copy link
Contributor

Thanks @zhangyangyu for the PR 🌮🎉.. I'm working now to backport this PR to: 3.6.
🐍🍒⛏🤖

@zhangyangyu zhangyangyu deleted the issue32583 branch January 31, 2018 12:48
@miss-islington
Copy link
Contributor

Sorry, @zhangyangyu, I could not cleanly backport this to 3.6 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 2c7fd46e11333ef5e5cce34212f7d087694f3658 3.6

zhangyangyu added a commit to zhangyangyu/cpython that referenced this pull request Jan 31, 2018
@zhangyangyu
[3.6] bpo-32583: Fix possible crashing in builtin Unicode decoders (p…
61f7c0a 
…ythonGH-5325)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash..
(cherry picked from commit 2c7fd46)
@bedevere-bot
Copy link

GH-5459 is a backport of this pull request to the 3.6 branch.

zhangyangyu added a commit that referenced this pull request Jan 31, 2018
@zhangyangyu
[3.6] bpo-32583: Fix possible crashing in builtin Unicode decoders (G…
ea94fce 
…H-5325) (#5459)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash..
(cherry picked from commit 2c7fd46)
sibiryakov
sibiryakov reviewed Jan 31, 2018
View reviewed changes
Objects/unicodeobject.c
}
new_inptr = *input + newpos;
if (*inend - new_inptr > remain) {
/* We don't know the decoding algorithm here so we make the worst
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error handler registered with codecs.register_error can produce any string as an output. This output will be inserted in the buffer you allocate. So I propose to expect that decoded output could be bigger and either check that and reallocate the buffer or raise a MemoryError exception with proper description.

Copy link
Member Author

@zhangyangyu zhangyangyu Jan 31, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review @sibiryakov . But the situation you mention has already been handled. See the previous condition replen - 1, if the handler returns any string longer than one, need_to_grow will be set and we will allocate more memory.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great @zhangyangyu! Thank you!

ned-deily pushed a commit to ned-deily/cpython that referenced this pull request Jan 31, 2018
@zhangyangyu @ned-deily
bpo-32583: Fix possible crashing in builtin Unicode decoders (python#…
86fdad0 
…5325)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sibiryakov sibiryakov sibiryakov left review comments

@vstinner vstinner Awaiting requested review from vstinner

@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Assignees
No one assigned
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zhangyangyu @bedevere-bot @miss-islington @sibiryakov @the-knights-who-say-ni