Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

tianlynn · 2017-08-10T04:48:25Z

BPO	31170
Nosy	@birkenfeld, @vstinner, @larryhastings, @benjaminp, @ned-deily, @serhiy-storchaka, @hartwork, @tianlynn
PRs	bpo-31170: Update libexpat from 2.2.3 to 2.2.4 #3315 [3.6] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3350 [2.7] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3351 [3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3352 [3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3353 [3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3354 bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug #3570 [2.7] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3745 [3.6] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3746 [3.5] bpo-31170: Fix inclusion of expat in Windows build projects. #3751 [3.4] bpo-31170: Fix inclusion of expat in Windows build projects #3785
Files	cpython_rebuild_expat_dir.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-04.15:52:09.415>
created_at = <Date 2017-08-10.04:48:25.430>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)'
updated_at = <Date 2019-05-10.18:04:35.827>
user = 'https://github.com/tianlynn'

bugs.python.org fields:

activity = <Date 2019-05-10.18:04:35.827>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-04.15:52:09.415>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-08-10.04:48:25.430>
creator = 'tianlynn'
dependencies = []
files = ['47118']
hgrepos = []
issue_num = 31170
keywords = ['patch']
message_count = 26.0
messages = ['300043', '300044', '300268', '300309', '300605', '301011', '301012', '301084', '301270', '301287', '301399', '301422', '301424', '302159', '302483', '302833', '302898', '302925', '302927', '302930', '302931', '302975', '303037', '307254', '309469', '309471']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'serhiy.storchaka', 'sping', 'tianlynn']
pr_nums = ['3315', '3350', '3351', '3352', '3353', '3354', '3570', '3745', '3746', '3751', '3785']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31170'
versions = ['Python 3.4', 'Python 3.5']

tianlynn · 2017-08-10T04:48:25Z

utf8_toUtf8(const ENCODING *UNUSED_P(enc),
            const char **fromP, const char *fromLim,
            char **toP, const char *toLim)
{
  char *to;
  const char *from;
  const char *fromLimInitial = fromLim;

/* Avoid copying partial characters. */
align_limit_to_full_utf8_characters(*fromP, &fromLim);

for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++)
*to = *from;
*fromP = from;
*toP = to;

  if (fromLim < fromLimInitial)
    return XML_CONVERT_INPUT_INCOMPLETE;
  else if ((to == toLim) && (from < fromLim))
    // <===== Bug is here. In case (to == toLim), it's possible that
    //        from is still pointing to partial character. For example,
    //        a character with 3 bytes (A, B, C) and form is pointing to C.
    //        It means only A and B is copied to output buffer. Next
    //        scanning will start with C which could be considered as invalid
    //        byte and got dropped. After this, only "AB" is kept in memory
    //        and thus it will lead to invalid continuation byte.
    return XML_CONVERT_OUTPUT_EXHAUSTED;
  else
    return XML_CONVERT_COMPLETED;
}

serhiy-storchaka · 2017-08-10T04:58:29Z

This is not our code. Please use corresponding bug tracker [1] if you have found a bug in Expat.

But I think there is not a bug here.

[1] https://github.com/libexpat/libexpat/issues

tianlynn · 2017-08-14T22:44:19Z

Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here)

hartwork · 2017-08-15T18:38:43Z

For the record, the upstream fix is commit libexpat/libexpat@74a7090 and it will be part of Expat 2.2.4.

hartwork · 2017-08-20T18:28:40Z

Expat 2.2.4 with a fix has been released now: https://github.com/libexpat/libexpat/releases

serhiy-storchaka · 2017-08-30T05:25:45Z

I confirm the regression (see bpo-31303 for reproducer). Victor, do you mind to update expat to 2.2.4?

This issue can be classified as a security issue, since a regression was added in security update.

serhiy-storchaka · 2017-08-30T05:41:03Z

Corresponding Expat issue: libexpat/libexpat#115.

benjaminp · 2017-09-01T04:58:20Z

We can put expat 2.2.4 in 2.7.14 final.

vstinner · 2017-09-04T21:42:52Z

I produced attached PR 3315 using attached cpython_rebuild_expat_dir.sh + revert Modules/expat/expat_external.h change to keep #include "pyexpatns.h".

vstinner · 2017-09-04T23:58:11Z

New changeset 759e30e by Victor Stinner in branch 'master':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315)
759e30e

vstinner · 2017-09-05T23:30:24Z

New changeset e5f2f80 by Victor Stinner in branch '3.6':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315) (bpo-3350)
e5f2f80

ned-deily · 2017-09-06T00:57:39Z

New changeset 297516e by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3352)
297516e

benjaminp · 2017-09-06T01:07:08Z

New changeset c00d508 by Benjamin Peterson (Victor Stinner) in branch '2.7':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315)
c00d508

vstinner · 2017-09-14T09:30:15Z

I wrote an non-regression test for the Python master branch using the test case attached to libexpat/libexpat#115: PR 3570.

ned-deily · 2017-09-18T19:43:08Z

I believe all the branches except 3.5 and 3.4 have been updated so adjusting the Versions field accordingly. All yours, Larry!

larryhastings · 2017-09-24T08:04:56Z

New changeset 86a713c by larryhastings (Victor Stinner) in branch '3.4':
[3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3353)
86a713c

larryhastings · 2017-09-25T00:58:36Z

New changeset f2492bb by larryhastings (Victor Stinner) in branch '3.5':
[3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3354)
f2492bb

vstinner · 2017-09-25T08:27:37Z

New changeset e6d9fcb by Victor Stinner in branch 'master':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570)
e6d9fcb

vstinner · 2017-09-25T08:43:59Z

New changeset 5f5da72 by Victor Stinner in branch '2.7':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570) (bpo-3745)
5f5da72

vstinner · 2017-09-25T09:00:07Z

New changeset ad051cb by Victor Stinner in branch '3.6':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570) (bpo-3746)
ad051cb

vstinner · 2017-09-25T09:02:31Z

libexpat has been upgraded to version 2.2.4 in Python 2.7, 3.3, 3.4, 3.5, 3.6 and master.

I added an unit test to Python 2.7, 3.6 and master. I'm not sure about backporting the new unit test to Python 3.3, 3.4 and 3.5.

I close the issue.

vstinner · 2017-09-25T19:48:48Z

Oh, it seems like Steve Dower found an issue on Windows: PR 3751.

I reopen the issue.

larryhastings · 2017-09-26T13:00:30Z

New changeset 44c1b62 by larryhastings (Steve Dower) in branch '3.5':
[3.5] bpo-31170: Fix inclusion of expat in Windows build projects. (bpo-3751)
44c1b62

larryhastings · 2017-11-29T18:50:45Z

New changeset 8b11e8d by larryhastings (Victor Stinner) in branch '3.4':
[3.4] bpo-31170: Fix inclusion of expat in Windows build projects (bpo-3785)
8b11e8d

serhiy-storchaka · 2018-01-04T14:45:35Z

Can this issue be closed now?

vstinner · 2018-01-04T15:52:09Z

Ah yes, it can be closed. I was waiting 3.4 and 3.5 fixes to be fixed, which is now the case.

tianlynn mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Aug 10, 2017

serhiy-storchaka closed this as completed Aug 10, 2017

tianlynn mannequin reopened this Aug 14, 2017

serhiy-storchaka added the release-blocker label Aug 30, 2017

vstinner changed the title ~~expat: utf8_toUtf8 cannot properly handle exhausting buffer~~ Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) Sep 4, 2017

ned-deily removed the 3.7 (EOL) end of life label Sep 18, 2017

vstinner closed this as completed Sep 25, 2017

vstinner reopened this Sep 25, 2017

vstinner closed this as completed Jan 4, 2018

ezio-melotti transferred this issue from another repository Apr 10, 2022

Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

tianlynn mannequin commented Aug 10, 2017

tianlynn mannequin commented Aug 10, 2017

serhiy-storchaka commented Aug 10, 2017

tianlynn mannequin commented Aug 14, 2017

hartwork mannequin commented Aug 15, 2017

hartwork mannequin commented Aug 20, 2017

serhiy-storchaka commented Aug 30, 2017

serhiy-storchaka commented Aug 30, 2017

benjaminp commented Sep 1, 2017

vstinner commented Sep 4, 2017

vstinner commented Sep 4, 2017

vstinner commented Sep 5, 2017

ned-deily commented Sep 6, 2017

benjaminp commented Sep 6, 2017

vstinner commented Sep 14, 2017

ned-deily commented Sep 18, 2017

larryhastings commented Sep 24, 2017

larryhastings commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

larryhastings commented Sep 26, 2017

larryhastings commented Nov 29, 2017

serhiy-storchaka commented Jan 4, 2018

vstinner commented Jan 4, 2018

Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

Comments

tianlynn mannequin commented Aug 10, 2017

tianlynn mannequin commented Aug 10, 2017

serhiy-storchaka commented Aug 10, 2017

tianlynn mannequin commented Aug 14, 2017

hartwork mannequin commented Aug 15, 2017

hartwork mannequin commented Aug 20, 2017

serhiy-storchaka commented Aug 30, 2017

serhiy-storchaka commented Aug 30, 2017

benjaminp commented Sep 1, 2017

vstinner commented Sep 4, 2017

vstinner commented Sep 4, 2017

vstinner commented Sep 5, 2017

ned-deily commented Sep 6, 2017

benjaminp commented Sep 6, 2017

vstinner commented Sep 14, 2017

ned-deily commented Sep 18, 2017

larryhastings commented Sep 24, 2017

larryhastings commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

vstinner commented Sep 25, 2017

larryhastings commented Sep 26, 2017

larryhastings commented Nov 29, 2017

serhiy-storchaka commented Jan 4, 2018

vstinner commented Jan 4, 2018