Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)

[tianlynn]

BPO	31170
Nosy	@birkenfeld, @vstinner, @larryhastings, @benjaminp, @ned-deily, @serhiy-storchaka, @hartwork, @tianlynn
PRs	bpo-31170: Update libexpat from 2.2.3 to 2.2.4 #3315 [3.6] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3350 [2.7] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3351 [3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3352 [3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3353 [3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3354 bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug #3570 [2.7] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3745 [3.6] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3746 [3.5] bpo-31170: Fix inclusion of expat in Windows build projects. #3751 [3.4] bpo-31170: Fix inclusion of expat in Windows build projects #3785
Files	cpython_rebuild_expat_dir.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-04.15:52:09.415>
created_at = <Date 2017-08-10.04:48:25.430>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)'
updated_at = <Date 2019-05-10.18:04:35.827>
user = 'https://github.com/tianlynn'

bugs.python.org fields:

activity = <Date 2019-05-10.18:04:35.827>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-04.15:52:09.415>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-08-10.04:48:25.430>
creator = 'tianlynn'
dependencies = []
files = ['47118']
hgrepos = []
issue_num = 31170
keywords = ['patch']
message_count = 26.0
messages = ['300043', '300044', '300268', '300309', '300605', '301011', '301012', '301084', '301270', '301287', '301399', '301422', '301424', '302159', '302483', '302833', '302898', '302925', '302927', '302930', '302931', '302975', '303037', '307254', '309469', '309471']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'serhiy.storchaka', 'sping', 'tianlynn']
pr_nums = ['3315', '3350', '3351', '3352', '3353', '3354', '3570', '3745', '3746', '3751', '3785']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31170'
versions = ['Python 3.4', 'Python 3.5']

[tianlynn]

utf8_toUtf8(const ENCODING *UNUSED_P(enc),
            const char **fromP, const char *fromLim,
            char **toP, const char *toLim)
{
  char *to;
  const char *from;
  const char *fromLimInitial = fromLim;

/* Avoid copying partial characters. */
align_limit_to_full_utf8_characters(*fromP, &fromLim);

for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++)
*to = *from;
*fromP = from;
*toP = to;

  if (fromLim < fromLimInitial)
    return XML_CONVERT_INPUT_INCOMPLETE;
  else if ((to == toLim) && (from < fromLim))
    // <===== Bug is here. In case (to == toLim), it's possible that
    //        from is still pointing to partial character. For example,
    //        a character with 3 bytes (A, B, C) and form is pointing to C.
    //        It means only A and B is copied to output buffer. Next
    //        scanning will start with C which could be considered as invalid
    //        byte and got dropped. After this, only "AB" is kept in memory
    //        and thus it will lead to invalid continuation byte.
    return XML_CONVERT_OUTPUT_EXHAUSTED;
  else
    return XML_CONVERT_COMPLETED;
}

[serhiy-storchaka]

This is not our code. Please use corresponding bug tracker [1] if you have found a bug in Expat.

But I think there is not a bug here.

[1] https://github.com/libexpat/libexpat/issues

[tianlynn]

Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here)

[hartwork]

For the record, the upstream fix is commit libexpat/libexpat@74a7090 and it will be part of Expat 2.2.4.

[hartwork]

Expat 2.2.4 with a fix has been released now: https://github.com/libexpat/libexpat/releases

[serhiy-storchaka]

I confirm the regression (see bpo-31303 for reproducer). Victor, do you mind to update expat to 2.2.4?

This issue can be classified as a security issue, since a regression was added in security update.

[serhiy-storchaka]

Corresponding Expat issue: libexpat/libexpat#115.

[benjaminp]

We can put expat 2.2.4 in 2.7.14 final.

[vstinner]

I produced attached PR 3315 using attached cpython_rebuild_expat_dir.sh + revert Modules/expat/expat_external.h change to keep #include "pyexpatns.h".

[vstinner]

New changeset 759e30e by Victor Stinner in branch 'master':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315)
759e30e

[vstinner]

New changeset e5f2f80 by Victor Stinner in branch '3.6':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315) (bpo-3350)
e5f2f80

[ned-deily]

New changeset 297516e by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3352)
297516e

[benjaminp]

New changeset c00d508 by Benjamin Peterson (Victor Stinner) in branch '2.7':
bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315)
c00d508

[vstinner]

I wrote an non-regression test for the Python master branch using the test case attached to libexpat/libexpat#115: PR 3570.

[ned-deily]

I believe all the branches except 3.5 and 3.4 have been updated so adjusting the Versions field accordingly. All yours, Larry!

[larryhastings]

New changeset 86a713c by larryhastings (Victor Stinner) in branch '3.4':
[3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3353)
86a713c

[larryhastings]

New changeset f2492bb by larryhastings (Victor Stinner) in branch '3.5':
[3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3354)
f2492bb

[vstinner]

New changeset e6d9fcb by Victor Stinner in branch 'master':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570)
e6d9fcb

[vstinner]

New changeset 5f5da72 by Victor Stinner in branch '2.7':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570) (bpo-3745)
5f5da72

[vstinner]

New changeset ad051cb by Victor Stinner in branch '3.6':
bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570) (bpo-3746)
ad051cb

[vstinner]

libexpat has been upgraded to version 2.2.4 in Python 2.7, 3.3, 3.4, 3.5, 3.6 and master.

I added an unit test to Python 2.7, 3.6 and master. I'm not sure about backporting the new unit test to Python 3.3, 3.4 and 3.5.

I close the issue.

[vstinner]

Oh, it seems like Steve Dower found an issue on Windows: PR 3751.

I reopen the issue.

[larryhastings]

New changeset 44c1b62 by larryhastings (Steve Dower) in branch '3.5':
[3.5] bpo-31170: Fix inclusion of expat in Windows build projects. (bpo-3751)
44c1b62

[larryhastings]

New changeset 8b11e8d by larryhastings (Victor Stinner) in branch '3.4':
[3.4] bpo-31170: Fix inclusion of expat in Windows build projects (bpo-3785)
8b11e8d

[serhiy-storchaka]

Can this issue be closed now?

[vstinner]

Ah yes, it can be closed. I was waiting 3.4 and 3.5 fixes to be fixed, which is now the case.