Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

Closed
tianlynn mannequin opened this issue Aug 10, 2017 · 26 comments
Closed

Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) #75353

tianlynn mannequin opened this issue Aug 10, 2017 · 26 comments
Labels
release-blocker stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@tianlynn
Copy link
Mannequin

tianlynn mannequin commented Aug 10, 2017

BPO 31170
Nosy @birkenfeld, @vstinner, @larryhastings, @benjaminp, @ned-deily, @serhiy-storchaka, @hartwork, @tianlynn
PRs
  • bpo-31170: Update libexpat from 2.2.3 to 2.2.4 #3315
  • [3.6] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3350
  • [2.7] bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (#3315) #3351
  • [3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3352
  • [3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3353
  • [3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 #3354
  • bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug #3570
  • [2.7] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3745
  • [3.6] bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (#3570) #3746
  • [3.5] bpo-31170: Fix inclusion of expat in Windows build projects. #3751
  • [3.4] bpo-31170: Fix inclusion of expat in Windows build projects #3785
  • Files
  • cpython_rebuild_expat_dir.sh
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-01-04.15:52:09.415>
    created_at = <Date 2017-08-10.04:48:25.430>
    labels = ['type-bug', 'library', 'release-blocker']
    title = 'Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer)'
    updated_at = <Date 2019-05-10.18:04:35.827>
    user = 'https://github.com/tianlynn'

    bugs.python.org fields:

    activity = <Date 2019-05-10.18:04:35.827>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-01-04.15:52:09.415>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2017-08-10.04:48:25.430>
    creator = 'tianlynn'
    dependencies = []
    files = ['47118']
    hgrepos = []
    issue_num = 31170
    keywords = ['patch']
    message_count = 26.0
    messages = ['300043', '300044', '300268', '300309', '300605', '301011', '301012', '301084', '301270', '301287', '301399', '301422', '301424', '302159', '302483', '302833', '302898', '302925', '302927', '302930', '302931', '302975', '303037', '307254', '309469', '309471']
    nosy_count = 8.0
    nosy_names = ['georg.brandl', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'serhiy.storchaka', 'sping', 'tianlynn']
    pr_nums = ['3315', '3350', '3351', '3352', '3353', '3354', '3570', '3745', '3746', '3751', '3785']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue31170'
    versions = ['Python 3.4', 'Python 3.5']

    @tianlynn
    Copy link
    Mannequin Author

    tianlynn mannequin commented Aug 10, 2017

    utf8_toUtf8(const ENCODING *UNUSED_P(enc),
                const char **fromP, const char *fromLim,
                char **toP, const char *toLim)
    {
      char *to;
      const char *from;
      const char *fromLimInitial = fromLim;

    /* Avoid copying partial characters. */
    align_limit_to_full_utf8_characters(*fromP, &fromLim);

    for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++)
    *to = *from;
    *fromP = from;
    *toP = to;

      if (fromLim < fromLimInitial)
        return XML_CONVERT_INPUT_INCOMPLETE;
      else if ((to == toLim) && (from < fromLim))
        // <===== Bug is here. In case (to == toLim), it's possible that
        //        from is still pointing to partial character. For example,
        //        a character with 3 bytes (A, B, C) and form is pointing to C.
        //        It means only A and B is copied to output buffer. Next
        //        scanning will start with C which could be considered as invalid
        //        byte and got dropped. After this, only "AB" is kept in memory
        //        and thus it will lead to invalid continuation byte.
        return XML_CONVERT_OUTPUT_EXHAUSTED;
      else
        return XML_CONVERT_COMPLETED;
    }

    @tianlynn tianlynn mannequin added 3.7 stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Aug 10, 2017
    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Aug 10, 2017

    This is not our code. Please use corresponding bug tracker [1] if you have found a bug in Expat.

    But I think there is not a bug here.

    [1] https://github.com/libexpat/libexpat/issues

    @tianlynn
    Copy link
    Mannequin Author

    tianlynn mannequin commented Aug 14, 2017

    Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here)

    @tianlynn tianlynn mannequin reopened this Aug 14, 2017
    @hartwork
    Copy link
    Mannequin

    hartwork mannequin commented Aug 15, 2017

    For the record, the upstream fix is commit libexpat/libexpat@74a7090 and it will be part of Expat 2.2.4.

    @hartwork
    Copy link
    Mannequin

    hartwork mannequin commented Aug 20, 2017

    Expat 2.2.4 with a fix has been released now: https://github.com/libexpat/libexpat/releases

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Aug 30, 2017

    I confirm the regression (see bpo-31303 for reproducer). Victor, do you mind to update expat to 2.2.4?

    This issue can be classified as a security issue, since a regression was added in security update.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Aug 30, 2017

    Corresponding Expat issue: libexpat/libexpat#115.

    @benjaminp
    Copy link
    Contributor

    benjaminp commented Sep 1, 2017

    We can put expat 2.2.4 in 2.7.14 final.

    @vstinner vstinner changed the title expat: utf8_toUtf8 cannot properly handle exhausting buffer Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly handle exhausting buffer) Sep 4, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Sep 4, 2017

    I produced attached PR 3315 using attached cpython_rebuild_expat_dir.sh + revert Modules/expat/expat_external.h change to keep #include "pyexpatns.h".

    @vstinner
    Copy link
    Member

    vstinner commented Sep 4, 2017

    New changeset 759e30e by Victor Stinner in branch 'master':
    bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315)
    759e30e

    @vstinner
    Copy link
    Member

    vstinner commented Sep 5, 2017

    New changeset e5f2f80 by Victor Stinner in branch '3.6':
    bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315) (bpo-3350)
    e5f2f80

    @ned-deily
    Copy link
    Member

    ned-deily commented Sep 6, 2017

    New changeset 297516e by Ned Deily (Victor Stinner) in branch '3.3':
    [3.3] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3352)
    297516e

    @benjaminp
    Copy link
    Contributor

    benjaminp commented Sep 6, 2017

    New changeset c00d508 by Benjamin Peterson (Victor Stinner) in branch '2.7':
    bpo-31170: Update libexpat from 2.2.3 to 2.2.4 (bpo-3315)
    c00d508

    @vstinner
    Copy link
    Member

    vstinner commented Sep 14, 2017

    I wrote an non-regression test for the Python master branch using the test case attached to libexpat/libexpat#115: PR 3570.

    @ned-deily
    Copy link
    Member

    ned-deily commented Sep 18, 2017

    I believe all the branches except 3.5 and 3.4 have been updated so adjusting the Versions field accordingly. All yours, Larry!

    @ned-deily ned-deily removed the 3.7 label Sep 18, 2017
    @larryhastings
    Copy link
    Contributor

    larryhastings commented Sep 24, 2017

    New changeset 86a713c by larryhastings (Victor Stinner) in branch '3.4':
    [3.4][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3353)
    86a713c

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Sep 25, 2017

    New changeset f2492bb by larryhastings (Victor Stinner) in branch '3.5':
    [3.5][Security] bpo-30947, bpo-31170: Update expat from 2.2.1 to 2.2.4 (bpo-3354)
    f2492bb

    @vstinner
    Copy link
    Member

    vstinner commented Sep 25, 2017

    New changeset e6d9fcb by Victor Stinner in branch 'master':
    bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570)
    e6d9fcb

    @vstinner
    Copy link
    Member

    vstinner commented Sep 25, 2017

    New changeset 5f5da72 by Victor Stinner in branch '2.7':
    bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570) (bpo-3745)
    5f5da72

    @vstinner
    Copy link
    Member

    vstinner commented Sep 25, 2017

    New changeset ad051cb by Victor Stinner in branch '3.6':
    bpo-31170: Write unit test for Expat 2.2.4 UTF-8 bug (bpo-3570) (bpo-3746)
    ad051cb

    @vstinner
    Copy link
    Member

    vstinner commented Sep 25, 2017

    libexpat has been upgraded to version 2.2.4 in Python 2.7, 3.3, 3.4, 3.5, 3.6 and master.

    I added an unit test to Python 2.7, 3.6 and master. I'm not sure about backporting the new unit test to Python 3.3, 3.4 and 3.5.

    I close the issue.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 25, 2017

    Oh, it seems like Steve Dower found an issue on Windows: PR 3751.

    I reopen the issue.

    @vstinner vstinner reopened this Sep 25, 2017
    @larryhastings
    Copy link
    Contributor

    larryhastings commented Sep 26, 2017

    New changeset 44c1b62 by larryhastings (Steve Dower) in branch '3.5':
    [3.5] bpo-31170: Fix inclusion of expat in Windows build projects. (bpo-3751)
    44c1b62

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Nov 29, 2017

    New changeset 8b11e8d by larryhastings (Victor Stinner) in branch '3.4':
    [3.4] bpo-31170: Fix inclusion of expat in Windows build projects (bpo-3785)
    8b11e8d

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Jan 4, 2018

    Can this issue be closed now?

    @vstinner
    Copy link
    Member

    vstinner commented Jan 4, 2018

    Ah yes, it can be closed. I was waiting 3.4 and 3.5 fixes to be fixed, which is now the case.

    @vstinner vstinner closed this as completed Jan 4, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    release-blocker stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants