Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) #73777

Closed
NatanaelCopa mannequin opened this issue Feb 17, 2017 · 35 comments
Assignees
Labels

Comments

@NatanaelCopa
Copy link
Mannequin

NatanaelCopa mannequin commented Feb 17, 2017

BPO 29591
Nosy @birkenfeld, @vstinner, @larryhastings, @tiran, @benjaminp, @jkloth, @ned-deily, @matrixise, @yan12125
PRs
  • bpo-29591: Upgrade Modules/expat to libexpat 2.2 #2164
  • [3.6] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2200
  • [3.5] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2201
  • [2.7] bpo-29591: Upgrade Modules/expat to libexpat 2.2 (#2164) #2202
  • [3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) #2203
  • [3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (#2164) #2204
  • bpo-29591: [2.7] Update VS project files #2310
  • bpo-30726: PCbuild _elementtree: remove duplicate defines #2348
  • [3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) #2349
  • [3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348) #2350
  • Files
  • rebuild_expat_dir.sh
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
    closed_at = <Date 2017-07-12.12:35:46.638>
    created_at = <Date 2017-02-17.15:39:39.676>
    labels = ['type-security', 'expert-XML', '3.7']
    title = 'expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472)'
    updated_at = <Date 2019-05-10.18:12:37.639>
    user = 'https://bugs.python.org/NatanaelCopa'

    bugs.python.org fields:

    activity = <Date 2019-05-10.18:12:37.639>
    actor = 'ned.deily'
    assignee = 'christian.heimes'
    closed = True
    closed_date = <Date 2017-07-12.12:35:46.638>
    closer = 'larry'
    components = ['XML']
    creation = <Date 2017-02-17.15:39:39.676>
    creator = 'Natanael Copa'
    dependencies = []
    files = ['46949']
    hgrepos = []
    issue_num = 29591
    keywords = []
    message_count = 35.0
    messages = ['288014', '288016', '288017', '288018', '288296', '295841', '295924', '296045', '296046', '296047', '296048', '296050', '296051', '296067', '296255', '296537', '296552', '296574', '296579', '296585', '296689', '296713', '296716', '297932', '298157', '298178', '298179', '298180', '298182', '298206', '298209', '298210', '298215', '298216', '298428']
    nosy_count = 11.0
    nosy_names = ['georg.brandl', 'vstinner', 'larry', 'christian.heimes', 'benjamin.peterson', 'jkloth', 'ned.deily', 'jeremy.kloth', 'matrixise', 'yan12125', 'Natanael Copa']
    pr_nums = ['2164', '2200', '2201', '2202', '2203', '2204', '2310', '2348', '2349', '2350']
    priority = None
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'security'
    url = 'https://bugs.python.org/issue29591'
    versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

    @NatanaelCopa
    Copy link
    Mannequin Author

    NatanaelCopa mannequin commented Feb 17, 2017

    cpython bundles expat in Modules/expat/ and needs to be updated to expat-2.2.0 to fix various security vulnerabilities.

    21 June 2016, Expat 2.2.0 released.
    Release 2.2.0 includes security & other bug fixes.

    Security fixes

    CVE-2016-0718 (issue 537)
    Fix crash on malformed input

    CVE-2016-4472
    Improve insufficient fix to CVE-2015-1283 / CVE-2015-2716 introduced with Expat 2.1.1

    CVE-2016-5300 (issue 499)
    Use more entropy for hash initialization than the original fix to CVE-2012-0876

    CVE-2012-6702 (issue 519)
    Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

    Fix should be applied to all maintained python branches.

    @NatanaelCopa NatanaelCopa mannequin added 3.7 type-security A security issue labels Feb 17, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Feb 17, 2017

    CVE-2012-6702 (issue 519)
    Resolve troublesome internal call to srand that was introduced with Expat 2.1.0 when addressing CVE-2012-0876 (issue 496)

    https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-6702

    Extract of Modules/pyexpat.c:
    ---

    #if ((XML_MAJOR_VERSION >= 2) && (XML_MINOR_VERSION >= 1)) || defined(XML_HAS_SET_HASH_SALT)
        /* This feature was added upstream in libexpat 2.1.0.  Our expat copy
         * has a backport of this feature where we also define XML_HAS_SET_HASH_SALT
         * to indicate that we can still use it. */
        XML_SetHashSalt(self->itself,
                        (unsigned long)_Py_HashSecret.prefix);
    #endif

    Python 2.7, 3.5, 3.6 and 3.7 have this call at least (I didn't check other versions).

    @vstinner
    Copy link
    Member

    vstinner commented Feb 17, 2017

    You may want to look also at https://pypi.python.org/pypi/defusedxml

    @tiran
    Copy link
    Member

    tiran commented Feb 17, 2017

    CVE-2016-0718 and CVE-2016-4472 might be relevant for Python. CVE-2016-5300 and CVE-2012-6702 are irrelevant. As Victor already pointed out, Python seeds libexpat from a good CPRNG.

    @tiran tiran self-assigned this Feb 17, 2017
    @NatanaelCopa NatanaelCopa mannequin changed the title Various security vulnerabilities in bundled expat Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) Feb 20, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Feb 21, 2017

    I'm working on a new documentation of Python vulnerabilities to help to handle such issue:
    http://python-security.readthedocs.io/en/latest/vulnerabilities.html

    @ned-deily
    Copy link
    Member

    ned-deily commented Jun 13, 2017

    Note that a duplicate of this issue was opened as bpo-30610 and @matrixise was working on a PR there to update the embedded expat to 2.2.0. Since there are CVE's and a demo crash supplied in bpo-30610, it seems to me we need to fix this for 3.6.2rc1 so I'm making this a "release blocker" and delaying the release. I'm willing to be convinced otherwise. Christian or Victor, can one of you please follow up on this for the 3.6 branch ASAP? Thanks!

    @vstinner
    Copy link
    Member

    vstinner commented Jun 13, 2017

    I upgraded Modules/expat/ to expat 2.2 using attached rebuild_expat_dir.sh script:
    #2164

    TODO: Should be done later in the master branch, once the security fix is handled.

    @vstinner
    Copy link
    Member

    vstinner commented Jun 14, 2017

    New changeset 23ec4b5 by Victor Stinner in branch 'master':
    bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164)
    23ec4b5

    @vstinner
    Copy link
    Member

    vstinner commented Jun 14, 2017

    Python 3.3 currently embeds a copy of libexpat 2.1.0, wheras other branches have libexpat 2.1.1:
    http://python-security.readthedocs.io/vuln/issue_26556_expat_2.1.1.html

    @vstinner
    Copy link
    Member

    vstinner commented Jun 14, 2017

    New changeset 86b9537 by Victor Stinner in branch '3.6':
    bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164) (bpo-2200)
    86b9537

    @vstinner
    Copy link
    Member

    vstinner commented Jun 14, 2017

    @ned Deily: I removed the "release blocker" flag, since I just merged my PR to update libexpat to 2.2 in the Python 3.6 branch.

    @vstinner
    Copy link
    Member

    vstinner commented Jun 15, 2017

    New changeset 0e4571a by Victor Stinner in branch '2.7':
    bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164) (bpo-2202)
    0e4571a

    @vstinner
    Copy link
    Member

    vstinner commented Jun 15, 2017

    New changeset 8c797ed by Victor Stinner in branch '3.5':
    bpo-29591: Upgrade Modules/expat to libexpat 2.2 (bpo-2164) (bpo-2201)
    8c797ed

    @ned-deily
    Copy link
    Member

    ned-deily commented Jun 15, 2017

    Thanks, Victor, for seeing this through and thanks, everyone else, for the reviews and assistance.

    @ned-deily
    Copy link
    Member

    ned-deily commented Jun 18, 2017

    FYI, expat 2.2.1 has now been released. See bpo-30694 for details.

    @jkloth
    Copy link
    Contributor

    jkloth commented Jun 21, 2017

    Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

    @vstinner vstinner changed the title Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) expat 2.2.0: Various security vulnerabilities in bundled expat (CVE-2016-0718 and CVE-2016-4472) Jun 21, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Jun 21, 2017

    Added pull_request2355 to address issues from upgrading to Expat 2.2.0 on Windows 2.7

    I would prefer to first fix the new vulnerabilities, by upgrading expat to 2.2.1, and then review your change.

    => #2312

    @vstinner
    Copy link
    Member

    vstinner commented Jun 21, 2017

    New changeset ab3b0ad by Victor Stinner (Jeremy Kloth) in branch '2.7':
    bpo-29591: Update VS project files (bpo-2310)
    ab3b0ad

    @jeremykloth
    Copy link
    Mannequin

    jeremykloth mannequin commented Jun 21, 2017

    Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
    PCbuild\_elementtree.vcxproj should probably be merged forward as
    well.

    On Wed, Jun 21, 2017 at 1:14 PM, STINNER Victor <report@bugs.python.org> wrote:

    STINNER Victor added the comment:

    New changeset ab3b0ad by Victor Stinner (Jeremy Kloth) in branch '2.7':
    bpo-29591: Update VS project files (bpo-2310)
    ab3b0ad

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue29591\>


    @vstinner
    Copy link
    Member

    vstinner commented Jun 21, 2017

    Jeremy Kloth added the comment:

    Just a note with the PR, the changes to PCbuild\pyexpat.vcxproj and
    PCbuild\_elementtree.vcxproj should probably be merged forward as
    well.

    PR 2310. Yes, I agree. Can you please propose patches for master, and
    then 3.6 and 3.5, please?

    @vstinner
    Copy link
    Member

    vstinner commented Jun 23, 2017

    New changeset c8fb58b by Victor Stinner in branch 'master':
    bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348)
    c8fb58b

    @vstinner
    Copy link
    Member

    vstinner commented Jun 23, 2017

    New changeset f42ce17 by Victor Stinner in branch '3.5':
    [3.5] bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348) (bpo-2350)
    f42ce17

    @vstinner
    Copy link
    Member

    vstinner commented Jun 23, 2017

    New changeset d32a059 by Victor Stinner in branch '3.6':
    [3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348) (bpo-2349)
    d32a059

    @ned-deily
    Copy link
    Member

    ned-deily commented Jul 8, 2017

    New changeset 5777e79 by Ned Deily (Victor Stinner) in branch '3.6':
    [3.6] bpo-30726: PCbuild _elementtree: remove duplicate defines (bpo-2348) (bpo-2349)
    5777e79

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Jul 11, 2017

    I don't quite understand what's happening on this issue. I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0. This issue was created to upgrade CPython to 2.2.0. But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!

    I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

    @vstinner
    Copy link
    Member

    vstinner commented Jul 11, 2017

    I don't quite understand what's happening on this issue. I see that master, 3.6, 3.6, and 2.7 have been upgraded to expat 2.2.0. This issue was created to upgrade CPython to 2.2.0. But the PR against 3.3 and 3.4 upgrade expat to 2.2.1?!

    I'm not against this change in principle, I'm just trying to understand why a) it doesn't match the issue, b) why 3.3 and 3.4 are special, c) why we don't upgrade master & 3.6 & 3.5 & 2.7 to expat 2.2.1.

    I upgraded libexpat to 2.2.0 in this issue, and then to libexpat 2.2.1 in bpo-30694.

    For 3.3 and 3.4 pull requests, I chose to use this bpo number.

    3.3: #2204
    3.4: #2203

    So these pull requests upgrade directly to 2.2.1.

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Jul 11, 2017

    Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1. I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0.

    @vstinner
    Copy link
    Member

    vstinner commented Jul 11, 2017

    Larry: "Please instead choose to use bpo-30694 for the upgrades of 3.3 and 3.4 to expat 2.2.1. I guess there are historical reasons why the PRs are here, but bpo stands as a historical record; let's not confuse posterity by upgrading to 2.2.1 using a bpo issue talking about--and upgrading four branches to--2.2.0."

    I just updated the 3.4 PR.

    In fact, the PR backports the libexpat 2.2.0 commit *and* then the libexpat 2.2.1 commit. Since it's not possible to create a "patch serie" (in GitHub, it would mean a PR which depends on another PR), I chose to stack the two commits in the same PR and reuse the existing PR to not loose context.

    @vstinner
    Copy link
    Member

    vstinner commented Jul 11, 2017

    I changed the PR title to mention the two bpo.

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Jul 12, 2017

    Okay. Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades. Job done.

    The discussions for PRs 2203 and 2204 should move to Issue bpo-30694, which is for the upgrade to expat 2.2.*1*.

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Jul 12, 2017

    New changeset 71572bb by larryhastings (Victor Stinner) in branch '3.4':
    [3.4] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (bpo-2164) (bpo-2203)
    71572bb

    @vstinner
    Copy link
    Member

    vstinner commented Jul 12, 2017

    Okay. Closing this bug, because all the branches that are being upgraded to expat 2.2.*0* have already gotten their upgrades. Job done.

    Well, technically 3.3 wasn't upgraded yet:
    #2204

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Jul 12, 2017

    Correct. But technically 3.3 is being upgraded to 2.2.*1*, which is being tracked on--repeating myself here--Issue bpo-30694.

    @larryhastings
    Copy link
    Contributor

    larryhastings commented Jul 12, 2017

    Here, I'll remove 3.4 and 3.5 from the versions affected. Now everybody can be pedantic!

    @ned-deily
    Copy link
    Member

    ned-deily commented Jul 16, 2017

    New changeset ab90986 by Ned Deily (Victor Stinner) in branch '3.3':
    [3.3] bpo-29591, bpo-30694: Upgrade Modules/expat to libexpat 2.2.1 (bpo-2164) (bpo-2204)
    ab90986

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants