Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

Closed
hartwork mannequin opened this issue Feb 18, 2022 · 14 comments
Closed

Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) #90950

hartwork mannequin opened this issue Feb 18, 2022 · 14 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes topic-XML type-security A security issue

Comments

@hartwork
Copy link
Mannequin

hartwork mannequin commented Feb 18, 2022

BPO 46794
Nosy @ned-deily, @ambv, @mgorny, @mattip, @hartwork, @corona10, @miss-islington
PRs
  • bpo-46794: Bump up the libexpat version into 2.4.6 #31487
  • bpo-46794: Bump up the libexpat version into 2.4.6 #31487
  • [3.10] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31518
  • [3.9] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31519
  • [3.8] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31520
  • [3.7] bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) #31521
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/corona10'
    closed_at = <Date 2022-03-02.09:20:33.019>
    created_at = <Date 2022-02-18.23:36:28.648>
    labels = ['type-security', '3.8', '3.9', '3.10', '3.11', 'expert-XML', '3.7']
    title = 'Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs)'
    updated_at = <Date 2022-03-05.16:57:31.480>
    user = 'https://github.com/hartwork'

    bugs.python.org fields:

    activity = <Date 2022-03-05.16:57:31.480>
    actor = 'mattip'
    assignee = 'corona10'
    closed = True
    closed_date = <Date 2022-03-02.09:20:33.019>
    closer = 'corona10'
    components = ['XML']
    creation = <Date 2022-02-18.23:36:28.648>
    creator = 'sping'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 46794
    keywords = ['patch']
    message_count = 14.0
    messages = ['413517', '413587', '413596', '413597', '413598', '413606', '413762', '413765', '413766', '413862', '414333', '414525', '414537', '414587']
    nosy_count = 7.0
    nosy_names = ['ned.deily', 'lukasz.langa', 'mgorny', 'mattip', 'sping', 'corona10', 'miss-islington']
    pr_nums = ['31487', '31487', '31518', '31519', '31520', '31521']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'security'
    url = 'https://bugs.python.org/issue46794'
    versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

    @hartwork
    Copy link
    Mannequin Author

    hartwork mannequin commented Feb 18, 2022

    @hartwork hartwork mannequin added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes topic-XML type-security A security issue labels Feb 18, 2022
    @mgorny
    Copy link
    Mannequin

    mgorny mannequin commented Feb 20, 2022

    BTW there are test regressions with expat 2.4.5, apparently due to some test snippets now being rejected as invalid XML:

    ======================================================================
    ERROR: test_issue3151 (test.test_xml_etree.BugsTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1718, in feed
        self.parser.Parse(data, False)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    xml.parsers.expat.ExpatError: syntax error: line 1, column 0
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/mgorny/git/cpython/Lib/test/test_xml_etree.py", line 2196, in test_issue3151
        e = ET.XML('<prefix:localname xmlns:prefix="${stuff}"/>')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1347, in XML
        parser.feed(text)
        ^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1720, in feed
        self._raiseerror(v)
        ^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/etree/ElementTree.py", line 1627, in _raiseerror
        raise err
        ^^^^^^^^^
    xml.etree.ElementTree.ParseError: syntax error: line 1, column 0

    ======================================================================
    ERROR: testEncodings (test.test_minidom.MinidomTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1150, in testEncodings
        self.assertRaises(UnicodeDecodeError, parseString,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 734, in assertRaises
        return context.handle('assertRaises', args, kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 218, in handle
        callable_obj(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
        return expatbuilder.parseString(string)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
        return builder.parseString(string)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
        parser.Parse(string, True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 5

    ======================================================================
    ERROR: testExceptionOnSpacesInXMLNSValue (test.test_minidom.MinidomTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/home/mgorny/git/cpython/Lib/test/test_minidom.py", line 1613, in testExceptionOnSpacesInXMLNSValue
        parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/dom/minidom.py", line 1998, in parseString
        return expatbuilder.parseString(string)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 925, in parseString
        return builder.parseString(string)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/mgorny/git/cpython/Lib/xml/dom/expatbuilder.py", line 223, in parseString
        parser.Parse(string, True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    xml.parsers.expat.ExpatError: syntax error: line 1, column 0

    @hartwork
    Copy link
    Mannequin Author

    hartwork mannequin commented Feb 20, 2022

    Hi Michal,

    TL;DR would be:

    • There is a regression but none of these test fails are related.

    • There will be a release Expat 2.4.6 with the regression fixed later today.

    • The 3 failing tests need (small) adjustments to Expat 2.4.5
      and these fails are not considered bugs in Expat.

    I will demo a fix to 2 of the 3 test fails below:

    # git diff -U1 | cat
    diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
    index 1663b1f114..38cea97a97 100644
    --- a/Lib/test/test_minidom.py
    +++ b/Lib/test/test_minidom.py
    @@ -12,2 +12,3 @@
     from xml.dom.minidom import getDOMImplementation
    +from xml.parsers.expat import ExpatError
     
    @@ -1149,4 +1150,6 @@ def testEncodings(self):
             # of crashing
    -        self.assertRaises(UnicodeDecodeError, parseString,
    -                b'<fran\xe7ais>Comment \xe7a va ? Tr\xe8s bien ?</fran\xe7ais>')
    +        self.assertRaises(ExpatError, parseString,
    +                b'<fran\xe7ais></fran\xe7ais>')
    +        self.assertRaises(ExpatError, parseString,
    +                b'<franais>Comment \xe7a va ? Tr\xe8s bien ?</franais>')
     
    @@ -1611,3 +1614,3 @@ def testEmptyXMLNSValue(self):
         def testExceptionOnSpacesInXMLNSValue(self):
    -        with self.assertRaisesRegex(ValueError, 'Unsupported syntax'):
    +        with self.assertRaisesRegex(ExpatError, "syntax error"):
                 parseString('<element xmlns:abc="http:abc.com/de f g/hi/j k"><abc:foo /></element>')
                 

    For the third test, the key is that the closing curly brace is used as the
    namespace separator in line 3660…

    self->parser = EXPAT(ParserCreate_MM)(encoding, &ExpatMemoryHandler, "}");

    …in file Modules/_elementtree.c (which is okay but part of the test fail).

    Best

    Sebastian

    @hartwork hartwork mannequin changed the title Please update bundled libexpat to 2.4.5 with security fixes (5 CVEs) Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) Feb 20, 2022
    @hartwork hartwork mannequin changed the title Please update bundled libexpat to 2.4.5 with security fixes (5 CVEs) Please update bundled libexpat to 2.4.6 with security fixes (5 CVEs) Feb 20, 2022
    @mgorny
    Copy link
    Mannequin

    mgorny mannequin commented Feb 20, 2022

    Could you make a PR to fix the test failures? I suppose that could speed things up and if not, I'd at least have something to pull into Gentoo.

    @hartwork
    Copy link
    Mannequin Author

    hartwork mannequin commented Feb 20, 2022

    I'm busy with the release upstream at the moment. I'll see what I can do.

    @hartwork
    Copy link
    Mannequin Author

    hartwork mannequin commented Feb 20, 2022

    I have created a dedicated ticket bpo-46811 now, test suite pull request upcoming.

    @corona10
    Copy link
    Member

    New changeset 1935e1c by Dong-hee Na in branch 'main':
    bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
    1935e1c

    @miss-islington
    Copy link
    Contributor

    New changeset 4955a9e by Miss Islington (bot) in branch '3.10':
    bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
    4955a9e

    @miss-islington
    Copy link
    Contributor

    New changeset 87cebb1 by Miss Islington (bot) in branch '3.9':
    bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487)
    87cebb1

    @ned-deily
    Copy link
    Member

    New changeset 15d7594 by Miss Islington (bot) in branch '3.7':
    bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31521)
    15d7594

    @ambv
    Copy link
    Contributor

    ambv commented Mar 2, 2022

    New changeset eb6c840 by Miss Islington (bot) in branch '3.8':
    bpo-46794: Bump up the libexpat version into 2.4.6 (GH-31487) (GH-31520)
    eb6c840

    @corona10 corona10 closed this as completed Mar 2, 2022
    @corona10 corona10 closed this as completed Mar 2, 2022
    @mattip
    Copy link
    Contributor

    mattip commented Mar 4, 2022

    On PyPy, the test test_issue3151 in test_xml_etree.py is failing with libexpat 2.4.6. I think the problem is connected to instantiation of the XMLParser() with parser = expat.ParserCreate(encoding, "}") where "}" is not a valid URI character. In any case, due to libexpat issue 577, libexpat/libexpat#577 they will be releasing a new version 2.4.7 soon.

    @hartwork
    Copy link
    Mannequin Author

    hartwork mannequin commented Mar 4, 2022

    Hi mattip,

    at the core the problem is not the use of non-URI character "}" for a namespace separator but the use of non-URI character "}" in a namespace URI. test_issue3151 is mistaken (meaning that non-URI characters in URIs are malformed XML) and the test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files . Expat pull request libexpat/libexpat#577 is related but it's about URI characters not about non-URI ones, so it does not change anything about test_issue3151 in PyPy. Does that make sense?

    Best, Sebastian

    @mattip
    Copy link
    Contributor

    mattip commented Mar 5, 2022

    [T]he test has been removed in CPython pull request https://github.com/python/cpython/pull/31453/files

    Thanks, I missed that. Makes sense.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes topic-XML type-security A security issue
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants