Heap-use-after-free issue in `pyexpat` related to `.ExternalEntityParserCreate`

[hartwork]

Bug report

Bug description:

When configured with --with-address-sanitizer, the test below crashes all versions of CPython:

# Copyright (c) 2025 Sebastian Pipping <sebastian@pipping.org>
# Licensed under Zero-Clause BSD ("0BSD")

import pyexpat as expat
import unittest


class ParentParserLifetimeTest(unittest.TestCase):
    """
    Subparsers make use of their parent XML_Parser inside of Expat.
    As a result, parent parsers need to outlive subparsers.
    Regression test for issue 139400
    """
    def test_parent_parser_outlives_its_subparsers(self):
        parser = expat.ParserCreate()
        subparser = parser.ExternalEntityParserCreate(None)

        # Now try to cause garbage collection of the parent parser
        # while it's still being referenced by a related subparser
        del parser


if __name__ == '__main__':
    unittest.main()

The finding was first documented at #139367 (comment) .

For 3.13, the AddressSanitizer crash details are: (click to expand)

# ./python ..test_file_with_test_class_above_added_dot_py.. -v
test_use_after_free__crash (__main__.UseAfterFreeCrashDemoTest.test_use_after_free__crash) ... =================================================================
==16187==ERROR: AddressSanitizer: heap-use-after-free on address 0x7cda1ad7e038 at pc 0x7f4a1af8a94f bp 0x7ffc61f63720 sp 0x7ffc61f63710
READ of size 8 at 0x7cda1ad7e038 thread T0
    #0 0x7f4a1af8a94e in getRootParserOf Modules/expat/xmlparse.c:8660
    #1 0x7f4a1af8a94e in expat_free Modules/expat/xmlparse.c:913
    #2 0x7f4a1af8a94e in expat_free Modules/expat/xmlparse.c:906
    #3 0x7f4a1af8a94e in PyExpat_XML_ParserFree Modules/expat/xmlparse.c:1997
    #4 0x7f4a1af70286 in xmlparse_dealloc Modules/pyexpat.c:1266
    #5 0x558a55f60555 in Py_DECREF Include/object.h:949
    #6 0x558a55f60555 in Py_XDECREF Include/object.h:1042
    #7 0x558a55f60555 in _PyFrame_ClearLocals Python/frame.c:104
    #8 0x558a55f60555 in _PyFrame_ClearExceptCode Python/frame.c:129
    #9 0x558a55e9df91 in clear_thread_frame Python/ceval.c:1682
    #10 0x558a55e9df91 in _PyEval_FrameClearAndPop Python/ceval.c:1709
    #11 0x558a55ec6b44 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:5222
    #12 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #13 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #14 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #15 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #16 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #17 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #18 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #19 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #20 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #21 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #22 0x558a55ebf7de in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1850
    #23 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #24 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #25 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #26 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #27 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #28 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #29 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #30 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #31 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #32 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #33 0x558a55ebf7de in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1850
    #34 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #35 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #36 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #37 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #38 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #39 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #40 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #41 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #42 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #43 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #44 0x558a55eaa121 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:813
    #45 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #46 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #47 0x558a55cfb652 in slot_tp_init Objects/typeobject.c:9816
    #48 0x558a55cd8107 in type_call Objects/typeobject.c:1997
    #49 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #50 0x558a55eaa121 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:813
    #51 0x558a55ed8b3e in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #52 0x558a55ed8b3e in _PyEval_Vector Python/ceval.c:1820
    #53 0x558a55ed8b3e in PyEval_EvalCode Python/ceval.c:604
    #54 0x558a5600eb0e in run_eval_code_obj Python/pythonrun.c:1381
    #55 0x558a5600eb0e in run_eval_code_obj Python/pythonrun.c:1348
    #56 0x558a5600f127 in run_mod Python/pythonrun.c:1489
    #57 0x558a560138d0 in pyrun_file Python/pythonrun.c:1295
    #58 0x558a560138d0 in _PyRun_SimpleFileObject Python/pythonrun.c:517
    #59 0x558a5601421c in _PyRun_AnyFileObject Python/pythonrun.c:77
    #60 0x558a560833ec in pymain_run_file_obj Modules/main.c:410
    #61 0x558a560833ec in pymain_run_file Modules/main.c:429
    #62 0x558a560833ec in pymain_run_python Modules/main.c:696
    #63 0x558a56085156 in Py_RunMain Modules/main.c:775
    #64 0x558a56085156 in pymain_main Modules/main.c:805
    #65 0x558a56085156 in Py_BytesMain Modules/main.c:829
    #66 0x7f4a1b9a733f  (/lib64/libc.so.6+0x2733f)
    #67 0x7f4a1b9a73f8 in __libc_start_main (/lib64/libc.so.6+0x273f8)
    #68 0x558a55a23754 in _start ([..]/cpython/python+0x19c754)

0x7cda1ad7e038 is located 952 bytes inside of 1096-byte region [0x7cda1ad7dc80,0x7cda1ad7e0c8)
freed by thread T0 here:
    #0 0x7f4a1bd6b9eb  (/usr/lib/gcc/x86_64-pc-linux-gnu/15/libasan.so.8+0x11f9eb)
    #1 0x7f4a1af87e12 in expat_free Modules/expat/xmlparse.c:934
    #2 0x7f4a1af87e12 in expat_free Modules/expat/xmlparse.c:906
    #3 0x7f4a1af87e12 in PyExpat_XML_ParserFree Modules/expat/xmlparse.c:2011
    #4 0x7f4a1af70286 in xmlparse_dealloc Modules/pyexpat.c:1266
    #5 0x558a55f60555 in Py_DECREF Include/object.h:949
    #6 0x558a55f60555 in Py_XDECREF Include/object.h:1042
    #7 0x558a55f60555 in _PyFrame_ClearLocals Python/frame.c:104
    #8 0x558a55f60555 in _PyFrame_ClearExceptCode Python/frame.c:129
    #9 0x558a55e9df91 in clear_thread_frame Python/ceval.c:1682
    #10 0x558a55e9df91 in _PyEval_FrameClearAndPop Python/ceval.c:1709
    #11 0x558a55ec6b44 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:5222
    #12 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #13 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #14 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #15 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #16 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #17 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #18 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #19 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #20 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #21 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #22 0x558a55ebf7de in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1850
    #23 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #24 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #25 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #26 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #27 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #28 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #29 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #30 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #31 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #32 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #33 0x558a55ebf7de in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1850
    #34 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #35 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #36 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #37 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #38 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #39 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #40 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #41 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #42 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #43 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #44 0x558a55eaa121 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:813
    #45 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #46 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #47 0x558a55cfb652 in slot_tp_init Objects/typeobject.c:9816
    #48 0x558a55cd8107 in type_call Objects/typeobject.c:1997

previously allocated by thread T0 here:
    #0 0x7f4a1bd6ceab in malloc (/usr/lib/gcc/x86_64-pc-linux-gnu/15/libasan.so.8+0x120eab)
    #1 0x7f4a1af8b227 in parserCreate Modules/expat/xmlparse.c:1364
    #2 0x7f4a1af6f0d1 in newxmlparseobject Modules/pyexpat.c:1211
    #3 0x7f4a1af6f0d1 in pyexpat_ParserCreate_impl Modules/pyexpat.c:1609
    #4 0x7f4a1af6f0d1 in pyexpat_ParserCreate Modules/clinic/pyexpat.c.h:511
    #5 0x558a55c4ed39 in cfunction_vectorcall_FASTCALL_KEYWORDS Objects/methodobject.c:440
    #6 0x558a55b48813 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #7 0x558a55b48813 in PyObject_Vectorcall Objects/call.c:327
    #8 0x558a55eaa121 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:813
    #9 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #10 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #11 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #12 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #13 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #14 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #15 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #16 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #17 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #18 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #19 0x558a55ebf7de in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1850
    #20 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #21 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #22 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #23 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #24 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #25 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #26 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #27 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #28 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #29 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #30 0x558a55ebf7de in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1850
    #31 0x558a55b523d3 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #32 0x558a55b523d3 in method_vectorcall Objects/classobject.c:93
    #33 0x558a55b4d5f6 in _PyVectorcall_Call Objects/call.c:273
    #34 0x558a55b4d5f6 in _PyObject_Call Objects/call.c:348
    #35 0x558a55b4d5f6 in PyObject_Call Objects/call.c:373
    #36 0x558a55eb1e8a in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1362
    #37 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #38 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #39 0x558a55ce88e6 in slot_tp_call Objects/typeobject.c:9570
    #40 0x558a55b470d1 in _PyObject_MakeTpCall Objects/call.c:242
    #41 0x558a55eaa121 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:813
    #42 0x558a55b4e383 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #43 0x558a55b4e383 in _PyObject_Call_Prepend Objects/call.c:504
    #44 0x558a55cfb652 in slot_tp_init Objects/typeobject.c:9816
    #45 0x558a55cd8107 in type_call Objects/typeobject.c:1997

SUMMARY: AddressSanitizer: heap-use-after-free Modules/expat/xmlparse.c:8660 in getRootParserOf
Shadow bytes around the buggy address:
  0x7cda1ad7dd80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7cda1ad7de00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7cda1ad7de80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7cda1ad7df00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7cda1ad7df80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x7cda1ad7e000: fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd
  0x7cda1ad7e080: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
  0x7cda1ad7e100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x7cda1ad7e180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7cda1ad7e200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7cda1ad7e280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==16187==ABORTING

My understanding is that there is a bug in the graph of object relations and that the same parser instance is being freed twice as a consequence.

CC @picnixz

CPython versions tested on:

3.15, 3.14, 3.13, 3.12, 3.11, 3.10, 3.9

Operating systems tested on:

Other, Windows, macOS, Linux

Linked PRs

• gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat #139403
• [3.14] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139606
• [3.13] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139608
• [3.12] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139609
• [3.11] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139612
• [3.10] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139613
• [3.9] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139614
• gh-139400: Move news item from section "Core and Builtins" to section "Security" (GH-139606) #139664
• [3.14] gh-139400: Move NEWS item from section "Core and Builtins" to section "Security" (GH-139606) #139663

[picnixz]

Out of curiosity, does it crash before Expat 2.7.2? I tried with on my Windows machine (no dev session) with Python 3.12.10, and there is no crash. It might be also related to some NDEBUG flag as I'm on a release build but I can't investigate more (or there might be a "silent" heap UAF). The Expat version being used is 2.7.1 there.

[hartwork]

@picnixz does it have ASan baked in? I'm trying with latest CPython 3.9 (commit 4dea0fb) now. It's Expat 2.7.1 there.

[hartwork]

@picnixz does it have ASan baked in? I'm trying with latest CPython 3.9 (commit 4dea0fb) now. It's Expat 2.7.1 there.

Result: not affected.

[hartwork]

@picnixz I'll try 3.13 with a revert to Expat 2.7.2 next, then with a revert to Expat 2.7.1. That should clarify involvement of Expat (although 3.14 with 2.7.3 not being affected already pointed towards "not involved" — but let's see).

[picnixz]

does it have ASan baked in?

No, and that's why I'd like to know if it's an ASAN-only issue or not. Because IIRC, the failing CI tests weren't just affecting the ASAN build; it was affecting the regular builds (e.g., https://github.com/python/cpython/actions/runs/18046453659/job/51358293507?pr=139367)

[hartwork]

No, and that's why I'd like to know if it's an ASAN-only issue or not.

The tests were all green for me before pushing the 3.13 pull request, so that means that with my local GCC the issue did not show prior to activating AddressSanatizer. Could be the same for you.

@picnixz I'll try 3.13 with a revert to Expat 2.7.2 next, then with a revert to Expat 2.7.1. That should clarify involvement of Expat (although 3.14 with 2.7.3 not being affected already pointed towards "not involved" — but let's see).

Update for 3.13: Same crash (as with 2.7.3) after reverting to 2.7.2, no crash after reverting to 2.7.1. That's not expected and not good 😞

[picnixz]

Tomorrow I'll (locally) try the following:

• Revert pyexpat.c in main to its state in 3.13. This will revert the UB fixes and small other stuff but not by much (IOW, I'll just C/C the content of 3.13 to my local main).
• I'll try to see if I also have the failures without ASAN locally. Can you check with clang maybe?

Aside from that, should I understand that it's possible that Expat 2.7.2 introduced a regression but only for 3.13 and below?