Skip to content

Conversation

@provinzkraut
Copy link
Contributor

When using msgspec.Meta(pattern="something"), msgspec creates a leaky reference to an re.Pattern, by increasing its reference count too many times.

This is caused by the fact that it creates the Pattern object with a call to PyObject_CallOneArg, which returns a new strong reference. Msgspec then add this to the Meta object, during which the Pattern's refcount is increased again. Due to this, its refcount can never fall to 0, keeping it alive indefinitely.

We can show this with a simple script:

import gc
import re
import secrets
import msgspec

string_pattern = secrets.token_hex()
msgspec.Meta(pattern=string_pattern)
re.purge()

print(
    any(o for o in gc.get_objects() if isinstance(o, re.Pattern) and o.pattern == string_pattern)
)

this should print False, since there should be no such re.Pattern objects arounds anymore, but this is not the case.


I have fixed this by foregoing the SET_FIELD macro, and instead simply assigning the field directly, without calling Py_XINCREF on its value.

@provinzkraut provinzkraut force-pushed the fix-meta-pattern-refleak branch 2 times, most recently from cc60a08 to 2f8c030 Compare October 26, 2025 15:09
@ofek ofek force-pushed the fix-meta-pattern-refleak branch from 3513781 to bc5b1e0 Compare November 23, 2025 18:48
Copy link
Collaborator

@ofek ofek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ofek ofek merged commit 233d916 into jcrist:main Nov 23, 2025
23 checks passed
@ofek
Copy link
Collaborator

ofek commented Nov 23, 2025

Just a heads up that in my last commit to your branch I fixed another leak I noticed that was already present!

@provinzkraut provinzkraut deleted the fix-meta-pattern-refleak branch November 23, 2025 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants