Garbage collection happens on thread switch, freeing in-use reference on 3.13

[gboutry]

Hello,

After a deeper dive on #1030, I'm now certain a garbage collection happens after a context switch in Nova.
Therefore, I am opening a separate issue as this is a very different error from #1030.

Cross-posting from: https://bugs.launchpad.net/nova/+bug/2103413

After more investigation, I'm now certain the core of the issue is the Garbage Collector freeing up object much before their last use.

Quickly setup a reproducer VM: https://pastebin.ubuntu.com/p/gzMwyCZSct/
ppa:gboutry/plucky-proposed-openstack is a ppa containing the RC1 packaging for most openstack projects, but the nova package is carrying the following patch to help track object finalization: https://pastebin.ubuntu.com/p/cj7tb3kmGV/

I have also reverse applied this patch out of bound to have the greenthread identity (the same error happens with and without this patch):
openstack/nova@cd980cd

The carried patch for finalization tracing is tracing: https://opendev.org/openstack/nova/src/commit/98226b60f3fe7b20e8d7f208c12f8d0086cd83d0/nova/network/neutron.py#L1216

This object gets it's attributes freed up very early in the method runtime. I can't pin point a precise LOC where this happens, as it changes all the time. (meaning it's not a subsequent call that frees up the internal objects).

Note that the admin_client object itself does NOT get freed, only its inner objects.

2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] /usr/lib/python3.13/weakref.py:590 - finalizer base_client
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] base_client - Monotonic delta, 0.158562s
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] base_client - end of finalizer
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - finalizer called: <Thread(Thread-1 (poll), started daemon 123734175578944)>
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - finalizer called: {123734291935488: <_MainThread(MainThread, started 123734291935488)>, 123734175578944: <Thread(Thread-1 (poll), started daemon 123734175578944)>, 123734174920256: <Thread(Rabbit-heartbeat, started daemon 123734174920256)>, 123734127568576: <Thread(Thread-2 (_native_thread), started daemon 123734127568576)>, 123734048372416: <Thread(tpool_thread_0, started daemon 123734048372416)>, 123734039979712: <Thread(tpool_thread_1, started daemon 123734039979712)>, 123734031587008: <Thread(tpool_thread_2, started daemon 123734031587008)>, 123734023194304: <Thread(tpool_thread_3, started daemon 123734023194304)>, 123734014801600: <Thread(tpool_thread_4, started daemon 123734014801600)>, 123734006408896: <Thread(tpool_thread_5, started daemon 123734006408896)>, 123733998016192: <Thread(tpool_thread_6, started daemon 123733998016192)>, 123733444392640: <Thread(tpool_thread_7, started daemon 123733444392640)>, 123733435999936: <Thread(tpool_thread_8, started daemon 123733435999936)>, 123733427607232: <Thread(tpool_thread_9, started daemon 123733427607232)>, 123733419214528: <Thread(tpool_thread_10, started daemon 123733419214528)>, 123733410821824: <Thread(tpool_thread_11, started daemon 123733410821824)>, 123733402429120: <Thread(tpool_thread_12, started daemon 123733402429120)>, 123733394036416: <Thread(tpool_thread_13, started daemon 123733394036416)>, 123732907521728: <Thread(tpool_thread_14, started daemon 123732907521728)>, 123732899129024: <Thread(tpool_thread_15, started daemon 123732899129024)>, 123732890736320: <Thread(tpool_thread_16, started daemon 123732890736320)>, 123732882343616: <Thread(tpool_thread_17, started daemon 123732882343616)>, 123732873950912: <Thread(tpool_thread_18, started daemon 123732873950912)>, 123732865558208: <Thread(tpool_thread_19, started daemon 123732865558208)>, 123734129856768: <Thread(Thread-3 (_runner), started daemon 123734129856768)>}
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] /usr/lib/python3.13/weakref.py:590 - finalizer httpclient
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - Monotonic delta, 0.171590s
2025-04-02 08:02:48.500 26057 INFO nova.network.neutron [-] httpclient - end of finalizer

PS: admin_client is the object where this happens the most, but not at all the runs, some of the other failures, it's neutron_client (https://opendev.org/openstack/nova/src/commit/98226b60f3fe7b20e8d7f208c12f8d0086cd83d0/nova/network/neutron.py#L1212) that gets its internal object freed.

using gc.disable() makes the issue disappear (yay! disable gc!) or actually holding a hard ref to admin_client.base_client.httpclient makes the method pass most of the time...

[hemanthnakkina]

Just want to add admin_client.base_client.httpclient still points to a reference but vars or __dict__ of the instance returns empty dict.

[cbf123]

Does anyone know of work that's being done to address this? Or do we just need to move off eventlet before moving to python 3.13?

[4383]

Same answer than on #1030 (comment)
Please migrates off Eventlet https://removal.eventlet.org/guide/getting-started/

[SeanMooney]

migrating off eventlet is not really an option at this tiem.

debian trixie and ubuntu 25.04 but ship python 3.13 by default so this need to be resovled to supprot openstack on those release until the eventlet removeal can be completed in 2026.2 or 2027.1

[4383]

@gboutry: Hey Guillaume, please can you recheck this scenario over the HEAD of master (or a branch that contains #1047). A parallel track is #1042. Would be good to test your use case over these 2 options. Please let us know your observations.

In all the case I'm now gonna go to prepare a new release to land several fixes merged during the latest weeks, including #1047

[gboutry]

Hey @4383, I have already tested a scenario with #1047, and it didn't fix.

About #1042, I don't really see how it will solve the issue. But I can try after it's been rebased on master.

[4383]

ack, no problem, just trying to confirm the current state.

[4383]

Hello,

TL;DR: Following an OpenStack IRC discussion, apparently this is a Python bug, not an Eventlet bug. The problem seems to be that GC doing stuff it shouldn't with circle references. See python/cpython#130327

I'm forwarding here a summary of a recent discussion that happened on the OpenStack IRC side.
Please find out the IRC logs:

• https://meetings.opendev.org/irclogs/%23openstack-eventlet-removal/%23openstack-eventlet-removal.2025-06-18.log.html#openstack-eventlet-removal.2025-06-18.log.html#t2025-06-18T13:29:19
• https://meetings.opendev.org/irclogs/%23openstack-eventlet-removal/%23openstack-eventlet-removal.2025-06-19.log.html

Indeed, a similar problem have been observed by Johannes Kulik (@joker-at-work?) through neutron-client.

Johannes reported his observation into the IRC channel (see the IRC log above).

Here is a summary of the way he found to trigger the problem into OpenStack:

• Johannes observed that bug appears when using neutron.get_client() inside a function called network() of his reproducer.
• The affected clients come from python-neutronclient, particularly SessionClient, defined in neutronclient/client.py.
• Keystone is involved indirectly through authentication (keystoneauth1.session.Session), but it is not where the bug is triggered or observed.
• By using (Pdb) [o for o in gc.get_objects() if isinstance(o, t)] When the exception is raised, Johannes observed that two client instances are present in memory. One of them (the second) has a completely empty __dict__, indicating that its attributes were unexpectedly cleared.

Johannes continued his investigation and patched the neutron client. From there he cannot reproduce the bug anymore in 100 runs with this patch.
Without this patch, any client has an endless list of base_client.base_client.base_client. Those eventually get their __dict__ cleared for some reason, while the ClientWrapper keeps its own. the httpclient (SessionClient) doesn't necessarily get the __dict__ emptied, though.

Thomas (@Zigo) tested this patch on Trixie and was able to spawn VM successfully.

Johannes was able to reduce the scope of this problem without involving OpenStack or Eventlet in the middle, meaning that we are able to reproduce it even outside of any Eventlet context.
Here is the recipe:

import gc


class A:

    def __init__(self, client):
        self.__dict__ = client.__dict__
        self.client = client

class B:
    def __init__(self):
        self.test_attr = "foo"


a = A(B())
print(a.__dict__)
print(a.client.__dict__)
gc.collect()
print("##  After gc.collect()")
print(a.__dict__)
print(a.client.__dict__)

Running this script with Python 3.13

$ python3.13 reproducer.py
{'test_attr': 'foo', 'client': <__main__.B object at 0x73ea355a8590>}
{'test_attr': 'foo', 'client': <__main__.B object at 0x73ea355a8590>}
##  After gc.collect()
{'test_attr': 'foo', 'client': <__main__.B object at 0x73ea355a8590>}
{}

Running this script with Python 3.12

$ python3.12 reproducer.py 
{'test_attr': 'foo', 'client': <__main__.B object at 0x79c86f355400>}
{'test_attr': 'foo', 'client': <__main__.B object at 0x79c86f355400>}
##  After gc.collect()
{'test_attr': 'foo', 'client': <__main__.B object at 0x79c86f355400>}
{'test_attr': 'foo', 'client': <__main__.B object at 0x79c86f355400>

The bug seems to be:
python/cpython#130327

And it looks like someone else mentioned it elsewhere python/cpython#135552

@gboutry do you mind checking that it fix your problem?

[gboutry]

Thanks for the explanation, then this is really between Nova and python3.13, and we'll need to figure out exactly how to fix that in either of them.

[4383]

I propose to close this discussion and to close this issue, is it ok for you? @gboutry

[4383]

@gboutry In all the case I encourage you to init a discussion with CPython maintainers through the github issues of the CPython repository I quoted in my previous summary.