PubSub: gRPC channel memory leak/not closed.

[codezizo]

Version 0.42.1

Steps to reproduce

1. Create a SubscriberClient
2. Delete it, and run a AddressSanitizerLeakSanitizer test to detect leak.

Code example

    subscriber = pubsub_v1.SubscriberClient(credentials=credentials)
    subscription_path = subscriber.subscription_path(project_id, subscription_id)
    messages = subscriber.pull(subscription_path, max_messages=1).received_messages

--- python code ends---

Stack trace

Direct leak of 2912 byte(s) in 4 object(s) allocated from:
    #0 0x55e0cdf45ee2 in calloc llvm/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:154:3
    googleapis/google-cloud-python#1 0x55e0ce6b7642 in gpr_zalloc grpc/src/core/lib/gpr/alloc.cc:43:7
    googleapis/google-cloud-python#2 0x55e0ce5fafd9 in grpc_channel_stack_builder_finish(grpc_channel_stack_builder*, unsigned long, int, void (*)(void*, grpc_error*), void*, void**) grpc/src/core/lib/channel/channel_stack_builder.cc:292:13
    googleapis/google-cloud-python#3 0x55e0ce68af4a in grpc_channel_create_with_builder(grpc_channel_stack_builder*, grpc_channel_stack_type) grpc/src/core/lib/surface/channel.cc:79:23
    googleapis/google-cloud-python#4 0x55e0ce68bba2 in grpc_channel_create(char const*, grpc_channel_args const*, grpc_channel_stack_type, grpc_transport*, grpc_resource_user*) grpc/src/core/lib/surface/channel.cc:290:7
    googleapis/google-cloud-python#5 0x55e0ce3d7ebf in grpc_core::(anonymous namespace)::CreateChannel(char const*, grpc_channel_args const*) grpc/src/core/ext/transport/chttp2/client/secure/secure_channel_create.cc:169:7
    googleapis/google-cloud-python#6 0x55e0ce3d7ba3 in grpc_secure_channel_create grpc/src/core/ext/transport/chttp2/client/secure/secure_channel_create.cc:217:15
    googleapis/google-cloud-python#7 0x7ff05f195052 in __pyx_pf_7google3_3net_4grpc_6python_7_cython_6cygrpc_7Channel___cinit__(__pyx_obj_7google3_3net_4grpc_6python_7_cython_6cygrpc_Channel*, _object*, _object*, __pyx_obj_7google3_3net_4grpc_6python_7_cython_6cygrpc_ChannelCredentials*) blaze-out/k8-asan-fastbuild/genfiles/net/grpc/python/_cython/cygrpc.pyx_cython_generated.cc:16812:39
    googleapis/google-cloud-python#8 0x7ff05f194578 in __pyx_pw_7google3_3net_4grpc_6python_7_cython_6cygrpc_7Channel_1__cinit__(_object*, _object*, _object*) blaze-out/k8-asan-fastbuild/genfiles/net/grpc/python/_cython/cygrpc.pyx_cython_generated.cc:16602:13
    googleapis/google-cloud-python#9 0x7ff05f17436d in __pyx_tp_new_7google3_3net_4grpc_6python_7_cython_6cygrpc_Channel(_typeobject*, _object*, _object*) blaze-out/k8-asan-fastbuild/genfiles/net/grpc/python/_cython/cygrpc.pyx_cython_generated.cc:64996:7
    googleapis/google-cloud-python#10 0x55e0d3319338 in type_call python_runtime/v2_7/Objects/typeobject.c:749:11
    googleapis/google-cloud-python#11 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18
    googleapis/google-cloud-python#12 0x55e0d3367433 in do_call python_runtime/v2_7/Python/ceval.c:4612:18
    googleapis/google-cloud-python#13 0x55e0d3367433 in call_function python_runtime/v2_7/Python/ceval.c:4417:17
    googleapis/google-cloud-python#14 0x55e0d3367433 in PyEval_EvalFrameEx python_runtime/v2_7/Python/ceval.c:3013:17
    googleapis/google-cloud-python#15 0x55e0d3360e59 in PyEval_EvalCodeEx python_runtime/v2_7/Python/ceval.c:3627:14
    googleapis/google-cloud-python#16 0x55e0d32e7a55 in function_call python_runtime/v2_7/Objects/funcobject.c:523:14
    googleapis/google-cloud-python#17 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18
    googleapis/google-cloud-python#18 0x55e0d32c9c62 in instancemethod_call python_runtime/v2_7/Objects/classobject.c:2600:14
    googleapis/google-cloud-python#19 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18
    googleapis/google-cloud-python#20 0x55e0d331d86f in slot_tp_init python_runtime/v2_7/Objects/typeobject.c:5872:11
    googleapis/google-cloud-python#21 0x55e0d3319448 in type_call python_runtime/v2_7/Objects/typeobject.c:765:13
    googleapis/google-cloud-python#22 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18
    googleapis/google-cloud-python#23 0x55e0d3367433 in do_call python_runtime/v2_7/Python/ceval.c:4612:18
    googleapis/google-cloud-python#24 0x55e0d3367433 in call_function python_runtime/v2_7/Python/ceval.c:4417:17
    googleapis/google-cloud-python#25 0x55e0d3367433 in PyEval_EvalFrameEx python_runtime/v2_7/Python/ceval.c:3013:17
    googleapis/google-cloud-python#26 0x55e0d3360e59 in PyEval_EvalCodeEx python_runtime/v2_7/Python/ceval.c:3627:14
    googleapis/google-cloud-python#27 0x55e0d32e7a55 in function_call python_runtime/v2_7/Objects/funcobject.c:523:14
    googleapis/google-cloud-python#28 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18
    googleapis/google-cloud-python#29 0x55e0d32c9c62 in instancemethod_call python_runtime/v2_7/Objects/classobject.c:2600:14
    googleapis/google-cloud-python#30 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18
    googleapis/google-cloud-python#31 0x55e0d331d86f in slot_tp_init python_runtime/v2_7/Objects/typeobject.c:5872:11
    googleapis/google-cloud-python#32 0x55e0d3319448 in type_call python_runtime/v2_7/Objects/typeobject.c:765:13
    googleapis/google-cloud-python#33 0x55e0d32ba471 in PyObject_Call python_runtime/v2_7/Objects/abstract.c:2544:18```

As can be seen here gRPC channels must be explicitly closes using the close() method. 
[https://github.com/grpc/grpc/issues/12531]

Thanks!

Let me know if you need anymore information!

[plamut]

@codezizo Thanks for the report!

When somebody reproduces the behavior and confirms it, the issue will be classified as a bug, but sticking with the question label until then.

[GrihmDaVaice]

Hello @plamut ,

I currently use the api google.cloud.datastore.client with gRPC too. I have the same behavior.

for my side the step to reproduce is the following :

    while True:
        old_entity = client.get(obj.key)
        ...
        client.put(old_entity) 

if I disable the gRPC channel on the google api client, the memory leak disappears.

Grihm

[plamut]

@GrihmDaVaice Thanks for the extra info!

I haven't been able to look at this myself yet, but it indeed appears that the client would need to handle this better. Maybe not by automatically closing the channel by itself, as this could affect other use cases, but perhaps an explicit close() method to clean up the resources.

FWIW, something similar was recently done to the BigQuery client in googleapis/google-cloud-python#9894 that also had a resource leak.

In the meantime the subscriber channel can also be closed directly:

subscriber.api.transport.channel.close()

but that goes quite deep into client internals, and is also not well documented. Also, the client should also not be used in the "closed" state, but it currently does not have any notion of that.

A higher-level close() method should be able to address these concerns pretty well.

[plamut]

Just as a quick update, I managed to reproduce this even in the latest subscriber client version (1.1.0 at the time of writing).

The issue is indeed caused by the underlying channel opening two sockets, but then not closing the channel when deleting the client:

(venv-3.7) $ lsof -i -n | grep python
python    13012 peter    7u  IPv4 10284873      0t0  TCP 127.0.0.1:53002->127.0.0.1:40475 (ESTABLISHED)
python    13012 peter   11u  IPv6 10286981      0t0  TCP 192.168.8.109:51452->172.217.18.42:https (ESTABLISHED)
python    13012 peter   20u  IPv4 10288755      0t0  TCP 192.168.8.109:50580->216.58.213.170:https (ESTABLISHED)

(the first socket is opened by my IDE, while the other two are the channel's)

FWIW, the workaround mentioned in my previous comment works (manually closing the channel).

[GrihmDaVaice]

thanks a lots @plamut ,
[Happy New years:) ]

Waiting for the next version, we will try to do this.