[BUG] Serious memory leak in struct-column construction

[rjzamora]

Describe the bug
@mpenn uncovered a serious memory leak related to cudf struct-columns.

Steps/Code to reproduce bug

import cudf

while(True):
    gdf = cudf.Series([{"test": {str(i): "test" * 100000 for i in range(50)}}])

Note that host memory consumption will grow continuously as this while loop runs.

EDIT: It seems that the root problem is in PyArrow. The following code produces the memory leak without cudf:

import pyarrow as pa

while(True):
    arr = pa.array(
        [{"test": {str(i): "test" * 100000 for i in range(50)}}],
    )

(Issue raised here: apache/arrow#41172)

[galipremsagar]

I've tried to avert this memory leak in cudf by doing pa.StructArray.from_pandas(pd.Series(...)) but there is a memory leak there too.

[wence-]

I think it is a pyarrow bug:

import pyarrow as pa

for _ in range(1000):
    long_strings = ["test" * 100000 for i in range(50)]
    val = [{"test": {str(i): s for i, s in enumerate(long_strings)}}]
    x = pa.array(val)

leaks

[rjzamora]

Note that I added a link to the root pyarrow problem above.

[JohnZed]

In the linked bug, @galipremsagar mentions that StructArray.from_arrays does not seem to manifest that issue. Is that something that could potentially be used to work around this?

[galipremsagar]

This leak has been patched in 14.0.2 and 15.0.2 builds of arrow:

• Backport #40412 to 14.0.2 conda-forge/arrow-cpp-feedstock#1369
• Backport #40412 to 15.0.2 conda-forge/arrow-cpp-feedstock#1368

Verified installing locally that this issue is resolved.