ArrowArray::try_from_raw should not assume the pointers are from Arc

[viirya]

Which issue does this PR close?

Closes #1333.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

[codecov-commenter]

Codecov Report

Merging #1334 (133cb01) into master (f4c7102) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1334   +/-   ##
=======================================
  Coverage   83.03%   83.03%           
=======================================
  Files         181      181           
  Lines       52949    52951    +2     
=======================================
+ Hits        43965    43968    +3     
+ Misses       8984     8983    -1     

Impacted Files	Coverage Δ
arrow/src/ffi.rs	84.61% <100.00%> (+0.08%)	⬆️
arrow/src/datatypes/field.rs	53.79% <0.00%> (-0.31%)	⬇️
arrow/src/array/transform/mod.rs	84.52% <0.00%> (+0.13%)	⬆️
parquet/src/encodings/encoding.rs	93.71% <0.00%> (+0.19%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4c7102...133cb01. Read the comment docs.

[sunchao]

LGTM. An Arrow array or schema can be allocated by other languages like Java (see here), so we shouldn't assume an Arc here.

cc @jorgecarleitao @alamb

[sunchao]

Merged, thanks @viirya !

[viirya]

Thanks @sunchao !

[alamb]

I am not super familiar with this code but it makes sense to me. The clone here seems to just be a clone of the FFI_ArrowArray struct itself (which is some ints and pointers) which seems reasonable enough to me.

If someone seems performance issues from this code, we can always add a try_from_raw_arc or something, but this looks good to me for now

[viirya]

The clone here seems to just be a clone of the FFI_ArrowArray struct itself (which is some ints and pointers) which seems reasonable enough to me.

That's right.

[viirya]

Thanks @alamb

[wangfenjin]

Guys, not sure if my understanding is right, but I think this commit will break the design and create memory leak.

If we clone the FFI struct, then it means we need to free the pointer by ourself, but if we free FFI_ArrowArray, then the data in this Array will also be free? Which means we can't free the pointer(until the data are used and ready to free, but in reality we can't hold this useless pointer in a big project for such a long time), which create memory leak.

As to the question @viirya raised in #1333 , when manage memory, the one who allocate it should free it, which means in our case, we need to alloc the struct in rust and pass the pointer to java and then also free the memory in rust.

• You can check my code in here: https://github.com/wangfenjin/duckdb-rs/blob/5083d39a4147f8017613304ae5f217a88ac42c2e/src/raw_statement.rs#L58
• When I try to upgrade to version 10, memory leak detected and there is no easy way to fix it.

I suggest we revert this commit. cc @alamb @sunchao

[viirya]

As to the question @viirya raised in #1333 , when manage memory, the one who allocate it should free it, which means in our case, we need to alloc the struct in rust and pass the pointer to java and then also free the memory in rust.

Simply said, I don't think this is correct. That is the whole point of C data interface. The release callback is designed to be called by not only the one allocates the C data interface structure.

[viirya]

If we clone the FFI struct, then it means we need to free the pointer by ourself, but if we free FFI_ArrowArray, then the data in this Array will also be free? Which means we can't free the pointer(until the data are used and ready to free, but in reality we can't hold this useless pointer in a big project for such a long time), which create memory leak.

As the raw pointers are converted to Arc, they will be released correctly eventually. But the data is still there, so of course you should not release the pointers.