ARROW-362: Remove redunant reference count #198
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: I7c9b6af682d569cf2a7c335b1da4fed39632eae9
wesm
reviewed
Nov 6, 2016
| @@ -628,7 +628,6 @@ class ArrowDeserializer { | |||
| PyAcquireGIL lock; | |||
|
|
|||
| // Zero-Copy. We can pass the data pointer directly to NumPy. | |||
| Py_INCREF(py_ref_); | |||
There was a problem hiding this comment.
I'm surprised by this -- PyArray_SetBaseObject claims to steal a reference (https://docs.scipy.org/doc/numpy-dev/reference/c-api.array.html#c.PyArray_SetBaseObject), so if we are not incrementing the memory owner's reference count, then it could get garbage collected while the NumPy array is still alive. I believe you that it's leaking memory. To really be sure about this, we may wish to expose the ConvertArrayToPandas method in pyarrow.Array.to_pandas and then examine the reference counts before and after the conversion.
There was a problem hiding this comment.
I'm going to dig into this a little myself out of curiosity
…row.Array.to_pandas method Change-Id: Ic66c86f6900ff95463228667305760f44d71185c
|
I believe this does what we want: wesm@d85228f feel free to cherry pick that here or i can open a new PR |
|
I went ahead and created #200 so we can get a green build sooner |
Change-Id: I164e7a554a2ac53e205eb67da9edc42f0ba4cc38
Reference count fixes in zero-copy to_pandas
asfgit
pushed a commit
that referenced
this pull request
Nov 7, 2016
close #198 Author: Wes McKinney <wes.mckinney@twosigma.com> Author: Uwe L. Korn <uwelk@xhochy.com> Closes #200 from wesm/ARROW-362 and squashes the following commits: 99df96b [Wes McKinney] Force gc to avoid non-deterministic failure d85228f [Wes McKinney] Be more careful about reference counts in zero-copy handoff, add pyarrow.Array.to_pandas method cc7a6b3 [Uwe L. Korn] ARROW-362: Remove redunant reference count
wesm
pushed a commit
to wesm/arrow
that referenced
this pull request
Sep 2, 2018
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com> Author: Uwe L. Korn <uwelk@xhochy.com> Closes apache#198 from xhochy/PARQUET-785 and squashes the following commits: cc173e1 [Uwe L. Korn] Add 1-level list encoding 467c611 [Korn, Uwe] PARQUET-785: LIST schema conversion for Arrow lists
wesm
pushed a commit
to wesm/arrow
that referenced
this pull request
Sep 4, 2018
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com> Author: Uwe L. Korn <uwelk@xhochy.com> Closes apache#198 from xhochy/PARQUET-785 and squashes the following commits: cc173e1 [Uwe L. Korn] Add 1-level list encoding 467c611 [Korn, Uwe] PARQUET-785: LIST schema conversion for Arrow lists Change-Id: I0a7fb0fa2876663978eaebe5b2de8caf68fe1a05
wesm
pushed a commit
to wesm/arrow
that referenced
this pull request
Sep 6, 2018
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com> Author: Uwe L. Korn <uwelk@xhochy.com> Closes apache#198 from xhochy/PARQUET-785 and squashes the following commits: cc173e1 [Uwe L. Korn] Add 1-level list encoding 467c611 [Korn, Uwe] PARQUET-785: LIST schema conversion for Arrow lists Change-Id: I0a7fb0fa2876663978eaebe5b2de8caf68fe1a05
wesm
pushed a commit
to wesm/arrow
that referenced
this pull request
Sep 7, 2018
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com> Author: Uwe L. Korn <uwelk@xhochy.com> Closes apache#198 from xhochy/PARQUET-785 and squashes the following commits: cc173e1 [Uwe L. Korn] Add 1-level list encoding 467c611 [Korn, Uwe] PARQUET-785: LIST schema conversion for Arrow lists Change-Id: I0a7fb0fa2876663978eaebe5b2de8caf68fe1a05
wesm
pushed a commit
to wesm/arrow
that referenced
this pull request
Sep 8, 2018
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com> Author: Uwe L. Korn <uwelk@xhochy.com> Closes apache#198 from xhochy/PARQUET-785 and squashes the following commits: cc173e1 [Uwe L. Korn] Add 1-level list encoding 467c611 [Korn, Uwe] PARQUET-785: LIST schema conversion for Arrow lists Change-Id: I0a7fb0fa2876663978eaebe5b2de8caf68fe1a05
rafael-telles
pushed a commit
to rafael-telles/arrow
that referenced
this pull request
Nov 11, 2021
…insqlite3Alt [C++] Fix FlightSQLite3Alt
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.