Skip to content

Let `fromiter` handle object arrays #4791

Open
fgregg opened this Issue Jun 7, 2014 · 1 comment

3 participants

@fgregg
fgregg commented Jun 7, 2014

In 2006, Tim Hochberg questioned the reason why fromiter does not handle object arrays:

            /* We would need to alter the memory RENEW code to decrement any
               reference counts before just throwing away the memory.
             */

This doesn't seem right. The array that we would be RENEWing is a bunch 
of PyObject*s. The reference counts don't reside there, but in the 
objects themselves. When we do the RENEW, we don't want the reference 
counts to change at all. The one tricky case is if we run out of memory, 
I'm not certain that the current setup correctly deals with reference 
counts in this case, although it appears likely that it should work 
since ret->data should still point to a valid chunk of memory and 
decreffing ret should result in the subsequent deallocation of all the 
stored objects.

http://mail.scipy.org/pipermail/numpy-discussion/2006-November/024747.html

@teoliphant responded

I'm not sure if you are still thinking about this.  I suspect I added 
the comment because I was concerned about two things:

1) a resize that deletes memory without first decrefing object counts 
--- I suspose this can't happen here, though, because RENEW is always 
"growing" memory, right?

2) a resize that grows memory without inserting NULLS.  For Object array 
manipulation to work, you need initialized (to NULL) "empty" memory. 

>The one tricky case is if we run out of memory, 
>I'm not certain that the current setup correctly deals with reference 
>counts in this case, although it appears likely that it should work 
>since ret->data should still point to a valid chunk of memory and 
>decreffing ret should result in the subsequent deallocation of all the 
>  
>
>stored objects.
>  
>
As long as you don't store the pointer to new memory until after the 
RENEW was successful then this should work.

I think my comment came from the possibility of #1 in general use of 
RENEW, but I don't think that is going to happen.  Therefore, the only 
tweak that seams necessary is to initialize any new memory to NULL for 
OBJECT arrays.

http://mail.scipy.org/pipermail/numpy-discussion/2006-November/024841.html

I couldn't find any further discussion of this issue since 2006. Is this analysis of what needs to be done to PyArray_FromIter still sound?

@seberg
NumPy member
seberg commented Jun 7, 2014

Since a new array is being created, I don't see any reason this shouldn't just work. It is probably simple to implement... You need to be careful if errors happen though to not decref unintialized elements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.