recfunctions.append_fields fails on arrays containing objects (Trac #1751)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/1751 on 2011-02-28 by trac user aickley, assigned to unknown.

The problem is that append_fields calls array.view() which does not accept arrays containing objects (See [http://projects.scipy.org/numpy/ticket/674])

Test case (to include in numpy/lib/tests/recfunctions.py):

class TestAppendFieldsObj(TestCase):
    """
    Test append_fields with arrays containing objects
    """
    def setUp(self):
        from datetime import date
        self.data = dict(obj=date(2000, 1, 1))

    def test_append_to_objects(self):
        "Test append_fields when the base array contains objects"
        obj = self.data['obj']
        x = np.array([(obj, 1.), (obj, 2.)], dtype=[('A', object), ('B', float)])
        y = np.array([10, 20], dtype=int)
        test = append_fields(x, 'C', data=y, usemask=False)
        control = np.array([(obj, 1.0, 10), (obj, 2.0, 20)],
                           dtype=[('A', object), ('B', float), ('C', int)])
        assert_equal(test, control)
    #
    def test_append_with_objects(self):
        "Test append_fields when the appended data contains objects"
        obj = self.data['obj']
        x = np.array([(10, 1.), (20, 2.)], dtype=[('A', int), ('B', float)])
        y = np.array([obj, obj], dtype=object)
        test = append_fields(y, 'C', data=y, dtypes=object, usemask=False)
        control = np.array([(10, 1.0, obj), (20, 2.0, obj)],
                           dtype=[('A', int), ('B', float), ('C', object)])
        assert_equal(test, control)






======================================================================
ERROR: Test append_fields when the base array contains objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", line 406, in test_append_to_objects
    test = append_fields(x, 'C', data=y, usemask=False)
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 629, in append_fields
    base = merge_arrays(base, usemask=usemask, fill_value=fill_value)
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 399, in merge_arrays
    return seqarrays.view(dtype=seqdtype, type=seqtype)
TypeError: Cannot change data-type for object array.

======================================================================
ERROR: Test append_fields when the appended data contains objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", line 416, in test_append_with_objects
    test = append_fields(y, 'C', data=y, dtypes=object, usemask=False)
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 627, in append_fields
    for (a, n, d) in zip(data, names, dtypes)]
TypeError: Cannot change data-type for object array.

----------------------------------------------------------------------
Ran 33 tests in 0.062s

Regarding the first failure, append_fields calls merge_arrays on the base array:

base = merge_arrays(base, usemask=usemask, fill_value=fill_value)

Actually this line can be completely removed without breaking existing unit tests. I don't like this call because if we are calling merge_array on a single array we are are not actually merging anything, but rather making sure that the array conforms to certain criteria. Maybe this code should be refactored out from merge_arrays...

[numpy-gitbot]

@rgommers wrote on 2011-03-02

[ahaldane]

Reopening this after the partial backtrack in #6562.

Now it is partially fixed, but appending object fields still doesn't work. Once it is fixed, remember to add back the test removed in #6562.

I suspect this will be fixed by #6053.

[hpaulj]

This

base = merge_arrays(base, usemask=usemask, fill_value=fill_value)

is also causing problems if base is a single element masked array.

http://stackoverflow.com/questions/36440557/typeerror-when-appending-fields-to-a-structured-array-of-size-one

If len(base) is 1, merge_arrays calls np.asanyarray([base][0]). In the case of masked structure array this returns a void array element, which gives a subsequent ravel a headache.

Simply dropping this merge_arrays call might the solution, if it's sole purpose is to cleanup some obscure way of passing a base to append_fields.

(I'm not familiar enough with the numpy issues to know whether adding this to this old issue is the best choice, or whether it would be better to raise a new issue.)

I ran the test_recfunctions.py on this modification, and it ran fine. So apparently this base 'cleanup' is not needed for the test cases.

[eric-wieser]

This is also fixed in #8514