ENH: concatenate array and raveling elements or arrays when dealing with lists (empty or not) and dtype objects

[HealthyPear]

Proposed new feature or change:

Apologies if this is somewhat already possible, but I couldn't find a solution.

Is it possible to concatenate elements from arrays such as

a = np.array([[],[], [1]],dtype=object)

or concatenate more arrays like these,

b = np.empty(len(a), dtype=object)
c = np.array([[1],[], [0, 2, 3]],dtype=object)

?

The operation I am looking for should do e.g.

d = np.concatenate(a, b)
---> np.array([1],dtype=object)

or

d = np.concatenate(b, c)
---> np.array([1, 0, 2, 3],dtype=object)

or

d = np.concatenate(a, b, c)
---> np.array([1, 1, 0, 2, 3],dtype=object)

I tried with numpy.ravel(), but it doesn't do anything

[JohnMckane]

Hi, for np.concatenate the first argument is an iterable, containing the objects you want to concatenate, in the order you want them concatenated. That is, if you want to concatenate a and b, you would use np.concatenate((a,b)) and get array([list([]), list([]), list([1]), None, None, None], dtype=object). More information is available in the docs https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html.

[HealthyPear]

Hi @JohnMckane ,

I might have explained my issue in an incomplete way:

I am not only interested in concatenate as that method does - which doesn't work with the example arrays I provided:

I am also interested in raveling applied on dtype object arrays elements with different length elements and/or empty elements and arrays obtained concatenating mor object arrays (or a mixture or arrays with different dtypes when casting all as e.g. float64:

[seberg]

Please carefully check the documentation, concatenate supports what you want.

[HealthyPear]

Again, the concatenation is not the root problem for me (apart from when concatenating with array objects containing all empty elements, which doesn't work as I show above, but I can maybe remove from my dataset and concatenate mixed dtype objects).

The raveling of this type of object doesn't work as I would expect - it works by keeping the elements from dtype objects as lists, even when they contain numbers (which I would expect should be possible to cast to some dtype) and/or empty lists (which I would expect to disappear when raveling).

[mattip]

NumPy is trying to never do value based casting: it will not look inside an array to try to determine the "best" alternative. This is because what for you may be "best" will not be the "best" for someone else. If you want casting of an object array to an int array, you must do so yourself.

It seems we are misunderstanding what you want to do. Could you provide a little more context: what is the goal you are trying to achieve? How do you get to a situation where you have empty/list-of-integer object arrays that you wish to compact or unravel?

[HealthyPear]

Hi @mattip ,

Well, the story is long...

Basically, I have data read using Python3 as ragged arrays (XCDF) into astropy QTables.

Unfortunately, astropy doesn't like dtype object arrays, even though they partially support it (which is the reason I managed to get a table with some units that I can at least take a look into - but I cannot really use, so I was trying to understand if I could make use of such column data a-posteriori).

At the same time, I think I might be better off with Awkward Arrays, but I have never used them, they for sure don't support units and it's a whole separate adventure (which is why I wanted to make sure here what was possible to do with this kind of arrays)

[seberg]

I still think concatenate after converting to a list seems like a decent solution to the problem as first stated something like np.concatenate(list(a) + list(b), axis=0) in the worst case, although just np.concatenate(a) might work.

That might not solve all the worries, but based on this issue, I don't see us looking into helpers to make working with ragged/jagged object arrays easier.

Closing, since I think it was more of a question to begin with.