How should parameters behave?

[agoose77]

Whenever we perform an Awkward operation, we ultimately have to choose how parameters are merged, and where to put them. In #2179, I have made some changes to the existing rules in order to fix a bug with string merging. These changes would make Indexed[Option]Array nodes more distinct, i.e. not merging their own parameters with their contents.

The purpose of this discussion is to formally discuss where we want our parameter system to go, serve as a reference for PRs that change these rules, and to leave room for #1391

The parameter system in Awkward is very broad, and used in a variety of ways internally. In particular, we have two kinds of parameters:

1. User-defined Parameters

   We want to be able to associate unique metadata with every layout node. Users can set any parameter, but we don't know how these should survive various Awkward operations, so we use check_equal as our metadata merging policy. This ensures that merging parameters only preserves identical key-value pairs. Unlike xarray-style "attrs", global and per-record field #1391, most of this metadata is easily destroyed i.e. it's scoped to the layout node.

   Merging of layouts does not care if user-defined parameters don't match; they're just dropped at the parameter merge step.

2. Awkward-defined Parameters
   In addition to user-generic parameters, we have a set of reserved (special) names e.g. __array__ and __record__. These parameters are required to match in order for layouts to be mergeable.

Array lookup problem

Of particular importance is __array__, whose location Awkward is highly sensitive to. Unlike __record__ , which walks through the layout tree recursively, __array__ is taken from the first-encountered layout node. This causes the following observation

>>> import awkward as ak
>>> import numpy as np
>>> type(ak.Array(ak.contents.IndexedArray(ak.index.Index64(np.arange(2)), ak.to_layout(["Hi", "Bye"]))))
awkward.highlevel.Array
>>> type(ak.Array(ak.to_layout(["Hi", "Bye"])))
awkward.behaviors.string.StringBehavior

The reason that the former returns ak.highlevel.Array in the first example is that __array__ is resolved against the IndexedArray node, which has no parameters.

In many scenarios, we use parameters for user-facing operations like setting a behaviour class. It's my assertion that the user should only be required to reason about the array type, which does not mention the index. Therefore, I argue that the above behaviour is undesirable.

To fix this, we have discussed introducing something like dimension_parameter that behaves like purelist_parameter within a particular dimension (i.e., stops resolution at list boundaries). A generalisation of this would be type_parameters, that are scoped to the dimension and option type.

Clearly, the reserved parameters are special, and therefore we can impose our own custom lifetime rules. An alternative solution, therefore, would be to "promote" the __array__ options, so that they're always set on both the indexed node and its content. I do not prefer this approach.

Array class of option?

If we treat the type as the mechanism for users to configure behaviours, then:

• should var * int64 have the same array class as Option[var * int64]?
• should var * int64 have the same array class as var * int64??

The current rules are (1) no, (2) yes.

Merging arrays with & without __array__

Right now, we treat two arrays as mergeable iff. they have the same __array__. I wonder whether it makes sense to loosen this; if the contents are mergeable, then we need the arrays to have compatible __array__, i.e. not __array__=X, __array__=Y. If the second array does not have an __array__ parameter, is it not mergeable?

[agoose77]

@jpivarski this discussion sets the scene for a tentative PR #2179. At some point, it would be good to get your thoughts on this. Not urgent, so could do so during a 1:1.

[jpivarski]

I agree with your assessment: the original parameters mechanism is more low-level than the thing (it turns out) we actually want. The parameters is a function of layout/form nodes, but users see types, and count "number of dimensions" (through axis) separately from option nodes, record nodes, union nodes, etc.

The idea that dimensions/list nodes are "more first-class" than everything else is a point I didn't appreciate in the original design. It comes up repeatedly, for example, here: Cloud-Drift/ragged-array-idioms#4 A naive attempt to preserve xarray dimension names as parameters lost the information when a node was replaced, and would not work well if that example is ever extended beyond pure lists.

For existing "well known parameters" like __array__, we already have special logic for handling them and rules for propagating them where they need to be (which apparently didn't work in #2173 and is being revised in #2179). I think that should continue because we're already pretty invested in the __array__ and __record__ rules, and they're publicly visible. (We are free to change them because we are fixing bugs; behavior like #2173 was not intended.)

In the above, it sounds like you're suggesting that we introduce a new, higher-level kind of parameter, which is attached to nested list dimensions, rather than layout nodes. That is a good idea: the need for it keeps coming up. I think that the higher-level thing can be implemented in terms of parameters, however—I don't think we want to confuse things by having two mechanisms side by side (prompting a developer to ask, "Which should I use?"). It could be a naming convention: all parameters that begin with "dim:" should stick to list dimensions, perhaps?

[agoose77]

it sounds like you're suggesting that we introduce a new, higher-level kind of parameter

At this stage, I was even thinking of not doing this, but rather defining a mechanism to search for parameters in a type-dependent way. I.e., not just dimensional, but sensitive to options too. So, instead of special property names, this type-sensitivity becomes a property of the caller, who must use layout.type_parameter(). As such, we'd want our standard parameter merging logic to behave such that these type-aware queries always resolve correctly. That might not be possible, though — I have not considered all of the ways that we create new layouts.

[jpivarski]

Okay, it's a higher-level concept than parameters, but it's not baked into the parameters themselves as a naming convention; it's a matter of how they're requested. You already mentioned it, but that makes it like purelist_parameter, a road we've already started down.

[agoose77]

Yes, that's an idea. Note, it's not the only idea. Your suggestion places the information about the parameter scope within the parameter itself, which is interesting.