linkml-arrays chat

[sneakers-the-rat]

Wanted to make a general discussion thread for linkml-arrays. as a meta-thread, let's try and use the in-post replies as much as possible to group conversations <3

plz edit this root post/propose edits to the root post as relevant.

Status

NDArrays

• metamodel syntax declared: First pass at native NDArray support. linkml-model#181
• generators
  • pydantic: list of lists - [pydanticgen] Add list of lists array implementation #1887
  • pydantic: nptyping - (@sneakers-the-rat will do)
  • json schema: Add support for linkml:array in jsonschemagen #1889
  • RDF/Shacl/ShEx: decide on strategy for serializing NDArrays as RDF and validating using shacl/shex #1890

Labeled Arrays

• metamodel syntax declared: (still WIP i think?? linkml-arrays has the current draft
• generators
  • pydantic: list of lists
  • pydantic: nptyping
  • ...

References

repo: https://github.com/linkml/linkml-arrays/

example data, particularly for the labeled arrays spec

.. (lmk what else)

[sneakers-the-rat]

@rly i had this thought and wanted to jot it down rl quick and realized we didn't have a public meta-discussion place and so made this. ok so part of what i have been trying to puzzle about with the labeled arrays spec is the sort of 'unseatedness' of the label axes - that it is good that they can be declared independently, but it still feels like it would be an implementation and authorship challenge to keep track of the labeling/index dimensions as basically parallel slots/attrs to the main array slot.

in trying to refactor the pydantic generator i got to thinking how nice it would be if we could just have a 1:1 map between metamodel properties and generator methods - if a slot has this prop, mutate the SlotDefinition in this way, and so on. the reason that's hard is because the relationship the props isn't/is hard to schematize - "range changes meaning when array is also defined."

I think that these problems can be seen as the same problem - It's difficult to schematize "horizontal" relationships between things at the same level. Since linkml is recursively defined, where the metamodel is in the syntax of the schema, could the same syntax be used to schematize relationships between props as eg. relationship of the indexing dimensions to the array?

just wanted to jot the thought down quickly without doing a full investigation of eg. existing ways it's possible to describe horizontal dependencies, but something as simple as a syntax like

array:
  modifies:
    - range

as a generic indication that "other things about this property range are subsumed by this property array, which is a superset of it." so in the context of a generator, you would skip the normal computation of the range and just use the array method

or if it's specified with a dict, the value specifies the way that the interaction happens

array:
  modifies:
    range: after

so "array" is to be handled "after" the usual range computation - do everything else you would do normally, and then this one. so in the case of a generator, you would compute the range as normal, and then pass it in to the array method.

and then the parameterized form (to complete the usual trirumvirate of array, Dict[str,str] and Dict[str,dict]

array:
  modifies:
    range:
      type: after
      param1: whatever

so eg. for generators, if a parameter had to specify it came in a specific order in a processing chain, etc.

To know when a keyword is applicable, it could use the usual metamodel way of defining itself

slots:
  after:
    is_a: linkml:modifier
    domain:
      - SlotExpression
      - SlotDefinition

and so then with the case of the labeled arrays you could have something like

MyClass:
  attributes:
    values:
      range: int
      array: 
      
    index1:
      range: TemperatureType
      modifies:
        values: indexes
    
    # or
    index2:
      range: LatitudeType
      modifies:
        values:
          type: indexes
          dimension: 0
          curie: # ...

[sneakers-the-rat]

I would expect to be able to address indexing dimensions by their alias as well. I think this is related to the more general namespacing discussion - I should be able to say

indexed_by: 
 .dim1: slot1
 .dim2: [slot2, ClassName.attribute3]

Etc. And generally reuse terms (or at least Definitions and definition expressions).

[rly]

re: indexed_by enthusiasm. Great! Let's move forward with that.

re: inverse, interesting idea. Yeah, I agree that is a can of beans and is something that we can think about and play with for now.

re: indexing by alias. I agree. That should be allowed.

[rly]

Just to add, the challenge with this specific syntax:

indexed_by: 
 .dim1: slot1
 .dim2: [slot2, ClassName.attribute3]

Is the 2D coordinate case, e.g, we would want to say something like

indexed_by:
  - [x, y]: latitude_in_deg
  - [x, y]: longitude_in_deg

Which is why we decided to go with the dict/object approach. This is also a little more flexible because we can add properties like an alias to the coordinate/index. We could still make use of namespaces and aliases as opposed to indices:

indexed_by:
  - alias: lat
    index_slot: TemperatureDataset.latitude_in_deg
    index_dims: ["x", "y"]

But I'm not sure we want to allow linking to slots not included within this class, and so namespaces may not be useful. What would be the use case for that?

[rly]

One more thing to consider: In Xarray, Pandas, and databases, "index" means a data structure optimized for efficient data selection and alignment, which is different from how we use it in the example above. https://docs.xarray.dev/en/stable/user-guide/terminology.html#term-Index. (Correction: In Pandas, the index of a DataFrame is a series of labels that identify each row, which is more aligned to our use here.) Using "indexed_by" in this context might 1) be a little confusing and 2) make it harder to schematize those kinds of indices in the future.

Should we use the term "coordinates" (which is used by NetCDF and xarray and is pretty well aligned to the definition of "coordinate"), or "coords" instead of "indexed_by"?

An example:

coordinates:
  - alias: lat
    coord_slot: latitude_in_deg
    coord_dims: ["x", "y"]

[sneakers-the-rat]

Will need to give this more thought when back at keyboard, but for now, yes lets think about terminology, good point. I think index may still fit, thinking about RDBs where its got a very specific meaning, and whenever we are making an index relationship between a slot and an array we would probably want to make an RDB index too, so maybe we want to use that term? I am worried about coordinates being a sort of loose semantic match to the function of indexes and also making a sort of crowded space with dimensions, but like I said let me think more about it when im back at work and can more carefully consider the prior art - also agreed matching existing tool terminology is good

[sneakers-the-rat]

First array representation just dropped, u can use it today!

Guide: https://linkml.io/linkml/schemas/arrays.html
Docs: https://linkml.io/linkml/generators/pydantic.html#arrays

this builds on the broader refresh of the pydantic generator and its templating system: https://linkml.io/linkml/generators/pydantic.html#templates

more 2 come <3

[rly]

On the topic of how to define an array slot within a DataArray class as an array class instead of as a slot, so that one can

1. reuse the class in multiple DataArray objects (e.g., you may want to store an array of a million timestamps once and use it to label multiple other arrays)
2. associate attributes / metadata to the array
3. extend the class in the schema

@cmungall and I thought about creating a simple, degenerate Array class that follows a particular pattern where exactly one attribute is an array. These classes would be identified using implements: linkml:Array (for now at least). I mocked that up in this example here:
https://github.com/linkml/linkml-model/pull/190/files#diff-84cb9e7e732cf37a327db45b4a6e1098fc0da6c6c423f6145c812918623abfb9

This raises an issue where it does not really make sense to put indexed_by on the TemperatureMatrix class because it does not know about / cannot access the indexes. So I put indexed_by on the temperatures_in_K attribute under the array key in the higher-level DataArray. This is kind of an odd usage of the array key. Perhaps it should be array_usage or something similar? (Here is a case where a modifies key might be useful.)

Open to thoughts and ideas!

[sneakers-the-rat]

since array currently is a SlotDefinition property, could we just also extend its domain to include ClassDefinition ?

classes:
  MyClass:
    array: # ...

[rly]

In that case, where would we specify other SlotDefinition properties like:

range: float
unit:
  ucum_code: deg
minimum_value: 0

[sneakers-the-rat]

Hmm hmm hmm.

Yes this is sort of what I was thinking about with that "can the implements keyword be a sort of "level shifter" half-idea. So eg. Say all the terms that modify an array are in some subset "array". then by doing implements: array what we are doing are bringing all those terms into scope in the containing context.

Trying to do 2 things: avoid special cases like one special dimension, and also give some reusable intuition/uniform implementation for what "implements" does. I def see the tension of a slot-only array and the want to also have a class-level array, so yes we need some way of doing this

[rly]

Something to follow but is not totally relevant to this effort is the Python Array API standard: https://data-apis.org/array-api/latest/index.html

Also potentially relevant is ZEP0004 on Zarr Conventions: zarr-developers/zarr-specs#262

[sneakers-the-rat]

ooo very interesting. i have some things to add to this esp re the typing section :)

[sneakers-the-rat]

oh boy that's heavy https://data-apis.org/array-api/latest/API_specification/index.html

[sneakers-the-rat]

Important difference between "api" and "interchange format" or "specification," lots of useful stuff there

[rly]

Also adding a note from @melonora that this package/effort may be useful https://github.com/loco-philippe/ntv-numpy

[melonora]

Question, I am not certain whether we actually cover this particular use case even with labeled arrays. Particularly when describing transformations you have a transformation matrix in which you have particular labels. For example in ome with the affine transform:

At the moment though with labeled arrays you could have a matrix being labeled, you can't necessarily add a description for an individual value, am I right? If so I think there are other use cases for this that would warrant a try for supporting it.

[sneakers-the-rat]

I think since its a linked data format we could do something like this at the schema level rather than the metamodel level, what do ya think?

(In pseudocode for illustration purposes)

classes:
  Affine:
    attributes:
      target_array:
        range: ArrayExpression
      transform:
        array:

[melonora]

sorry for not responding earlier. I am not certain whether we are talking about the same thing. I mean to describe what each value of the transformation matrix means, the matrix when you have yx instead of xy looks different for example from the affine matrix when having xy.

In any case with the transforms, usually the coordinate space is named and then transforms are described for how to get data in a particular coordinate system.

[sneakers-the-rat]

oh sorry, I thought you meant like 'can we have an additional transformation array describe an existing labeled array' . rather than 'can we label individual values in an array.'

no we haven't covered that yet, but i do see the case you're talking about here with a specific stereotyped array where each element is meaningful (as opposed to arrays as we have been thinking about them where it's the dimension that's the meaningful thing). In the past we had decided not to put things like column/row major encoding into the top-level schema and treat that as an implementation detail, and that sort of seems like the same thing here: like the xy also needs to match the ordering of the array it's being applied to, etc. and we get into a situation where the schema specification is overstrict. like this would be something i would probably lean towards being at a schema rather than metamodel level, but i would want to try and implement it and see if it is necessary to be in the metamodel because i also see what you're saying.

[melonora]

Yeah I agree this should be more defined at the schema level, but it would be good to test whether the metamodel allows for specifying this at the schema level. Main reason is that different kinds of software define the affine transformations in different ways, so if you are able to provide a schema on how your software defines it, it can make the ecosystem more interoperable:) I am not certain whether there are other usecases like this but I just wasn't certain whether the current metamodel allows to specify this in a schema.

[sneakers-the-rat]

Could an array range be a type that has a label annotation? Just spitballing

[melonora]

hmm think with the next session we could have a small discussion. Optionally we could also define a labeled array which consists out of a collection of elements which have indices and also individually can have a label and description, which I think is slightly different from the other labeled array implementation.

[sneakers-the-rat]

@melonora see nptyping's handling of explicitly labeled array elements: https://github.com/ramonhagenaars/nptyping/blob/master/USERDOCS.md#dimension-breakdowns

so something like

myslot:
  array:
    dimensions:
    - alias: first_dim
      items: [x, y]
    - alias: second_dim
      items: [a, b]

for when one wants to label the entire nth element of that whole dimension, ie.

		second_dim
		a	b
first_dim	x	(x, a) = 0	(x, b) = 2
	y	(y, a) = 1	(y, b) = 3

where the value 0 is labeled as x in the first dimension and y in the second dimension. That would imply that exact_cardinality: len(items) and should probably be mutually exclusive.

or like

myslot:
  array:
    dimensions:
      - alias: first_dim
        exact_cardinality: 2
      - alias: second_dim
        exact_cardinality: 2
    items: [[a, b], [x, y]]

for when you want to label each item in an array independently. That would mean that the shape of items would have to match that of all the specified dimensions, and i don't want to also be able to switch row v. column major in the items specification so i would be in favor of just picking one. that is also sort of annoying where you might just want to annotate a single item, but idk if i also want to support yet another syntax like

items:
  - first_dim: 0
    second_dim: 1
    item: x
  - first_dim: 1
    second_dim: 1
    item: y

and so on.

Ideally it would be cool if instead of just strings we could specify classes/slots so that the item annotations can have their own metadata like a linked data URI or whatnot, but then we get into cool recursion territory where an annotation has an array which has item annotations...

[sneakers-the-rat]

Returning to work partially, pushed an update to numpydantic that takes over control of the shape syntax (part of the longer term plan to get rid of nptyping and make a cleaner, more inspectable syntax) and adds shape ranges so annotations dont have to be so verbose:
https://numpydantic.readthedocs.io/en/latest/syntax.html#ranges
https://github.com/p2p-ld/numpydantic/releases/tag/v1.2.0
p2p-ld/numpydantic@07ab3d1

Should be able to pull the numpydantic generator early next week, really not much work at all since a lot of the work is done in the npd package. Then with that linkml arrays become extremely implementation general and ready to use in normal contexts where ppl are usually using array libraries instead of lists of lists.

Some fun stuff in there like direct support for videos, and will be standing by to add in support for labeled arrays and an interface to xarray.

[sneakers-the-rat]

What is up arrays chat we got ourselves a new array representation and its a hoot and a half- linkml about to be able to hook into whatever the heck array library you want

#2178