Order of lists when merging across schemas

[bruno-f-cruz]

When considering composition across multiple schemas, some fields, such as stimulus_epochs, are concatenated into a final list.

I am wondering what guarantees we have about the order of these arrays. For instance, if we have a given code.version we want to search via docdb, in the behavior component of the metadata, serialized it as position "0" in the array. Still, after merge, this could easily be position 1 (eg: merged with fip, where fip goes first).

What is the recommended pattern to deal with this?

• Is there a query that is robust to these issues?
• Is there an informal rule on how these lists get merged? Looking at the code it doesnt look like it but maybe something in the metadata gather step?

Thanks,
Bruno

[bruno-f-cruz]

@dbirman @dougollerenshaw any idea?

[dbirman]

Sorry that this didn't get a faster response. Heads up that I don't track the discussions and I've asked several times to close this whole section since Saskia/David aren't responsive here. I'd prefer you make an issue for discussions!

There's no guarantees on merge order. We can certainly add some guarantees though and I've thought about that, the simplest thing is just to guarantee order by start time although it's also easy to re-order the lists accordingly. Are there other guarantees you would want to see?

[bruno-f-cruz]

Makes sense.

I am mostly just wondering how to fetch data via queries if i do not know the order a priori. But I am also not sure what heuristic would work since, when i give you a partial and data-schema instance, there are no guarantees of "modality" that would allow you to say: "Behavior is always first". I will try to see if i can fix this at the query level.

[dbirman]

You can do unordered list queries, e.g. this will find assets that have behavior in the modalities list: {"data_description.modalities.abbreviation": "behavior"}

[bruno-f-cruz]

Is there a way to do it nested? e.g: find me sessions where ONE OF the streams has software version == 42.42.42

[dbirman]

I think in our setup you have to do an aggregation pipeline to check if exactly one stream matches a field. The query I put above would return records where at least one of the modalities matches but it doesn't guarantee that there's one specific match.