Combine the concepts of pins/labels and modified lookups

[michaelhkay]

We have two rather separate mechanisms, both designed to solve aspects of what is essentially the same problem: lookup expressions lose too much information.

Pinning tries to solve the problem by saying that if the origin of the lookup is pinned, then the results of the lookup carry a label containing information about the key and the parent.

Modifiers like pair::* try to solve the problem by returning a map containing the key and the value as separate fields.

But pinning only solves part of the problem, in particular it doesn't prevent X?* flattening the result, and the pairs modifier only solves part of the problem, in particular it doesn't retain parentage.

I would like to try combining them and trying to create a mechanism that is better than either. I don't yet know exactly how this might work, but I'm thinking along the lines:

• Replace the concept of labelled items with labelled values (that is, a label can be attached to any value, not just an item)
• Scrap pin() as an explicit function
• A lookup expression like $X ? child::Y returns a sequence of labelled values (no flattening)
• The properties of a labelled value include:
  ** target - the actual value
  ** key - the associated key (or array index)
  ** parent - the containing map or array
• These properties might be made available through syntax such as $LV ? target::* , $LV ? key::*, $LV ? parent::* (or otherwise)
• ancestor and ancestor-or-self can be made available as derived properties
• Many operations when given a labelled value should automatically operate on its target, and ignore the label (rather like atomisation). Exactly which operations do this is an interesting question to which I don't yet know the answer. It's tricky because child::* returns a sequence of labelled values, and we want to be able to manipulate this in unflattened form. Perhaps child::* should instead return an array of labelled values? But then you end up with another lookup operation to extract the members of this array.

[ChristianGruen]

Good to see this discussion evolve. I must admit we haven't implemented fn:pin and fn:label yet because we are not convinced yet that it is the best path to go. We did something similar for XQFT in the past, for scores that got attached to boolean values, and positional data attached to nodes, and it was always unsatisfactory that scores may get lost at some stage in more complex operations, without the user noticing it. For simple traversals, it did its job, but everything gets more sophisticated after a while.

I like the idea of a child modifer: It makes explicit that something more is going on. Maybe, instead of labeling a value, we could return a record(key, value, ancestors) value and make the “labels” explicit:

{ 'a': { 'b': 2 } }?child::a

→ { 'key': 'a', 'value': { 'b': 2 }, 'ancestors': { 'a': { 'b': 2 } } }

If a lookup operator is executed on such a record, it would be executed on its value entry:

{ 'a': { 'b': 2 } }?child::a?child::b

→ { 'key': 'b', 'value': 2, 'ancestors': ({ 'b': 2 }, { 'a': { 'b': 2 } }) }

Similarly, a parent modifier could be defined…

{ 'a': { 'b': 2 } }?child::a?child::b?parent::a

→ { 'key': 'a', 'value': 1, 'ancestors': { 'a': { 'b': 2 } } }

…maybe also ancestor, self and so on.

For update operations, it would be simple to demand that axis modifers need to be used – and it would be great if the same functions or expressions could be used for nodes (#1702) and maps/arrays:

fn:delete($node, fn { ./child::a })
fn:delete($map , fn { .?child::a })

[michaelhkay]

Over at #1786, I was looking at the question of how to replicate the XSLT key definition

<xsl:key name="classKey" match="class | interface" use="ancestor::module/@package || '.' || string-join(ancestor-or-self::*[self::class|self::interface]/@name, '.')"/>
when the input is JSON rather than XML.

Let's suppose that $P?child::* applied to a map or array returns a sequence of JNode objects, each having as primitive properties a key, a value, and a parent (=$P), and we'll worry about where this fits into the type system later.

We'll provisionally use the notation $N¶key, $N¶value, $N¶parent to refer to these properties of a JNode $N, and we'll worry about concrete syntax later.

The existing 3.1 construct $P?KS is defined to mean ($P?child::KS) ! ¶value. Note the flattening.

More specifically we could say that if $P is a JNode, then the parent property of $P?child::* is $P, while if $P is a simple map or array, then the parent property of $P?child::* is a JNode with ¶key absent, ¶value equal to $P, and ¶parent absent. A JNode with absent key and parent is referred to as a root JNode.

Now we can use the syntax $N?parent::* to get the parent of a JNode.

And we can define $N?ancestor::* as the transitive closure of parent::*, and $N?descendant::* as the transitive closure of child::*. (Forget ?? for the moment, that's new in 4.0 so we have a free choice).

Looking at the XSLT key definition above, if we simplify it to

<xsl:key name="classKey" match="class" use="ancestor::module/@package || '.' || string-join(ancestor-or-self::class/@name, '.')"/>
we could now say that the equivalent with JSON is

map:build(?descendant::class,
  fn{?ancestor::module?package || '.' || string-join(?ancestor-or-self::class?name, '.')})

which is much better than any of the solutions I came up with in #1786.

To handle the original (class|interface) union we can use the existing key specifier ?descendant::("class", "interface"), or we could try to come up with better syntax. Ideally syntax that extends to selecting * except class.

This approach looks very promising, and it raises follow-up questions

• what kind of thing is a JNode, and where does it fit in the type hierarchy?
• what operations are available on JNodes, other than axis navigation as described above?
• what syntax is used to access the properties of a JNode?

I'm coming to the thought that modelling a JNode as a record(key, value, parent) is not ideal, because many operations (such as atomisation) on a JNode $J should delegate to perform the corresponding operation on $J¶value. In particular, a lookup operation on a JNode should operate on its ¶value, which means some syntax other than ? is needed to get the JNode's properties -- not necessarily new syntax, it could potentially be simple functions such as get-key(JNode) and get-value(JNode). The alternative of modelling it as a value with a semi-hidden label isn't ideal either, because it's not clear which operations retain the label and which lose it.

One possibly crazy idea is to make a JNode a kind of Node, allowing the use of the familiar / operator for navigation.

Another possibility is to make a JNode a a subtype of function() as item()*, with $J() returning the ¶value property.

[ChristianGruen]

Interesting thoughts. – If we extended the type system, we would certainly have more freedom.

many operations (such as atomisation) on a JNode $J should delegate to perform the corresponding operation on $J¶value

In principle, we could extend the atomization concept to key/value records:

(: would always return 6 :)
[ (1, 2, 3) ] => sum()
[ (1, 2, 3) ] => array:members() => sum()
{ 'key': 'numbers', 'value': (1, 2, 3) }) => sum()
{ 'numbers': (1, 2, 3) } => map:pairs() => sum()

But I agree that atomization is just one of many aspects that need to be understood. The implicit semantics that current exist for labeled items would still be present, and just transferred to other operations, and we may end up observing that an explicit data structure is like a crutch for something that should be supported on a lower level.

[michaelhkay]

Sketch proposal.

In the data model, drop labeled items and introduce J-Nodes.

An item is either a G-Node (generalised node), an atomic item, or a function item. A GNode is either an X-Node (XML node) or a J-Node (json node). For continuity, the term "node" unless otherwise stated means XNode, and the item type node() matches an X-Node. New item types g-node() and j-node() are introduced.

All G-Nodes have node identity and document order, and the operators is ,<<, >>, union, intersect, and except are promoted to apply to G-Nodes.

A J-Node has properties ¶selector, ¶value, and ¶parent.

A J-Node is either a root or a non-root. In a root, the ¶selector and ¶parent properties are both absent. In a non-root, both properties are present.

The operation of getting the ¶value property of a J-Node is called unwrapping, and can be performed explicitly using the function unwrap(). The coercion rules are changed so that when a J-Node is used and the required type does not allow a J-Node, then it is unwrapped (this may be followed by other coercions such as atomisation). We need to decide whether EBV applied to a J-Node should always return true or should unwrap: there are arguments either way.

A root J-Node wrapping a map or array can be created using the function j-node((map(*)|array(*)). Normally this will be done implicitly. If a map or array is supplied on the LHS of the lookup operator ?, it is automatically wrapped as a root J-Node. Like X-Node constructors, the function creates a new instance with distinct identity; though we can make it implementation-defined whether calling the function twice on "the same" map or array gives you two new J-Nodes.

Non-root J-Nodes are created by navigation from another J-Node using J-Axis steps. The child J-Axis step applied to a J-Node A whose ¶value is an array returns a sequence of J-Nodes, one for each member of the array, with ¶value set to the member, ¶parent set to A, and ¶selector set to the index in the array. Similarly for maps, but the ¶selector is the key value. The parent J-Axis returns the value of the ¶parent property; other J-axes such as descendant and ancestor are defined in terms of these primitives.

The lookup expression A?axis::KS filters the result of A?axis::* to return those J-Nodes that satisfy the key selector. If A is a map or array rather than a J-Node then it is first wrapped as a J-Node. A?KS is an abbreviation for A?child::KS =!> unwrap(). A??KS is an abbreviation for A?descendant::KS =!> unwrap().

The function selector(j-node()) returns the selector (map key or array index) of a J-Node.

[ChristianGruen]

…a proposal which would result in rather fundamental changes. Perhaps we could discuss first when we want to finalize 4.0… Would 2026 be realistic?

Now that maps are ordered, it would indeed be possible to define node oder (is, <<, >>) also for map entries.

[michaelhkay]

when we want to finalize 4.0

As soon as we have a specification that works.

I conducted the case study at #1786 (and will be reporting on the results at Markup UK) as a usability test to see whether the features we have defined in 4.0 meet the aim of making transformations on JSON work as well as transformations on XML. The result show that we still have work to do.

[michaelhkay]

Closed by adoption of the JNodes proposal.