2025 JNodes

[michaelhkay]

Fix #2025

This is a first draft for review.

It includes changes to the data model, functions and operators, and XQuery/XPath. It does not yet include changes to XSLT.

It's a big proposal, but I think it removes more complexity from the spec than it adds. It's basically a unification of two concepts, both of which were addressing aspects of the same problem, namely that lookup expressions lose too much information. It gets rid of the pin/label mechanism, and modifiers on lookup expressions, and introduces JNodes and JAxes in their place. (Any suggestions for improved terminology are more than welcome.)

I think we get a lot more "bangs for the buck" with this solution, and it makes navigation of JSON trees work in a much closer way to familiar navigation of XML trees. It needs a lot more work on examples and explanation, of course.

[ChristianGruen]

Impressive, and a lot to digest. This review only contains some editorial notes.

[ChristianGruen]

Maybe:

Suggested change

	return pin($data) ? descendant::languages[. = 'German'] ? parent::* ?capital)[1]</eg></fos:expression>
	return pin($data) ? descendant::languages[. = 'German'] ? parent::* ?capital[1]</eg></fos:expression>

[ChristianGruen]

Wouldn't fn:data do the same?

[michaelhkay]

Thanks, @ChristianGruen . I have incorporated these comments and have made some other improvements to the spec.

[Arithmeticus]

Yes! This is exactly the sort of navigation reform I was hoping for in 4.0. This gets us to a place where XPath treats JSON with a consistent, intuitive syntax comparable to what it provides for XML. It is a pity that the term "JPath" is already taken, because that could become an alias for XPath. JXPath?

I find the use of GNode, JNode, and XNode intuitive.

I fear that the reform will have a lot of knock-on effects that will emerge in the course of writing robust tests.

I also think that the change is so significant, we should take more than one meeting. The implications need to be fathomed.

A couple of more general questions that cannot be attached to specific code:

• Is the JNode where we would place other new constructs such as set?
• How does the concept of JNode correlate to the concept of map member (KVPs)? And does this need to be mentioned in the specs?

[Arithmeticus]

Readers coming from 3.0 may wonder how maps could have document order, and they may need a gentle nudge here, or at point 4 below, about the shift in default map entry ordering

[Arithmeticus]

Along with this, today's presentation indicated that axis order and document order are different for JNodes. More about that needs to be said.

[Arithmeticus]

I do not understand how a root JNode will always have an empty ¶value. How then does it encapsulate the branches of the tree? Or is ¶selector meant? In fact, it seems that ¶selector has to be included in this list.

[Arithmeticus]

Can "a sequence" be further detailed to "a singleton sequence that is an array or a may" or something like that? As currently expressed phrase feeds the imagination too much.

[Arithmeticus]

Is it possible for the value to be null/the empty sequence? If so, does the property exist for that JNode?

[Arithmeticus]

Should this read: <fos:arg name="input" type="JNode(map(*)|array(*))"/>?

Would/does JNode(*) or JNode() (is there a difference?) imply JNode(map(*)|array(*))?

[Arithmeticus]

This question shows my curiosity/ignorance about what it would mean for a JNode to have a value that is not an array or map. We need more on that topic.

[michaelhkay]

If you have an array of maps [{'a':2, 'b':3}, {'a':12, 'b':13}] and pin the array, then the root JNode wraps an array (it is of type JNode(array(map(string, integer))). When you apply child::* to this you get two JNodes that wrap maps. When you apply child::* to these maps, you get 4 leaf JNodes that wrap integers (these are of type JNode(xs:integer); they have a parent property that is the map, a selector property that is the map key ("a" or "b"), and a value property that is the xs:integer. Applying child::* to the leaf JNodes returns an empty sequence; they are like the text nodes in an XTree.

[Arithmeticus]

The two general questions I concluded my last comment are illustrative of how I didn't fully understand the concept. Nevertheless I hope they are useful in revising the PR, to prevent future readers from meandering that direction.

[michaelhkay]

JNodes.pdf

Attached is the slideshow I presented at the QT4CG meeting today. LibreOffice presentation converted to PDF.

[Arithmeticus]

I understand why the mechanics of / prevent us from using this as the operator, and why we need ?. Is it possible, though, to do the reverse, and allow the LHS of the ? to be an XNode? Redefine the ? such that if its LHS is an XNode, treat it as an alias for /?

Some users may find that mixing both operators is confusing. If they cannot use / universally, they may settle for using ? universally. Hmm. Not sure if that's a good idea. But I think I should air it.

[michaelhkay]

The main thing that prevents us using the same operator for "/" and "?" is that the syntax on the RHS is different. Neither interprets the RHS as a general expression, and as well as having different syntax, the semantics is also subtly different. There's a superficial similarity between $elem/X and $map?X, but $elem/$X and $map?$X have completely different meanings.

[Arithmeticus]

In the current conceptualization, there is an inconsistency worth discussing. Every GNode is either an XNode or a JNode. A JNode is a kind of wrapper for an XDM value, and it does not have subsets. A GNode, on the other hand, is not anything in and of itself, but is a taxonomic category under which the traditional nodes fall.

I came to the text under the impression that a JNode would behave like a node/XNode and bifurcate into different types, i.e., a map or an array. (You'll see this mistaken impression informs some of my earlier comments and questions.) But really, a JNode is more analogous to each of the seven other nodes, in that it is a (metaphorical) concrete something. Why not nudge the JNode into an eight type of node? Wouldn't specifying how all the dm:* accessor functions would treat a JNode be a good thing?

If we retain the current taxonomic structure, we need to make it quite clear that XNode is a taxonomic category whereas JNode is a node-like construct. It still doesn't feel quite right to me.

[michaelhkay]

I have realised that there is a problem in the case where an entry within a map or array is a sequence that contains several maps and/or arrays. This doesn't arise from JSON (a value in a JSON object or array is always a sequence of length 0 or 1) but it can arise in the general case. The specification of the child::* axis says that it processes each of the maps and arrays and forms the sequence concatenation, but this means you can have several children with the same ¶selector and ¶parent. I think the implication is that we need the JNode to contain an additional property, ¶position, holding the position of the child within this sequence: it is the combination of (parent, position, selector) that will be unique. In maps derived from JSON, ¶position will always be 1.

This raises the question of what child::* should return when applied to a JNode that wraps a sequence of several arrays (or maps), for example pin(([10,20], [30,40])). As currently defined, with the addition of the ¶position property, it returns a sequence of 4 JNodes having:

• ¶position=1, ¶selector=1, ¶value=10
• ¶position=1, ¶selector=2, ¶value=20
• ¶position=2, ¶selector=1, ¶value=30
• ¶position=2, ¶selector=2, ¶value=40

I think that's probably adequate, since this is an edge case; there is no loss of information and the original structure can be reconstituted using grouping. Having child::* return a more complex structure would penalise the user doing something relatively simple (i.e., processing JSON).

Note also, with this input, $input?child::2 would return all the entries with ¶selector=2, that is,

• ¶position=1, ¶selector=2, ¶value=20
• ¶position=2, ¶selector=2, ¶value=40

The other thing that's not explicit here is that ¶selector=1 is an index into an array rather than a map. But this is discoverable by going up to the parent, extracting the value, and accessing the item at position ¶position within the value.

[ChristianGruen]

I think we will need to take duplicate removal serious, in particular for the parent axis, and to avoid calls to fn:distinct-ordered-nodes, which hardly anyone would understand. This may force us to have more thoughts on JNode identities: It may not be possible to reuse identities.

Restoring order could be simplified by adding an (invisible?) ¶index property. For arrays, it could be identical to the ¶selector property. Implementations can possibly avoid storing the ¶selector property and retrieve its value from the map via the ¶index property.

I think that's probably adequate, since this is an edge case; there is no loss of information and the original structure can be reconstituted using grouping.

The ¶position approach looks feasible. I imagine it won’t be just an edge case, but a rather common case in practice once the feature is used more extensively (I would regard JSON to be only one use case of many).

The other thing that's not explicit here is that ¶selector=1 is an index into an array rather than a map. But this is discoverable by going up to the parent, extracting the value, and accessing the item at position ¶position within the value.

We have several open issues on naming map/array properties and functions, and it would be great if we could settle on a common terminology. Maybe it would simplify various upcoming JNode challenges.

Selectors would be yet another new concept that people would need to grasp, so instead of ¶selector, I hope we can have ¶key properties (related: #1871, where I motivated the addition of an array:keys function).

[Arithmeticus]

As I reflect on the current draft, I think of a JNode as a kind of handle for a single map, a single map KVP-entry, a single array, or a single array member. Put another way, every time one creates a map or an array, one creates one JNode for that map/array, and one JNode for each KVP-entry/member. If my reasoning is off, it would be good to think of revisions that will prevent others from straying down this path. If it is on, perhaps the prose can be sharpened to make this association more explicit.

If a given JNode's selector is 1, is there any way to know, without traversal to the parent, whether the JNode corresponds to an array member or a map member? Responding to myself: does it matter? Why would I ever get into a situation where I wouldn't know and would care? I respond: I don't know.

[michaelhkay]

I think of a JNode as a kind of handle for a single map, a single map KVP-entry, a single array, or a single array member.

Yes, that's a good mental model. It's a little bit more complicated in the case where an entry in a map or array is a sequence with multiple items (and it's possible that case might need further work).

If a given JNode's selector is 1, is there any way to know, without traversal to the parent, whether the JNode corresponds to an array member or a map member?

No, you have to test what kind of parent you have got. I thought about this but didn't feel there was a strong enough case to provide a convenience method for this: it can always be added.

[ndw]

Supersceded by #2083