Extend FLWOR expressions to maps

[rhdunn]

Edit: Current proposal (#31 (comment)):

for key $key in ...
return …

for value $value in ...
return …

for key $key value $value in ...
return …

---

With the addition of the for member syntax for arrays, it is possible to use a ForExpr/FLWORExpr to enumerate the contents of sequences and arrays, but not maps. In order to be consistent and symmetric across these types, the for member syntax should be extended to support maps by enumerating the key/value entries of the map.

Given a map of type map(K, V) the member RecordTest would be record(key as K, value as V). Given a map of type map(*), the member RecordTest would be record(key, value).

This would allow a user to write expressions like:

for member $entry in $map
return element { $entry?key } { $entry?value }

NOTE: With the addition of the array:values and map:entries functions in issue #29, it is possible to avoid the need of the for member syntax for arrays and maps, but may be worth keeping for people who prefer the wordy style of the XPath/XQuery syntax.

[joewiz]

I like this, but shouldn't it be for entry? Arrays have members, and maps have entries. From XDM:

An array is an ordered list of values; these values are called the members of the array.

XDM doesn't explicitly define entry, but it's used exclusively in FO, e.g.:

The function map:entry returns a map which contains a single entry. The key of the entry in the new map is $key, and its associated value is $value.

[benibela]

It could also be done without the temporary map. For example:

for key $key in $map return ...

for value $value in $map return ...

for key $key value $value in $map return ...

for value $value at $key in $map return ...

[rhdunn]

I would be happy with for entry ... for maps, so ForExpr/FLWORExpr would support:

1. for ... for sequences (existing);
2. for member ... for arrays (new in the draft spec);
3. for entry ... for maps (from this proposal).

[michaelhkay]

I'm not entirely convinced. Mainly because "entries" in a map aren't values in the data model; also because maps are unordered, and also because unlike processing arrays, it's not very difficult or inconvenient to do it using "for $k in map:keys($m), $v in $m($k) return....

[dnovatchev]

@michaelhkay commented:
"... I'm not entirely convinced. Mainly because "entries" in a map aren't values in the data model; also because maps are unordered,"

It is high time that we come up with a set type in XPath. We actually have to deal all the time with sets (not just node-sets, but sets of any-type values), and it is painful to read in the spec how two maps are compared for equality when explaing fn:deep-equal():

"If $i1 and $i2 are both ·maps·, the result is true if and only if all the following conditions apply:

Both maps have the same number of entries.

For every entry in the first map, there is an entry in the second map that:

has the ·same key· (note that the collation is not used when comparing keys), and

has the same associated value (compared using the fn:deep-equal function, under the collation supplied in the original call to fn:deep-equal)."

When if we had the set type the above would simply say:

"If $i1 and $i2 are both ·maps·, the result is true if and only if the sets of their keys are equal, and the corresponding values for each key in the two maps are deep-equal."

I propose that starting with XPath 4.0 we introduce the set type and define set equality, the union ( | ), intersection (intersect) and set difference (except) not only for node-sets but for sets of any-typed values.

Then we can have a function: to-set($collection as item()*) as set, which would produce a set (of the distinct values) of any collection-typed argument supplied to it: sequence(its distinct values) , map (a set of its entries), array (a set of its members).

This makes fn:distinct-values() almost unnecessary.

We will no longer have to explain in a "Remarks" section that the result of a function is "unordered" or that its order is "implementation-defined" -- just by making this function return a set.

How can almost all major programming languages (not even speaking of SQL), such as C#, Python and Java can have a set data type / interface, but even in XPath version 4 we still have to describe it in a free language narrative?

Thanks,
Dimitre

[michaelhkay]

From a technical point of view, I concur. However, I want to restrict the scope of what we're attempting in this version: the reason the last version took 10 years to complete is that we were too ambitious. The most expensive changes to make are those that change the data model, and I think we should strongly resist doing that.

[dnovatchev]

From a technical point of view, I concur. However, I want to restrict the scope of what we're attempting in this version: the reason the last version took 10 years to complete is that we were too ambitious. The most expensive changes to make are those that change the data model, and I think we should strongly resist doing that.

Let's see what other people have to say.

Should the language still be incomplete and encumbered, or can we do something good here?

[michaelhkay]

What we could contemplate, however, is a library of functions for manipulating sets of atomic values, using a map as the underlying representation. For example

set:of("a", "b", "c")
set:union($set1, $set2)
set:intersection($set1, $set2)

Going beyond sets of atomic values to sets of arbitrary values immediately gets you into the problem of defining equality between arbitrary values, which is a quagmire.

[ChristianGruen]

Let's see what other people have to say.

I completely agree with Michael Kay here. Smaller steps to start with won’t prevent us from doing bigger steps later on.

And I personally feel it's helpful not to go beyond the scope of the original issue in the discussion (here, it was “Extend the for member syntax to maps”). Otherwise, issues like this tend to be closed later on and replaced by new issues.

[dnovatchev]

Let's see what other people have to say.

I completely agree with Michael Kay here. Smaller steps to start with won’t prevent us from doing bigger steps later on.

And I personally feel it's helpful not to go beyond the scope of the original issue in the discussion (here, it was “Extend the for member syntax to maps”). Otherwise, issues like this tend to be closed later on and replaced by new issues.

Agreed, I submitted this as a new issue:

Proposal to introduce the set datatype in XPath 4

[dnovatchev]

Going beyond sets of atomic values to sets of arbitrary values immediately gets you into the problem of defining equality between arbitrary values, which is a quagmire.

People do it in other PLs. In the .NET world one simply uses object.GetHashCode()

[michaelhkay]

It's unfortunate that in 3.1, the map:entry() and map:merge() functions model a key-value pair as a singleton map. I'm coming to the conclusion that decomposing a map into (key, value) records (record(key as xs:anyAtomicType, value as item()*)* is much more useful. But mixing the two representations is awkward.

[ChristianGruen]

Instead of for entry $entry in ..., it might be more user-friendly to offer:

for key $key in ...
for value $value in ...
for key $key value $value in ...

Related: https://lists.w3.org/Archives/Public/public-xslt-40/2023Jun/0026.html

[ChristianGruen]

The grammar for the for key $k value $v in ... syntax could be as follows (provided we simplify the existing grammar for member bindings as proposed in #706):

InitialClause     ::=  ForClause | LetClause | WindowClause
ForClause         ::=  "for" ForBinding ("," ForBinding)*
ForBinding        ::=  (SimpleForBinding | ForMemberBinding | ForMapBinding) PositionalVar? "in" ExprSingle
SimpleForBinding  ::=  VarBinding AllowingEmpty?
ForMemberBinding  ::=  "member" VarBinding
ForMapBinding     ::=  (ForKeyBinding ForValueBinding?) | ForValueBinding
ForKeyBinding     ::=  "key" VarBinding
ForValueBinding   ::=  "value" VarBinding
VarBinding        ::=  "$" VarName TypeDeclaration?
AllowingEmpty     ::=  "allowing" "empty"
PositionalVar     ::=  "at" "$" VarName

[michaelhkay]

I would love to find a more specific term than "value" for use when we talk of key-value pairs. Both for use in narrative prose and for use in this new syntax. It needs something that alerts the reader that we're talking about values in a key-value pair, not just any old value.

[dnovatchev]

I would love to find a more specific term than "value" for use when we talk of key-value pairs. Both for use in narrative prose and for use in this new syntax. It needs something that alerts the reader that we're talking about values in a key-value pair, not just any old value.

Maybe key-projection or just projection ?

Or mapping-result or map-result or just result ?

[rhdunn]

The terms key and value are widely used in computer science literature in reference to maps and other similar constructs (such as key-value data stores). Those names are also used consistently across programming languages (JavaScript, Java, Kotlin, C#, C++, Python, etc.) when referring to the entries in a map.

Therefore, I would personally object to something else used in the proposed syntax.

For the prose, we could use something like "map value" or "value of the map" if we want to be specific -- I wouldn't object to something like that.

Things like "projection" and "result" have different meaning that is likely to confuse a user.

[ChristianGruen]

I would love to find a more specific term than "value" for use when we talk of key-value pairs. Both for use in narrative prose and for use in this new syntax. It needs something that alerts the reader that we're talking about values in a key-value pair, not just any old value.

Yes, that’s a tricky one. In an ideal world, I would prefer another term as well, but I don’t expect it to be misleading in the given context: When talking about maps, directories, associative arrays etc., it’s just too common to talk about keys & values or names & values. If you don’t have maps as input, there’ll be no reason to use the keyword in your FLWOR expression.

[ChristianGruen]

Shall we go for for key $k value $v in ... , or are there any serious concerns against it?

[dnovatchev]

Now when we have established the record type (or haven't we?) why not use just a single $kvp variable of type "key-value" record?

[ChristianGruen]

Now when we have established the record type (or haven't we?) why not use just a single $kvp variable of type "key-value" record?

A while ago when we gathered feedback, separate key/value variables were considered to be more intuitive (see also #31 (comment)).

[dnovatchev]

for key $k value $v in

Speaking about convenience, I would prefer something like:

for ($k, $v) in $someMap

Another possible issue is that the order of the results is unpredictable, thus it may be useful to have:

for ($k, $v) in sorted $someMap

which would return the ($k, $v) tuples in key-sorted order

[michaelhkay]

I'd be happy to have either

for key $k value $v in ...

or

for {$k, $v} in ...

With a preference for the former as it's better aligned with the syntax for member $m.

But what about "as" and "at" clauses?

As for sorting, I think a standard order by $kclause is perfectly adequate.

[dnovatchev]

I'd be happy to have either

for key $k value $v in ...

or

for {$k, $v} in ...

My preference is for the latter - more concise and succinct.

[ChristianGruen]

If we added for {$k, $v}, we could (in alignment with #37) change for member $m to for [$m]; but I think we should stop changing features that have already been added to the spec (users are getting impatient for understandable reasons and increasingly use 4.0 features in productive code).

Next, for key $k in $map and for value $v in $map are helpful if you only need one segment of a map entry.

But what about "as" and "at" clauses?

at is beneficial if you want to enumerate map entries and don’t care about their order. People already do things like:

for $key at $pos in map:keys($map)
for $value in $map($key)
return $pos || '. ' || $key || ': ' || $value

Similarly, I would keep as even if it may not be used excessively. The more similar for works with and without member, key and value, the better.

[rhdunn]

Note that the for {$x, $y} ... syntax is similar to the map part of the decomposition syntax I proposed in expath/xpath-ng#8.

[rhdunn]

The XPath 4.0 discussion is in issue #37 for the for {$x, $y} ... map syntax. I propose keeping this about the for key $k value $v in ... and for member $e in ... syntax, and having the destructuring {$k, $v} in map:entries($map) syntax ib #37 as that extends the support to let expressions as well as sequences and arrays.

[ChristianGruen]

The XPath 4.0 discussion is in issue #37 for the for {$x, $y} ... map syntax. I propose keeping this about the for key $k value $v in ... and for member $e in ... syntax, and having the destructuring {$k, $v} in map:entries($map) syntax ib #37 as that extends the support to let expressions as well as sequences and arrays.

I would favor that.

[dnovatchev]

If we added for {$k, $v}, we could (in alignment with #37) change for member $m to for [$m]; but I think we should stop changing features that have already been added to the spec (users are getting impatient for understandable reasons and increasingly use 4.0 features in productive code).

What about the standard F&O 3.1 functions fn:load-xquery-module and fn:transform that even now (after waiting 10+ years) are still not implemented in BaseX?

Yes, someone could tell us that "there is something like this... in BaseX", but the fact remains that BaseX is not even 3.1-compliant 10 years after publishing the official F&O Specs. Isn't this an intentional effort to prevent portability and interoperability in order to lock users into BaseX?

Why worrying only about some users and not about other, who need to rely on compliance and interoperability?

Please, be aware of this fact, which suggests that at least some users may have become desperate by now.

[ChristianGruen]

Please, be aware of this fact, which suggests that at least some users may have become desperate by now.

I’m not sure how that relates to my observation. The point I was trying to make is that we are seeing more and more 4.0 features being used in production environments – it’s not just a hypothesis. I certainly won’t object, though, if the majority of us believes that it’s better to revert or revise features of the drafts.

If you rather want to point out we should take our work on BaseX more seriously, or set other priorities, and if you want to learn what others think about it, our mailing list may be a better platform than this thread. Another option is to participate. As you may know, it’s all Open Source and freely available to everyone…

[dnovatchev]

I’m not sure how that relates to my observation.

I was just pointing out the fact that BaseX is in a hurry to implement new 4.0 features, while at the same time this application still remains incompliant with 3.1.

This points out to particular, special and selective preferences in the implementors' decision-making.

Personally, as a user I would prefer a compliant and interoperable XPath implementation to one that might be even more optimized and might have implemented a lot of new features, but at the same time has remained incompliant with established, official versions and specifications, which hinders interoperability and results in user-lockdown.

[ChristianGruen]

Personally, as a user […]

Just use the alternatives ;)

[michaelhkay]

This is not the right place to discuss the conformance level of implementations.

As regards the point:

I think we should stop changing features that have already been added to the spec

I think we should avoid encouraging users to believe that they can rely on stability in the current draft specifications. A level of stability is desirable because it is the only way we will ever finish, but we should not allow that kind of argument to prevent us improving the work we have done where improvements are identified.