HOF Sequence Functions with Positional Arguments

[ChristianGruen]

Motivated by @michaelhkay’s question in #80 (comment), I wondered if we should add optional positional arguments to built-in sequence functions that take higher-order function arguments (thanks, Dimitre:) extend the types of the functions that are themselves parameters to certain standard functions, by adding to their signature one more parameter, which is the index of the item that is also passed as an argument to this function.

Some examples:

(: fn:filter: Accept items that are identical to their predecessors :)
fn:filter(
  $sequence,
  fn($item, $pos) { $item = $sequence[$pos - 1] }
)

(: fn:for-each: Create enumerated strings for all items :)
for-each(
  $sequence,
  fn($item, $pos) { $pos || '. ' || $item }
)

(: fn:for-each-pair: Create enumerated strings for the maximum value of all item pairs :)
for-each-pair(
  $seq1, $seq2,
  fn($item1, $item2, $pos) { $pos || '. ' || max(($item1, $item2)) }
)

(: fn:fold-left: Return positions of the items matching a specific value :)
let $input := (11 to 21, 21 to 31)
let $search := 21
return fold-left($input, (), fn($seq, $curr, $pos) { $seq, $pos[$curr = $search] })

Edit: I was wrong with my claim that the filter function could be used to replace index-where, and I have modified the example.

[dnovatchev]

@ChristianGruen I am really sorry, but I read and re-read this multiple times and I really cannot understand what is being proposed here.

All of the listed standard functions (filter, for-each, for-each-pair, fold-left) already have a positional parameter which is a function... So what is proposed? What is new?

I apologize for being that dumb :( :( :(

---

Now that I am reading it again, I begin to understand the meaning: it is not to add a function parameter to certain standard functions, but the proposal may be made easier to understand by formulating it as:

"Extend the types of the functions that are themselves parameters to certain standard functions, by adding to their signature one more parameter, which is the index of the item that is also passed as an argument to this function"

Or something similar

[ChristianGruen]

@dnovatchev Thanks for your reply. It’s definitely not you who’s dumb; instead, I should invest more time on phrasing things. Your formulation sounds fine, which is why I adopted it directly in my comment. In addition, I added some comments to the example queries.

Again, my idea is something that has already been realized in other programming languages. The most prominent example may be forEach in JavaScript, which can be called with 1, 2 or 3 arguments (current element, index, original array).

Do you think the extension would be helpful?

[michaelhkay]

Yes, this is what I have been intending. The change is hinted at in §4.4.4 of the draft XPath specification, where I wrote:

Note:
This mechanism makes it easier to design versatile and extensible higher-order functions. For example, in previous versions of this specification, the second argument of the fn:filter function expected an argument of type function (item()) as xs:boolean. This has now been extended to function (item(), xs:integer) as xs:boolean, but existing code continues to work, because callback functions that are not interested in the value of the second argument simply ignore it.
TODO: this change to fn:filter has not yet been made.

Note also that Javascript filter callbacks work this way, so the concept will be familiar to many users.

[ChristianGruen]

@michaelhkay Great; thanks.

[dnovatchev]

Again, my idea is something that has already been realized in other programming languages. The most prominent example may be forEach in JavaScript, which can be called with 1, 2 or 3 arguments (current element, index, original array).

Do you think the extension would be helpful?

Yes, and this is also one of the overloads of .NET LINQ IEnumerable.Select() method.

I myself proposed such general amendment at one of our previous meetings (https://qt4cg.org/meeting/minutes/2022/09-20.html when discussing map:filter())

We need to add this new argument to all parameter-function-values -- of all current and projected new standard functions that accept as input a sequence or an array and pass to that parameter-supplied function just the item (member), but not also the index.

[dnovatchev]

This has now been extended to function (item(), xs:integer) as xs:boolean, but existing code continues to work, because callback functions that are not interested in the value of the second argument simply ignore it.

Can't we eliminate this problem altogether by having these signatures ?

function (item(), xs:integer?) as xs:boolean -for callbacks on items of sequences

and

function (item()*, xs:integer?) as xs:boolean -for callbacks on array members

[michaelhkay]

We obviously can't require the caller of fn:filter to supply a function that is an instance of function (item(), xs:integer?) as xs:boolean, because that would invalidate existing callers. That's why I proposed doing it by extending the function coercion rules, so that we can declare the required type as function (item(), xs:integer) as xs:boolean, and a function of type function (item()) as xs:boolean will still be accepted.

[hrennau]

Doubtless a very useful extension! As Michael Kay so succinctly said: "makes it easier to design versatile and extensible higher-order functions". To add a (pseudo?) theoretical note: the information content of the function input is extended from the "item" to the "item as a member of the sequence", which gives it a sequence-scoped identity. Any aspect of the relationship between the information content of the item and of the sequence becomes accessible.

[dnovatchev]

The obvious and hanging suggestion that we need to do the same to all array:functions that are the counterparts of the standard functions defined on sequences.

With the risk of missing something, these are:

• array:filter
• array:for-each
• array:for-each-pair
• array:fold-left
• array:fold-right

[ChristianGruen]

I’m summarizing the functions for which positional arguments would be beneficial:

• fn:filter
• fn:fold-left
• fn:fold-right
• fn:for-each
• fn:for-each-pair
• array:filter
• array:fold-left
• array:fold-right
• array:for-each
• array:for-each-pair

In addition, it could be added to:

• Functions for splitting a sequence (or array) based on predicate matching  #149 (fn:items-before, ...)
• fn:all
• fn:some
• fn:index-where

[michaelhkay]

Add fn:sort. People use position() in xsl:sort, it can provide a neat way of arranging data in tables. Probably map:build() too.

[dnovatchev]

Here is an observation and proposal following from it:

Within a for - expression in XPath we currently don't have access to the index(es) of the bound variable(s).

In the spirit of the current proposal, I also propose to have such facility within a for - expression* .

For example, we could have an expression like this:

for $x in (1,  3, 5, 7, 9, 11),
    $y in (1,  3, 5, 7, 9, 11)
 return
     if(index($x) le index($y)) then [$x, $y]
       else ()

Which produces all pairs of items in which the 1st item in the pair is from the first sequence (x-s) and the second item in the pair is from the 2nd sequence (y-s). From a set of duplicate pairs (as sets) , like (1,3) and (3,1) only the first is selected and returned:

[1,1]
[1,3]
[1,5]
[1,7]
[1,9]
[1,11]
[3,3]
[3,5]
[3,7]
[3,9]
[3,11]
[5,5]
[5,7]
[5,9]
[5,11]
[7,7]
[7,9]
[7,11]
[9,9]
[9,11]
[11,11]

Note: Probably we need just to expand the XPath syntax with the at keyword which already exists in XQuery, thus the above expression will simply be:

for $x at $ind-x in (1,  3, 5, 7, 9, 11),
    $y at $ind-y in (1,  3, 5, 7, 9, 11)
 return
     if($ind-x le $ind-y) then [$x, $y]
       else ()

Your thoughts appreciated.

[ChristianGruen]

@dnovatchev Would you mind creating a separate issue for the proposal?

I immediately thought of the at keyword.

[ChristianGruen]

Closed in favor of #516.

[michaelhkay]

Sorry about the duplication.