XPath: Optional parameters in the definition of an inline function

[dnovatchev]

This is a proposal to extend the definition of an inline-function item with the ability to specify a set of optional/keyword-value parameters, following the sequence of positional parameters of the function.

This is very similar to what we already have for static function definitions: https://qt4cg.org/specifications/xquery-40/xpath-40.html#dt-function-definition and https://qt4cg.org/specifications/xquery-40/xpath-40.html#id-static-functions

While a static function definition has the following parts:

• The function name, which is an expanded QName.

• A (possibly empty) list of required parameters, each having:

• a parameter name (an expanded QName)

• a required type (a sequence type)

• A (possibly empty) list of optional parameters, each having:

• a parameter name (an expanded QName)

• a required type (a sequence type)

• a default value expression (an expression: see 4 Expressions)

• A return type (a sequence type)

• A (possibly empty) set of function annotations

• A body. The function body contains the logic that enables the function result to be computed from the supplied arguments and information in the static and dynamic context.

For an inline function definition we will have:

• A name of a variable to contain the function item being defined.

• A (possibly empty) list of required parameters, each having:

• a parameter name (an expanded QName)

• an optional type (a sequence type)

• A (possibly empty) list of optional parameters, each having:

• a parameter name (an expanded QName)

• an optional type (a sequence type)

• a default value expression (an expression: see 4 Expressions)

• An optional return type (a sequence type)

• A body. The function body contains the logic that enables the function result to be computed from the supplied arguments and information in the static and dynamic context.

What is accomplished by introducing optional parameters?

The answer is the same as for the effect of having optional parameters in a static function definition: increased brevity, conciseness and clarity .

let $myFun := fn($pos1, $pos2, $posK, $kw1 := expr1, $kw2 := expr2, ...,  $kwN := exprN) { (: Some expression :)} 

replaces what would otherwise be a set of N! + 1 separate inline function definitions, each of which must be assigned to a separate variable.

Similarly to the static function calls, with this new feature a call to such an inline function must provide values for all positional arguments, followed by an optional set (meaning in any order) of assignments of values to specific keyword-valued (optional) arguments. The rules for an inline function call are similar to those for a call to a static function - the provided values for the positional arguments must precede all other provided values and the values for the optional arguments may be provided in any order.

Here is a short example of an inline function definition and calling it:

let $incr := fn($arg1, $increment := 1) {$arg1 + $increment }
 return
  (
   $incr(5),
   $incr(5, increment := 2),
   $incr(5, increment := 3)
 )

produces:

6, 7, 8

[michaelhkay]

(a) How does this affect function items created using named function references, or partial function application?

(b) What is the effect on the type system, especially function item types and the subsumption rules?

[michaelhkay]

And the other problem (already raised repeatedly) with keyword parameters on dynamic function calls is, how does the caller know what keywords are available? There needs to be some mechanism that associates keywords with the type of the function, not just with the individual function instance; and then some mechanism for re-mapping the keywords of a supplied funcion item to the keywords of the required item type.

[dnovatchev]

I wanted to be as brief as possible. The rules for optional parameters defined on inline function definitions must be as similar as possible to the rules for optional parameters on statically-defined functions.

(a) How does this affect function items created using named function references, or partial function application?

(b) What is the effect on the type system, especially function item types and the subsumption rules?

Maybe you could provide examples that clarify the questions?

[dnovatchev]

And the other problem (already raised repeatedly) with keyword parameters on dynamic function calls is, how does the caller know what keywords are available?

The caller is typically the one who defined the function - otherwise this makes little sense. An inline function is alive in the scope of its definition, so are its optional parameters.

There needs to be some mechanism that associates keywords with the type of the function, not just with the individual function instance; and then some mechanism for re-mapping the keywords of a supplied funcion item to the keywords of the required item type.

I don't understand this statement - again, an example would be welcome.

[michaelhkay]

Examples.

1. Given a function declaration

declare f($a, $b := 3)

what do f#1 and f#2 return? Do both return function items with fixed arity? Is there some way of returning a function item that has one mandatory and one optional parameter?

2. Given a higher order function
   declare f($a as function($x, optional $y))

• is it even possible to declare the argument type as being a function with optional parameters, and what syntax is used?
• what kinds of function item can legally be supplied in a function call as values of this argument?
• how are the parameter names in the supplied function mapped to the parameters names of the required type?
• or perhaps the required type doesn't specify parameter names, in which case how are keyword arguments possible?

I suspect you are thinking of XPath expressions in which "dynamic" functions are not really dynamic at all - because standalone XPath doesn't have static user-written functions at all. But that scenario in my view is atypical; function items defined using inline function expressions are used primarily as arguments to higher-order functions, and in that situation it's all about the relationship of the supplied function to the type required by the higher-order function.

You say

The caller is typically the same who defined the function

but I think that's very untypical. The only reason for using dynamic functions rather than static functions is so that you can define a function and pass it around to be called by someone else.

[dnovatchev]

Examples.

1. Given a function declaration

declare f($a, $b := 3)

Sorry, but the above is a statically defined function. We need an example of an inline function item.

what do f#1 and f#2 return? Do both return function items with fixed arity?

Yes.

Is there some way of returning a function item that has one mandatory and one optional parameter?

No, and there is no evidence of the usefulness of being able to do so - see below.

2. Given a higher order function
   declare f($a as function($x, optional $y))

Exactly the same remark. The above is a statically defined function. We need an example of an inline function item.

Added to this: it is not at all clear how the knowledge would be useful in the body of $f, that the $a function has an optional second parameter. If this information has no use, then not having to specify it would be the right decision to make.

Even if $f knew that $a has a second argument that is optional, $f can never call $a with a specific value for this second argument (like should it be 0 for a sum() function or 1 for a product() function -- because it is completely unknown if $a is a sum() or a product(), or ... ? function). Thus, the caller of $f is the complete and only authority deciding what the exact/correct default value of the 2nd argument of the provided $a function should be.

A correct way to pass the function as parameter, is to pass its partial application, where all such uncertainties (default values) have already been fixed. This is completely and exactly in line with major programming principles.

• is it even possible to declare the argument type as being a function with optional parameters, and what syntax is used?

No such "feature" is part of this proposal as there is no evidence it would be useful.

• what kinds of function item can legally be supplied in a function call as values of this argument?

Just function items with only positional arguments - at the point of the call - and certainly any partial application of another function can be provided.

• how are the parameter names in the supplied function mapped to the parameters names of the required type?

No need for any such specific mapping.

• or perhaps the required type doesn't specify parameter names, in which case how are keyword arguments possible?

No, once a function is passed around, its definition is lost, thus the receiver would never know whether this function was produced from a function that was defined with some optional parameters. The receiver will not even know whether or not the passed function was a statically defined or a dynamically defined one.

I suspect you are thinking of XPath expressions in which "dynamic" functions are not really dynamic at all - because standalone XPath doesn't have static user-written functions at all. But that scenario in my view is atypical; function items defined using inline function expressions are used primarily as arguments to higher-order functions, and in that situation it's all about the relationship of the supplied function to the type required by the higher-order function.

The caller is able to pass a partial application of the function where the values of some arguments have been fixed, and the call to this partial application is equivalent to calling the complete function with some of the optional arguments omitted.

You say

The caller is typically the same who defined the function

but I think that's very untypical. The only reason for using dynamic functions rather than static functions is so that you can define a function and pass it around to be called by someone else.

As already commented, the choice of the shortened form of the call (or of the needed equivalent partial application) can be made upon that call, so that the receiver is completely unaware of any arguments-values-fixing.

[ChristianGruen]

but I think that's very untypical. The only reason for using dynamic functions rather than static functions is so that you can define a function and pass it around to be called by someone else.

I haven't fully digested the details of this thread, but my experience is that there are various reasons for using dynamic functions. As a developer, you may be completely aware of the parameter names of your functions, no matter whether they are static or dynamic:

declare function local:inc($n, $by := 1) { $n + $by };
local:inc(123),
local:inc(123, by := 2)

declare variable $inc := fn($n, $by := 1) { $n + $by };
$inc(123),
$inc(123, by := 2)

let $inc := fn($n, $by := 1) { $n + $by }
return (
  $inc(123),
  $inc(123, by := 2)
)

[michaelhkay]

If the function item with optional parameters can't usefully be passed as an argument to a higher-order function, then in my view the cost of the feature is far higher than any benefit.

[dnovatchev]

If the function item with optional parameters can't usefully be passed as an argument to a higher-order function, then in my view the cost of the feature is far higher than any benefit.

Nobody said this cannot be done - this can be done even at present, but we are still waiting even for a single use case.

@ChristianGruen in his comment already pointed out a good case of using an inline function item that is defined with one optional parameter. And there is a huge set of similar use-cases, which are the targets of the current proposal.

The main achievement of the proposal was clearly defined in the opening comment:

What is accomplished by introducing optional parameters?

The answer is the same as for the effect of having optional parameters in a static function definition: increased brevity, conciseness and clarity .

let $myFun := fn($pos1, $pos2, $posK, $kw1 := expr1, $kw2 := expr2, ...,  $kwN := exprN) { (: Some expression :)} 

replaces what would otherwise be a set of N! + 1 separate inline function definitions, each of which must be assigned to a separate variable.

If the compacting into one single function of N! +1 different functions that each has to be defined separately and assigned to one of N! +1 separate variables each with distinct name - if this is not appreciated as a "good enough benefit", then indeed, further discussions would not be meaningful.

[michaelhkay]

What you say is all true in a pure XPath world, where you are using dynamic functions because XPath doesn't have any capability for static function declarations. But in XQuery or XSLT, you would be using static function declarations for this kind of use case. I know we differ on this, but I don't regard standalone XPath as important enough to justify additional features and data model extensions that are of very little use in XSLT or XQuery.

And even if you're not concerned with the implications on the type system, those implications can't be ignored: we need to define what types these extended function items belong to, and how the subsumption and coercion operations work on these types.

[dnovatchev]

What you say is all true in a pure XPath world, where you are using dynamic functions because XPath doesn't have any capability for static function declarations. But in XQuery or XSLT, you would be using static function declarations for this kind of use case. I know we differ on this, but I don't regard standalone XPath as important enough to justify additional features and data model extensions that are of very little use in XSLT or XQuery.

And even if you're not concerned with the implications on the type system, those implications can't be ignored: we need to define what types these extended function items belong to, and how the subsumption and coercion operations work on these types.

This proposal can be implemented even as a lexical shorthand mechanism - that would not require any additions or changes to the type system.

[michaelhkay]

So if I write

let $inc as X := fn($n, $by := 1) { $n + $by }

what might be some valid values for X?

And (a different question because it's about type matching rather than coercion), what might be some values for T in the following, such that the result of the expression is true?

let $inc := fn($n, $by := 1) { $n + $by } return $inc instance of T

Perhaps, because $inc can be called with either 1 or 2 arguments, T needs to be a choice type. That might well work - but it needs detailed rules, you can't just ignore the question.

[dnovatchev]

what might be some values for T in the following, such that the result of the expression is true?

let $inc := fn($n, $by := 1) { $n + $by } return $inc instance of T

Perhaps, because $inc can be called with either 1 or 2 arguments, T needs to be a choice type. That might well work - but it needs detailed rules, you can't just ignore the question.

What appears on the surface to be calling $inc with one argument is just a lexical sugar for the call $inc($x, 1).

Thus the question is equivalent to: "What is the value for T so that the result of the expression
let $inc := fn($n, $by) { $n + $by } return $inc instance of T is true.

There is nothing new here.

We don't have any problems with fn:op. Why should we have problems here?

[michaelhkay]

"Lexical sugar" implies you can expand it during parsing. But we don't know what $inc is until we have evaluated it. Consider for example

let $inc := fn($n, $by := 1) { $n + $by }
let $inc2 := if (current-date() gt $baseDate) then $inc else $somethingElse
return $inc2(17)

How can $inc2 be "lexical sugar"? What is its type?

[dnovatchev]

How can $inc2 be "lexical sugar"? What is its type?

Thank you for clarifying.

We can define the function $myKwdFun:

fn($pos1, $pos2, $posK, $kw1 := expr1, $kw2 := expr2, ..., $kwN := exprN)

to be equivalent to:

let $myKwdFun :=
    let $kwd := {$kw1 := expr1, $kw2 := expr2, ...,  $kwN := exprN}
      return
          fn($pos1, $pos2, $posK)
         {  (: expression where every reference to $kwM is lexically replaced by the expression $kwd?kwM  :)}

and a call to such a function that overrides some keyword parameters' values:

$myKwdFun($pos1, $pos2, ..., $posK, $kwX1 := exprX1, $kwX2 := exprX2, ..., $kwXt := exprXt)

is lexically replaced by (is a synonym of):

let $myKwdFun2 :=
 (
    let $kwd := {$kw1 := expr1, $kw2 := expr2, ...,  $kwN := exprN}
      return
       (
         let $kwd2 := $kwd => map:put('kwX1', exprX1) => map:put('kwX2', exprX2) => ...=>map:put('kwXt', exprXt)
           return
             fn($pos1, $pos2, ..., $posK)
                {  (: expression where every reference to $kwM is lexically replaced by the expression $kwd2?kwM  :)}
       )
 )
return 
  $myKwdFun2($pos1, $pos2, ..., $posK)

Thus, the type of the function is:

fn($pos1 as T1, $pos2 as T2, ... $posK as Tk, $keywords as map(xs:string, item()*) )