Support optional parameters on dynamic functions

[rhdunn]

This proposal extends #155 to dynamic functions.

Motivation

Within XPath there isn't a defined mechanism for declaring static functions. It can be such that declaring a function outside of the XPath expression is more cumbersome (such as defining it within XSLT) and makes the resulting expression less portable between XSLT and XQuery.

There are 2 cases where this applies:

1. Defining inline functions -- These should work like declaring default values for static functions, with the rules for applying default values working at the dynamic call site.

2. Specifying named function references for functions with additional default arguments, such as fn:tokenize#2("lorem ipsum") where the fn:tokenize#2 binds the $flags default value and the dynamic call applies the default value for $pattern. -- This is possible to define as creating an inline function such that any default parameters are preserved along with the parameter types at the corresponding parameter index.

Note

For FunctionTest (i.e. function types), I'm not sure of a use case where extending optional parameters to that would be useful, as any higher-order function would always call the input function with the correct number of arguments. If this is useful, then that should be moved to a separate proposal.

[michaelhkay]

It's not really clear what change to the specification is being proposed here.

I'm not at all confident that optional parameters on dynamic function calls can be made to work. It certainly affects the type system for functions and the substitutability rules in quite a complex way. Currently the type of a function defines a fixed arity, and if fn:for-each-pair(), for example, requires as its argument a function of arity 2, then supplying a function of arity 1 isn't going to work.

You're suggesting, I think, that for some arity-2 functions the second argument is optional, and for others it isn't. If that's the case, then a function in which the second argument is required can't be used in a context where the required type has arity 1. This is a pretty fundamental change to the type system and this proposal doesn't address the implications at all.

[rhdunn]

The changes are: 1) allow inline function calls to support default arguments; 2) support default argument resolution in dynamic function calls.

I'm capturing this for completeness. If we agree that this problem is too hard, then that's fine. It also allows us to focus on the easier parts captured in the other proposals. -- I.e. focus on pinning down a complete specification for the easier parts first, and getting those agreed on, before moving onto the harder parts.

[dnovatchev]

I think this proposal is useful.

We could think of a function call to an inline function using default values for some of its arguments as a kind of closure, where the main function produces a (closure) function that uses some of the (default) values of the main function, but still has the same name as the main function.

A code example will make this clear:

let $greet := 
     function($greeting as xs:string)
       as function(xs:string) as xs:string
       {
         function($name as xs:string) as xs:string
         {
           $greeting || $name || '!'
         }
       },
       $greet := $greet('Hello: ')
  return
    ( $greet('John'), $greet('Peter') )

When the above Xpath expression is evaluated, the expected result is produced:

Hello: John!
Hello: Peter!

This proposal allows us to substitute the above lengthy code with just:

let $greet := function($name as xs:string, $greeting = 'Hello: ' as xs:string) as xs:string
              {
                 $greeting || $name || '!'
              }
   return
      ( $greet('John'), $greet('Peter') )

[michaelhkay]

Before we get carried away with ideas for how we could use a new feature, let's consider the impact on the language fundamentals: how does adding default values to (dynamic) function items affect the data model and type system, how does it affect subsumption, how does it affect existing operations on function items such as partial applications.

I've said before, I want to see very strong rationale for anything that involves changing the data model or type system. I've spent the last 2 days trying to write up the specification for the much simpler proposal of adding default parameters to static functions, and even that is very disruptive and very hard work.

[dnovatchev]

Let us note that the example provided by Reece:

fn:tokenize#2("lorem ipsum")

can be defined as a shorthand of the correct Xpath expression:

tokenize#2(?,  ' ' )('lorem ipsum')

Therefore, we can strictly define a function call to a function:

let  $f := function($arg1, $arg2 =  $someValue2,  $arg3 =  $someValue3, ...  $argN =  $someValueN) { (: Some Expression :) }
 return
      $f($someValue1, $someValue2, $someValue3, ..., $someValueN) 

as :

let  $f := function($arg1, $arg2 =  $someValue2,  $arg3 =  $someValue3, ...  $argN =  $someValueN) { (: Some Expression :) }
 return
      $f(?, $someValue2, $someValue3, ..., $someValueN) 

and use a shorthand for this:

$f($someUnknownValue)

that omits some of its arguments (that have default values

@michaelhkay and everyone else: What issues do you see in introducing this shorthand for the completely valid/legal/correct XPath 3 (even XPath 2) call above?

[michaelhkay]

A function item does not have default values. The properties of function items are defined in the data model spec, and do not include default values. It's not a syntactic shorthand, it's a change to the data model.

[ChristianGruen]

and everyone else: What issues do you see in introducing this shorthand for the completely valid/legal/correct XPath 3 (even XPath 2) call above?

I certainly like all the examples. As Michael indicated, we need a solid specification to ensure that all extensions go hand in hand with the existing features and the basics. So far, I confess I avoided diving into variadic functions and optional parameters, as I can easily feel there are many detail questions to be solved (I’d just be lucky to be able to name them).

Obviously, the same observation applies to the previous issues here, and it was a good idea to split them up. We should start off with and finalize the easier ones first. Optional parameters on dynamic functions are rather advanced, I’d tend to give them an initial lower priority.

[dnovatchev]

A function item does not have default values. The properties of function items are defined in the data model spec, and do not include default values. It's not a syntactic shorthand, it's a change to the data model.

I didn't say that. to quote my previous comment, what I said was:

... some of its arguments (that have default values

And a shorthand for a currently-valid expression certainly doesn't affect the DM at all.

[michaelhkay]

I am coming round to the idea that this idea is reasonably feasible so long as we stick to positional arguments.

First, we should start with changes to the data model. We make a change to the properties of [dynamic] functions (XDM §2.8.1):

• Parameters can be required or optional.
• If a parameter is optional, then all subsequent parameters must be optional
• If a parameter is optional, then it has a default value, which is any XDM value (note, not an expression).

Then we define how constructs that create new functions set the values of these properties. There are three ways of creating new [dynamic] functions:

Function references: A function reference refers to a declared function in the static context. F#4 creates a [dynamic] function corresponding to the declared function named F whose arity range includes 4. If any of the first four parameters of this function are optional, then the corresponding parameters in the new dynamic function become optional, and their values are set to the result of evaluating the default values of the parameters in the dynamic context of the function reference.

Partial application: I'm inclined to say that partial application should deliver a function whose parameters are all mandatory. Anything else feels too complicated.

Inline functions: We can extend the syntax of inline function expressions to allow default values to be specified, in the same way as for XQuery function declarations, and with similar rules. The default value expressions are evaluated in the dynamic context of the inline function expression.

Function types do not change; the type of a function ignores the name, optionality, and default value of its parameters. So the type matching and subsumption rules are not affected.

A call to a dynamic function supplies argument values by position only (for the time being), and unbound parameters take their default values.

The function coercion rules should allow a function with arity 3 to be supplied where a function with arity 2 is expected provided the last parameter is optional. The effect of function coercion is to do a partial application of the supplied function in which the last parameter is bound to its default value.

Separately, function coercion should also allow a function with arity 1 to be supplied where a function with arity 2 is expected. The effect of function coercion here is to create a wrapper function that calls the supplied function with the first argument and drops the second argument.

This doesn't address the requirement in issue #160 to allow keyword arguments on dynamic function calls. The issue there is substitutability: if a user function U expects a parameter P of type function(xs:integer) as xs:boolean, different callers are going to supply different functions with different parameter names, so the implementation of U has no idea what keywords to use when calling $P. We can address that in issue #160.

[dnovatchev]

Separately, function coercion should also allow a function with arity 1 to be supplied where a function with arity 2 is expected. The effect of function coercion here is to create a wrapper function that calls the supplied function with the first argument and drops the second argument.

This is just millimeters from saying that "arguments on the function call shall be provided and evaluated only upon demand (reference) from the code of the function".

Please, !!!

[michaelhkay]

Re-reading my comment of Oct 11, in the light of work done meanwhile on static functions, I think it's all still valid, except that I'm not sure whether the default value held in a function item should be an expression (to be evaluated in the context of the dynamic function call) or a value (evaluated when the function item is created).

[dnovatchev]

except that I'm not sure whether the default value held in a function item should be an expression (to be evaluated in the context of the dynamic function call) or a value (evaluated when the function item is created).

It seems to me that this can be just the value when the function item is created. At least this is the case in C#, AFAIK.

Do we have compelling use cases for an expression that needs to be evaluated dynamically on every call?

[michaelhkay]

Consider

let $order-value := function($n as node() := .) { sum($n/order-line !(@price * @qty)) }
return avg( $orders ! $order-value() )

Here the user is clearly expecting to use the context node at the point of the dynamic function call. Whether this is "compelling" or not I leave others to judge.

[dnovatchev]

Consider

let $order-value := function($n as node() := .) { sum($n/order-line !(@price * @qty)) }
return avg( $orders ! $order-value() )

Here the user is clearly expecting to use the context node at the point of the dynamic function call. Whether this is "compelling" or not I leave others to judge.

Not compelling for me. Actually confusing.

If a default value is to be evaluated dynamically, let us make the rule that in this case the user should specify a function that is to be invoked at the point of evaluation, that will produce the wanted (dynamic) default value.

let $order-value := function($getNode as ->() as node() :=  ->(){.}) { sum($getNode()/order-line !(@price * @qty)) }
return avg( $orders ! $order-value() )

[ChristianGruen]

The proposal would be helpful for focus functions as well: If the empty sequence is defined as the default value for its parameter, focus functions can be called without arguments:

let $hi := fn { 'Hi!' }
return $hi()

#129 (Context Values) is another prerequisite to make this work.

The complementary behavior (support for functions with a lower arity) is already available via function coercion (see #705).

[ndw]

Part of a nexus of issues related to variable arity dynamic functions. (See minutes of 4 June, 2024)