What would it take/would it be possible to build a module repository for QT?

[ndw]

We have an ever growing list of proposed convienence functions. I am not opposed, in principle, to adding convenience functions, but we don’t have any principled criteria (AFAICT) for which ones to add and which ones to reject. That’s not surprising, and I’m also not opposed to that. But I’m sure there are hundreds, perhaps thousands, of such functions. At some point, we’re going to start to resist adding more simply because we’ve added so many. Some of us may already be nearing that point.

It seems to me that the alternative is to do what TeX, Perl, Python, Node, etc. do: make it easy for users to download, install, and use libraries. (I’m carefully using the term “library” here where I might prefer to use “package” or “module” because we already have “package” and “module” which mean other things.)

What would it take to make that possible?

One problem we have is that there are two (perhaps three, or more, depending on how you count) different QT languages and they aren’t all mutually interoperable. My XSLT implementation of fn:parse-uri for example, isn’t directly usable by an XQuery product that doesn’t implement XSLT or some other product that only uses XPath.

Suppose we added an import library declaration to XPath, similar to the import module declaration in XQuery

LibraryImport := "import" "library"
                   ("namespace" NCName "=")?
                   "at" URILiteral

and a corresponding <xsl:library> instruction to XSLT.

<xsl:library
  namespace = uri
  href = uri />

The semantics of each is that it searches an implementation-defined set of locations for a module that matches the URI. If it finds one, it loads the functions declared in that library. If a namespace is given, it loads only the functions in the namespace provided.

We’d expect all implementations to be able to load libraries that only used XPath constructions. An XSLT processor might also be able to load XSLT constructions. An XQuery processor might also be able to load XQuery constructions.

We could define a library file format that allowed an implementor to provide several different implementations of a function, where the processor could choose the best one (in some implementation-dependent way). This would also give us a place to hang version numbers and other relevant metadata.

With that much in place, would it be more practical to use XPath extension modules?

Consider the following scenario. I want to use a URI relativization function (as requested in #269). Dimitre provided a pure XPath implementation, so we don’t actually have to implement it as a native function, we just have to make it easy to use. Imagine that EXPath.org (for example) provided a machine readable list of of libraries.

I run a hypothetical “expath” command to search the machine readlabe directory.

$ expath search relative
xpath uri-relativize -- returns the relative location between two URIs
xslt  doc-relative -- convenience functions for accessing “uncles”, “aunts”, etc.
xpath relative-rank -- funtions to score XML documents

It found three libraries that matched “relative”. That first one sounds promising.

$ expath show uri-relativize
The uri-relativize library provides uri-relativize(), an XPath
function that resolves one absolute URI relative to another.

That sounds like what I want, so I install it.

$ expath install uri-relativize
Downloading uri-relativize … installing … done.

Then in my stylesheet I simply add

<xsl:library href="uri-relativize"/>

Or in a language that only uses XPath, I add

import library at "uri-relativize"

and I can use the uri-relativize() function.

I think the important parts are that the implementation searches for libraries so that I don’t have to identify precisly where they were installed and that we somehow make it practical to use them without, though it pains me to say this, explicit namespace bindings.

Perhaps we could allow libraries to “inject” functions into the default function namespace, or we could have a function namespace search list and maybe libraries could extend that.

The format of the library might be something like this:

<library xmlns="xpath-library" name="uri-relativize" version="1.0.3"
         namespace="http://example.com/my/namespace">
<provides>
function uri-relativize($path1 as xs:anyURI, $path2 as xs:anyURI) as xs:anyURI
</provides>
<xpath version="3.0">
…xpath implementation…
</xpath>
<xslt version="4.0">
…xslt 4.0 implementation…
</xslt>
<xslt version="3.0">
…xslt 3.0 implementation…
</xslt>
<library>

But I’m sure if we looked closely at the metadata provided in other system’s packages, we’d see I’ve left a bunch of stuff out. You’d probably, for example, want some way of saying one package depends on another and having the processor load those automatically.

[dnovatchev]

I have implemented and been using "dynamic loading of XPath function libraries" since many years.

First one has to define what an "XPath function library" is.

I use this term for any set of maps, whose keys are strings and whose leaves are function items.

Thus an XPath function library is a text file with a contents like this (the general.xpath function library / module of FunXPath) :

let $id := function($arg as item()*) as item()*
{ $arg},

  $flip := function($f as function(item()*,item()*) as item()*) as function(item()*, item()*) as item()*
  {
    function($y as item()*, $x as item()*) {$f($x, $y)}
  },
  
  $compose := function($f as function(item()*) as item()*, $g as function(item()*) as item()*) as function(item()*) as item()*
  {
    function($x as item()*)
    {
      $f($g($x))
    }
    
  }

  return
    map {
      'id'      : $id,
      'flip'    : $flip,
      'compose' : $compose
    }

There is a single "extension function" written in XSLT for loading a set of function libraries: f:loadFunctionLibraries() and below is its code (I have also a similar one written in XQuery, but because my origin is from XSLT, I am using exclusively the below loader):

<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema" 
  xmlns:map="http://www.w3.org/2005/xpath-functions/map"
  xmlns:f="http://www.fxpath.org">
    
  <xsl:variable name="vBaseFXpathUri" 
    select="(environment-variable('FXpathDir')[normalize-space()], '..')[1]"/>
  
  <xsl:template name="loadFunctionLibraries" as="map(*)">
    <xsl:param name="pFilePaths" as="xs:string+"/>
      <xsl:sequence select="f:loadFunctionLibraries($pFilePaths)"/>
  </xsl:template>
    
  
  <xsl:function name="f:loadFunctionLibraries" as="map(*)" visibility="public">
    <xsl:param name="pLibrariesUris" as="xs:string+"/>
    
    <xsl:variable name="vAllText" as="xs:string+" select=
      "string-join(
                  (for $p in $pLibrariesUris
                    return unparsed-text($p)
                  ),
                  ',&#xA;'
                 )"/>
    
    <xsl:evaluate xpath="'map:merge((' || $vAllText || '))'">
      <xsl:with-param name="pBaseFXpathUri" select="$vBaseFXpathUri"/>
    </xsl:evaluate>
  </xsl:function>
</xsl:stylesheet>

The f:loadFunctionLibraries() loader function simply reads the text of the files that it receives as parameter(s), concatenates these texts and evaluates them dynamically. The result must be an XPath definition of one or more maps, and these maps are used to access any wanted functions from the function libraries.

Here is a very simple test of the above general.xpath function library. It is written in XSLT (for convenience), calls the loader to load the needed XPath function libraries, then evaluates any wanted function from them and can assert the correctness of the results:

<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:array="http://www.w3.org/2005/xpath-functions/array"
  xmlns:f="http://www.fxpath.org">
  <xsl:import href="../initialization/Loader.xsl"/>
  
  <xsl:output method="text" omit-xml-declaration="yes" indent="yes"/>
  
  <!-- ======= Load the file with the XPath functions to be tested ================== -->
  <xsl:variable name="vBaseFXpathUri" 
    select="(environment-variable('FXpathDir')[normalize-space()], '..')[1]"/>
 
  <xsl:variable name="vGen" as="map(*)"
    select="f:loadFunctionLibraries($vBaseFXpathUri || '/f/general.xpath')"/>
  
  <!-- ======== Load the Ops Xpath Function Library ================================ -->
    <xsl:variable name="vOps" as="map(*)"
    select="f:loadFunctionLibraries($vBaseFXpathUri || '/f/operators.xpath')"/>

  
  <xsl:template name="xsl:initial-template">
    <xsl:sequence select="$vGen?flip($vOps?div) (3, 6)"/>

  </xsl:template>
</xsl:stylesheet>

When this test is executed, the expected correct result for flip($ops?div)(3, 6) is produced:

2

An XPath function library can internally use another, separate XPath function library, as in the case of special-folds.xpath, which uses (dynamycally loads) both the folds.xpath and the operators.xpath function libraries:

(: =============== Include operators.xpath ============================:)
let $ops := Q{http://www.fxpath.org}loadFunctionLibraries#1(
concat($pBaseFXpathUri, '/f/operators.xpath')),
                                                            
(: ================ Include folds.xpath ===============================:)
  $folds := Q{http://www.fxpath.org}loadFunctionLibraries#1(
concat($pBaseFXpathUri, '/f/folds.xpath')
                                                            ), 

 (:    Special Folds
   ====================================================================:)
   $and := function ($booleans as xs:boolean*) as xs:boolean
{
  $folds?foldl_($ops?conjunction, true(), $booleans)
}, 

   $or := function ($booleans as xs:boolean*) as xs:boolean
{
  $folds?foldl_($ops?disjunction, false(), $booleans)
},

   $any := function($f as function(item()) as xs:boolean, $list as item()* ) as xs:boolean
{
  $or($list!$f(.))
},   

   $all := function($f as function(item()) as xs:boolean, $list as item()* ) as xs:boolean
{
  $and($list!$f(.))
},

  $sum := function ($nums as xs:numeric*) as xs:numeric
{
  $folds?foldl_($ops?add, 0, $nums)
},

  $product := function ($nums as xs:numeric*) as xs:numeric
{
  $folds?foldl_($ops?mult, 1, $nums)
},

  $minimum := function ($orderedItems as item()*) as item()
{
 $folds?foldl_(function($a, $b){if($a le $b) then $a else $b}, $orderedItems[last()], $orderedItems)
},

  $maximum := function ($orderedItems as item()*) as item()
{
 $folds?foldl_(function($a, $b){if($a ge $b) then $a else $b}, $orderedItems[last()], $orderedItems)
},

  $concat := function ($strings as xs:string*) as xs:string
{
 $folds?foldr_($ops?concatenation, '', $strings)
}

   return
      map { 
            'and'     : $and,
            'or'      : $or,
            'any'     : $any,
            'all'     : $all,
            'sum'     : $sum,
            'product' : $product,
            'minimum' : $minimum,
            'maximum' : $maximum,
            'concat'  : $concat
            }

Here is a test for the functions from the special-folds.xpath XPath function library:

<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:f="http://www.fxpath.org">
  <xsl:import href="../initialization/Loader.xsl"/>
  
  <xsl:output method="text" omit-xml-declaration="yes" indent="yes"/>
  
 <!-- ======= Load the file with the XPath functions to be tested ================== -->
  <xsl:variable name="vBaseFXpathUri" 
    select="(environment-variable('FXpathDir')[normalize-space()], '..')[1]"/>
  
  <xsl:variable name="vFuns" as="map(*)" 
    select="f:loadFunctionLibraries($vBaseFXpathUri || '/f/special-folds.xpath')"/>
  <!-- ============================================================================== -->
  
  <xsl:template name="xsl:initial-template">
    <xsl:sequence select="$vFuns?and((true(), true(), true()))"/>
    <xsl:sequence select="$vFuns?and((true(), true(), false()))"/>
    
    <xsl:sequence select="$vFuns?or((false(), false(), false()))"/>
    <xsl:sequence select="$vFuns?or((true(), false(), false()))"/>
    
    <xsl:sequence select="$vFuns?any(function($n) {$n idiv 2 * 2 eq $n}, (1,3,5))"/>
    <xsl:sequence select="$vFuns?any(function($n) {$n idiv 2 * 2 eq $n}, (1,3,6))"/>
    
    <xsl:sequence select="$vFuns?all(function($n) {$n idiv 2 * 2 eq $n}, (1,2,6))"/>
    <xsl:sequence select="$vFuns?all(function($n) {$n idiv 2 * 2 eq $n}, (0,2,6))"/>
    
    <xsl:sequence select="$vFuns?sum(1 to 10)"/>
    <xsl:sequence select="$vFuns?product(1 to 6)"/>
    
    <xsl:sequence select="$vFuns?minimum((6,8,3, -2))"/>
    <xsl:sequence select="$vFuns?minimum(('Alas', 'Baba', 'A2', 'Zebra'))"/>
    <xsl:sequence select="$vFuns?minimum((xs:date('2001-07-14'), current-date()))"/>
    
    <xsl:sequence select="$vFuns?maximum((6,8,3, -2))"/>
    <xsl:sequence select="$vFuns?maximum(('Alas', 'Baba', 'A2', 'Zebra'))"/>
    <xsl:sequence select="$vFuns?maximum((xs:date('2001-07-14'), current-date()))"/>
    
    <xsl:sequence select="'&#xA;'"/>
    <xsl:sequence select="$vFuns?concat(('a1', 'b2', 'c3'))"/>
  </xsl:template>
</xsl:stylesheet>

And here is the result of executing these tests:

true false false true false true false true 55 720 -2 A2 2001-07-14 8 Zebra 2022-11-26-08:00 
 a1b2c3

To summarize:

1. This has already been done almost entirely in pure XPath 3.1 and used for years.

2. There is a set of function libraries in FunXpath that I intentionally haven't brought to this Group's attention, because I really want to avoid being involved in arguments whether or not any particular function should be implemented as "standard" or not. It does exist in the FunXpath function libraries, and it is useful for me, and for me this is all that matters.

3. I propose that we only add as a standard function

   f:loadFunctionLibraries($pLibrariesUris as xs:string+) as map(*)
   with the semantics described above.

No magic words as "expath", "FunXPath" or whatever else. Just this facility loader and we must be incognizant of the ways and intentions of anyone's using it.

[ndw]

That's clever, but I'm not sure it's sufficient for a packaging system that would be useful to general users. But it might be closer than I thought. Putting the functions in a map certainly removes the namespace prefix issue, but is it too complex for casual users?

[michaelhkay]

I think we should look for a mechanism that's powerful enough to act as a polyfill. Specifically, once the capability is enabled in a processor and configured by the user, it should be possible for new convenience functions (in the fn or any other namespace) to be made available to users simply by posting an implementation of the function on a web site somewhere. By "convenience function" I mean a function that can be implemented in XSLT or XQuery (or ideally XPath).

On the XSLT side I think packages give us most of what's needed except for the ability to inject functions into the standard namespaces.

[dnovatchev]

Specifically, once the capability is enabled in a processor and configured by the user, it should be possible for new convenience functions (in the fn or any other namespace) to be made available to users simply by posting an implementation of the function on a web site somewhere.

This is exactly what the code shown in my 1st comment above does.

No need to "enable the capability" in any processor -- this just works Now and even in XPath 3.1 with every compliant XPath processor.

The user's experience is also the simplest possible one -- just call loadFunctionLibraries() for any needed functions (quite similar to the import statement in Python and in Javascript

Of course, if someone needs to put their time to nothing more useful than the goal of making the user's experience more compex and confusing, nobody can stop them.

[dnovatchev]

Putting the functions in a map certainly removes the namespace prefix issue, but is it too complex for casual users?

@ndw Can we define an XPath / XQuery /XSLT ver. 4 casual user as one who has no experience with maps?

Probably not, because in such case they will not understand maybe 80% of the functions available in these languages.

And more people probably misunderstand namespaces much more than maps, therefore the concept of a map containing functions may be much easier to them than a namespace containing functions...

Add to this the confusing fact that unlike namespaces in programming languages, XML namespaces cannot be composite (contain one another). This may even more confuse "casual users".

On the other side it is perfectly possible to have nested maps and in this they fully correspond to nested namespaces in ordinary programming languages, thus there exists no such confusion if maps are used as containers of functions.

Anyone who has used a programming language such as Java, C#, Python, Javascript, Typescript, C++, ... etc (almost all PLs have the concept of class/object) already knows what a map (or anything that has members/properties/attributes) is.

Or should we not suppose they are not familiar with the concept of a "function" ? But in this case how could such an user be a caller of any function?

[LinguaCelta]

Can we define an XPath / XQuery /XSLT ver. 4 casual user as one who has no experience with maps?
Probably not, because in such case they will not understand maybe 80% of the functions available in these languages.

Do you argue that 80% of functions are literally unusable if one doesn't fully understand maps? I find that very hard to accept. Perhaps understanding how some functions are implemented requires understanding maps, but that's not the same thing at all. I don't have to understand a function in detail in order to understand how I can use it. I don't even have to understand what maps are, in most cases. I only have to know what their structure is, so that I can provide them as an argument if necessary.

Honestly, I understand maps in theory, but I rarely if ever need to use them in my XSLT. As a result, using maps is still not easy or intuitive for me. Unlike most of the other members of this group, I am not an expert with decades of experience using XSLT, XQuery, and XPath. I'm not someone who's using XSLT every day, and I'm not using it for very complex projects. I suspect, therefore, that I have a perspective that is much closer to the "casual user".

You're quite obviously a brilliant programmer, Dimitre; I mean that sincerely. I am in awe of your skill. But I think perhaps that your depth of knowledge makes it very difficult for you to imagine the experience of someone who only uses these languages occasionally and for basic purposes. I therefore offer my perspective, which I hope will help you to understand how a less-experienced user might react to your proposal.

For me, your solution is breathtakingly clever, but it is not immediately easy to understand. The syntax you use - e.g.

$flip := function($f as function(item()*,item()*) as item()*) as function(item()*, item()*) as item()*
  {
    function($y as item()*, $x as item()*) {$f($x, $y)}
  }

is ingenious, but presents a high barrier for anyone who is not both experienced and a gifted programmer. It would be as though we insist that only people who qualify to join Mensa are allowed to write extension functions.

I believe that this would mean less engagement from people who might otherwise make very worthwhile contributions to the store of useful extension functions. It would also be offputting to users who want to understand exactly how the functions they are using work, particularly when they're debugging.

Some users (like me!) will enjoy the challenge of puzzling out how this code works. But other users will be cautious about using code they do not understand, especially if they are not confident programmers. I believe that it is important, where possible, to show respect to users who do not have the time, inclination, or ability to understand the complex and advanced features which are available. Some of these people may develop into experienced and enthusiastic users over time, if they are not put off at the beginning of their journey. Sometimes, catering to such users means that we should choose a solution that is easier to grasp, even if its implementation is not as clever or elegant.

[dnovatchev]

@LinguaCelta I completely agree with you, Bethany!

Do you know well both XML namespaces and maps? Which of the two is more complicated and confusing?

As for commenting on my "skill", please, be aware that the only code that is important in this comment is that of the loadFunctionLibraries() loader function.

All other code is like "data" for this function, and isn't directly related to the topic that we are discussing. I just used for this "data" existing code in one of my function libraries, as it is easy to copy + paste. This could equally be a simple function adding two items together. Once again, I do apologize if the "code as data" caused you any inconvenience - it wasn't meant to be read and understood, in the first place; it is just a "filler" / "data",

BTW, I am absolutely for Diversity in this group and would be happy to study (or even contribute to) the result of your action item.

[ChristianGruen]

I was wondering if we could use the existing syntax, but e.g. use a custom scheme to trigger dynamic imports:

import module namespace html = 'expath://parse-html';
html:parse(...)

It could be up to the query processor to decide if a module is imported from a repository, or if an internal implementation of the module is used. In addition, users wouldn't need to learn the differences between importing namespaces and libraries, and code wouldn't need to change if a formerly external module will be added internally in a new version of a processor.

[Arithmeticus]

I welcome the proposal, but I also don't understand what the real-world difference is between a library and a package. I apologize if my comments below blur two sides of a line I don't see.

The keyword at suggests to me that I can say where the package is. But if I read the proposal correctly, it is merely asking for a name, not a location. (A URIliteral only identifies, and does not locate.) So I recommend thinking of a different term, e.g., called or named.

What keyword should I use if I want to tell the processor where to find a particular library or library version that it may not know? Are we forced to use the shell? In the past, I have encountered many frustrations working with languages/platforms that deeply bury the location of dependency libraries. One of the beautiful things about <xsl:include> and <xsl:import> is that I have direct, immediate control over location. Yes, in some situations not having to declare where a package/library is can be a convenience. But in other cases it can be quite the inconvenience.

Perhaps more could be said about the mechanism a library/package publisher would use to declare what is available. Would it be like the collection file concept?

I am reminded that (AFAICT) <xsl:use-package> lacks an attribute to tell the processor where to find a desired package. I raised this question some months ago on Slack, and the only feedback I remember was that of Liam, who suggested it would be a good thing.

Is it worth considering supplying <xsl:package> and its XPath equivalent a node such as @source-location or <xsl:source-location> to declare one or more places where one could go to find new/authorized versions of the package/library?

If there is a question of security, LibraryImport and <xsl:use-package> could be supplied with a field/attribute to declare a hash value of the desired package. A provider's collection manifest could also declare what those hash values are.

We should bear in mind that in the QT languages the term namespace does not have exactly the same ramifications it does in other languages. Some approaching the language may wonder why you would need nothing more than simply import library [NAMESPACE]. I realize that this point touches only on how a design choice is communicated, not how it is determined.

Pointing to things. Name versus place or both. An old topic worth revisiting. Can we put it on the agenda of a weekly meeting for informal discussion?

[dnovatchev]

@Arithmeticus,

Is it worth considering supplying <xsl:package> and its XPath equivalent a node such as @source-location or <xsl:source-location> to declare one or more places where one could go to find new/authorized versions of the package/library?

This is exactly what the loadFunctionLibraries() loader function from my initial comment does.

Just a function! No need for special keywords or an "enabled processor" ...

[ndw]

I think there are lots of QT users who don’t use or understand maps. I’m thinking of the folks who started using XSLT a couple of decades ago because they were able to understand how the pattern matching works and could transform data with XSLT. It was a transformative experience for them. They don’t self-identify as programmers (I’ve met some who sternly reject the notion that they are programmers, even though we think they are.)

No longer did they need to find budget to get someone from the computing center to write code for them. They could do it themselves. They don’t especially care (perhaps) about the theoretical elegance of structured markup, it just lets them get work done on Their Important Thing. Maybe Their Important Thing isn’t technical at all, it just involves wrangling some data. Or some prose.

They changed the “1.0” at the top of their stylesheet to “2.0” when someone told them about xsl:for-each-group. They had managed to get Muenchian grouping working with a little help from the XSL List, but they never really understood it and they’ve never been able to get numbers to sort to the bottom, they’ve just been fixing that by hand. Using xsl:for-each-group made the stylesheet easier for them to understand.

They changed the “2.0” at the top of the stylesheet to “3.0” when someone started giving them JSON instead of XML and they worked out that fn:parse-json() would turn it back into XML so they could use XSLT to transform it.

Maybe they’ve seen maps. They’ve used them to set options on a function, maybe, but only with literal values and not by using them in variables.

I think these are important users because QT technologies have made readily available something that would otherwise be fundamentally inaccessible to them. If you or I or any members of the CG wanted to do something that we couldn’t do in XPath or XSLT or XQuery, we could write shell script or a Python or Haskell program to do it. We’re not like “ordinary people” and I think one of the really significant things about QT is that it is manifestly the case that it allows ordinary people to do transformative things.

If they need to calculate the phase of the moon and the instructions are:

<xsl:some-thing-we-invent name="moon-phase"/>
…
<xsl:value-of select="moon-phase(current-dateTime())"/>

I think that perfectly easy to explain. You could argue that the code you propose is “just as easy”:

<xsl:variable name="lunar" as="map(*)"
    select="f:loadFunctionLibraries('path/to/moon-phase.xpath')"/>
…
<xsl:value-of select="$lunar?moon-phase(current-dateTime())"/>

I’m not persuaded, but I grant that from a “use the library” perspective, it isn’t vastly more complicated. You probably don’t have to know that “?” is doing a map dereference to learn to use it.

I think using XPath directly to construct a map of function items as you propose is going to mean only 1 in a 1,000 XPath programmers are able to write libraries, but maybe that’s less important.

And none of this addresses the question of metadata for managing versions and dependencies or alternate implementations.

But if the group feels that making things easier for ordinary people to use isn’t an important goal, I guess that’s a coherent position.

[dnovatchev]

Nobody has described precisely the exact requirements, thus my feeling is that a lot of time is being spent here in a sub-optimal way ...

I’m not persuaded, but I grant that from a “use the library” perspective, it isn’t vastly more complicated. You probably don’t have to know that “?” is doing a map dereference to learn to use it.

Actually, if we want to provide the user with a set of functions in a set of predefined namespaces, this can be done with just a small (one or a few lines) adjustment of the current loadFunctionLibraries() function to make it return not a map but a sequence of functions items, in any wanted (different) namespaces.

I think using XPath directly to construct a map of function items as you propose is going to mean only 1 in a 1,000 XPath programmers are able to write libraries, but maybe that’s less important.

As pointed above, we can as easily return a set of functions.

And none of this addresses the question of metadata for managing versions and dependencies or alternate implementations.

A map can have different "branches" for different versions. We just need to have a precise definition of the metadata, and anyone can implement this as a part of a (nested) map. As I mentioned in a previous comment, a map (tree) exactly corresponds to a hierarchical namespace structure, where any version or other metadata can be represented.

But if the group feels that making things easier for ordinary people to use isn’t an important goal, I guess that’s a coherent position.

As pointed above, all of these "easiness" requirements can easily (really easily) be implemented by just slightly modifying the basic map representation.

Norm, I would greatly appreciate it if you or someone else provide the exact requirements, then I will gladly give you the loadFunctionLibraries() function that implements these.

Let's not artificially split into "elite" vs. "people" and do our work responsibly, as we are supposed to.

[michaelhkay]

As far as the spec is concerned, I think we could say for XSLT:

A stylesheet may contain an xsl:use-package declaration specifying the package name http://qt4cg.org/xpath-functions, optionally with a version such as package-version="4.*". A conforming implementation must accept this declaration, and should handle it by providing access to a library of function implementations for functions in the core namespaces (fn, map, array, math) that have been specified by the community group but which are not yet available natively in the version of the XSLT processor currently in use. The processor should allow static bindings to such functions if an implementation exists; stylesheet authors may prefer to write code that works whether or not an implementation exists by using fn:function-lookup to obtain such functions dynamically.

We can't do exactly the same thing for XQuery because the nearest equivalent is "import module" and modules are tied to namespaces (and lack a version mechanism). But we can say that queries are allowed to do

import module fn="http://www.w3.org/2005/xpath-functions";

with no location hint, and implementations should recognise this as providing access to additional functions in the fn namespace; similar declarations are allowed for the map, array, and math namespaces. Of course the set of additional functions may be empty.

A secondary question is whether we as a community group should provide reference implementations that can be invoked using this mechanism. I suggest (at least initially) not; we'll leave it to implementors to do that.

With this mechanism in place, it becomes much easier to deploy implementations of new functions, because it doesn't mean releasing a new version of the XSLT or XQuery processor; and over time we can create a mechanism to allow new functions to be added without publishing a new version of the language specifications.

[dnovatchev]

A secondary question is whether we as a community group should provide reference implementations that can be invoked using this mechanism. I suggest (at least initially) not; we'll leave it to implementors to do that.

I completely agree that any such mechanism goes way beyond the current goals and tasks of the CG (BTW, do we have a document describing these?).

As a developer of XPath functionality I will never be interested in having and using such mechanism.

Moreover, I feel that this would lead to increased confusion amongst developers:

• Which are the really "standard" functions?
• What functions belong to what namespace?
• What is the exact set of functions that a particular namespace provides?
• Which is the processor that implements all functions that I want to use?
• Which is the processor that implements version m of function X? I don't want version n ?
• If I commit to a particular processor because it implements "function X" will I not pay a too-high price of losing portability?
• How to promote my own functions?
• Is there anything anymore that can be considered to be the "standard" / "core" function library that any implementation must provide?

These are just the first questions that spontaneously come to mind. There might be more questions with uncertain answers...

Thus for me it is absolutely clear: I don't want such chaos to exist and will not be part in / of it.

Compare this to the perfectly clear alternative for any developer of providing their own XPath function libraries that are written in standard XPath and use standard XPath functions, thus are executable on any XPath-compliant processor, be it XQuery, XSLT, pure XPath or any other X* (name it here).

[ChristianGruen]

I think we shouldn’t make functions in the fn, array and map namespace optional, and only use dynamical/implementation-defined import mechanisms for modules in other namespaces. It would get confusing if we can’t rely on XPath/XQuery/XSLT 4 implementations to provide support for the same standard functions.

[michaelhkay]

I think the web user community (especially in the Javascript world) has come to value a situation where specifications evolve rather more incrementally than a big bang major version every 7-10 years. There's logically no reason why individual functions shouldn't be endorsed and published incrementally by the CG.

User expectations are also set by the "Polyfill" capabilities of Javascript, where implementations of functions can be made available without needing to upgrade the Javascript engine.

On both fronts (publishing function specifications incrementally, and making implementations available without the need for a software upgrade), I think we can serve users a lot better than we did 20 years ago.

For a user there are many ways of managing the "chaos". One way is just to be conservative, and not use anything that's not consolidated yet into a "major version". But in the Javascript world there's lots of data available on the web about what has been implemented where. If you need functionality that isn't available everywhere, there are plenty of mechanisms available for deciding whether and when to use it, and it's entirely the users' choice.

[michaelhkay]

Another issue that led to a stimulating discussion but shows no signs of leading to a concrete proposal for the WG to consider. I therefore think it's time to close it.

[ndw]

The CG decided to close this issue without any further action at meeting 069.