XSLT 3.0: xsl:package and xsl:use-package

[jcronk]

It seemed to me that #127 is too broad, plus it's pretty old, so I wanted to pull this out and call a little more attention to it. There are two points here:

1. XSpec won't run on xsl:package files. Since the new package concept allows for reusable library code, people - I hope - would particularly want to write tests for XSL packages.
2. XSpec errors when running it on a stylesheet that contains xsl:use-package. XSpec uses xsl:import to get the stylesheet under test, and using xsl:import is an error when trying to import a stylesheet which uses a package (see dependencies between packages in the spec).

I'm not sure how you get around the conflict between xsl:import and xsl:use-package, though, because I'm sure it's xsl:import and not xsl:include for a reason.

Package: using the example package here

<?xml version="1.0" encoding="UTF-8"?>
<xsl:package
    name="http://example.org/complex-arithmetic.xsl"
    package-version="1.0"
    version="3.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:f="http://example.org/complex-arithmetic.xsl">
    
    <xsl:function name="f:complex-number" 
        as="map(xs:integer, xs:double)" visibility="public">
        <xsl:param name="real" as="xs:double"/>
        <xsl:param name="imaginary" as="xs:double"/>
        <xsl:sequence select="map{ 0:$real, 1:$imaginary }"/>
    </xsl:function>
    
    <xsl:function name="f:real" 
        as="xs:double" visibility="public">
        <xsl:param name="complex" as="map(xs:integer, xs:double)"/>
        <xsl:sequence select="$complex(0)"/>
    </xsl:function>
    
    <xsl:function name="f:imag" 
        as="xs:double" visibility="public">
        <xsl:param name="complex" as="map(xs:integer, xs:double)"/>
        <xsl:sequence select="$complex(1)"/>
    </xsl:function>
    
    <xsl:function name="f:add" 
        as="map(xs:integer, xs:double)" visibility="public">
        <xsl:param name="x" as="map(xs:integer, xs:double)"/>
        <xsl:param name="y" as="map(xs:integer, xs:double)"/>
        <xsl:sequence select="
            f:complex-number(
            f:real($x) + f:real($y), 
            f:imag($x) + f:imag($y))"/>
    </xsl:function>
    
    <xsl:function name="f:multiply" 
        as="map(xs:integer, xs:double)" visibility="public">
        <xsl:param name="x" as="map(xs:integer, xs:double)"/>
        <xsl:param name="y" as="map(xs:integer, xs:double)"/>
        <xsl:sequence select="
            f:complex-number(
            f:real($x)*f:real($y) - f:imag($x)*f:imag($y),
            f:real($x)*f:imag($y) + f:imag($x)*f:real($y))"/>
    </xsl:function>    
</xsl:package>

Stylesheet with use-package:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:f="http://example.org/complex-arithmetic.xsl"
    exclude-result-prefixes="xs"
    version="3.0">
    <xsl:use-package name="http://example.org/complex-arithmetic.xsl" version="3.0"/>
    <xsl:template name="xsl:initial-template">
        <xsl:variable name="val1" select="f:complex-number(2,5)"/>
        <xsl:variable name="val2" select="f:complex-number(3,2)"/>
        <xsl:variable name="sum" select="f:add($val1, $val2)"/>
        <xsl:sequence expand-text="yes">The sum of the two numbers is {f:real($sum)} + {f:imag($sum)}i</xsl:sequence>
    </xsl:template>
</xsl:stylesheet>

XSpec test:

<?xml version="1.0" encoding="UTF-8"?>
<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    stylesheet="use-package-example.xsl">
    <x:scenario label="Scenario for testing function initial-template">
        <x:call template="xsl:initial-template"/>
        <x:expect label="it returns the expected message" select="'The sum of the two numbers is 5 + 7i'"/>
    </x:scenario>
</x:description>

Here's the saxon config telling it how to find the package:

<?xml version="1.0" encoding="UTF-8"?>
<configuration xmlns="http://saxon.sf.net/ns/configuration">
    <xsltPackages>
        <package name="http://example.org/complex-arithmetic.xsl" version="3.0" sourceLocation="package-example.xsl" />       
    </xsltPackages>  
</configuration>

In Oxygen XML, everything shows up without errors, and the test stylesheet will run and produce output, but if you try to run the XSpec test, you will get:

run-xslt-test:
     [echo] Running XSLT Tests...
     [java] Error on line 9 column 115 of package-test-compiled.xsl:
     [java]   XTSE3008: xsl:use-package cannot appear in an imported stylesheet
     [java] xsl:use-package cannot appear in an imported stylesheet

[cmarchand]

This one is quite complicated. First, we must change the xsl:import by a xsl:use-package in the compiled XSpec. This can be done without problem, with some limitations :

• private functions / templates / modes won't be accessible, so we can not call them to test them.
• a special configuration must be provided to Saxon, to explain where packages are located. See http://saxonica.com/documentation/index.html#!configuration/configuration-file for defining a package in a configuration.

For privacy work-around, we may forced to modify tested XSL visibilities, to be able to test them.

[AirQuick]

I'm not sure how you get around the conflict between xsl:import and xsl:use-package, though, because I'm sure it's xsl:import and not xsl:include for a reason.

Without xsl:import, global x:param and x:variable cannot overwrite the ones in the stylesheet being tested.

So, with the given example, what if XSpec is written like this

<x:description stylesheet="package-example.xsl">
   <x:scenario>
      <x:call function="f:complex-number">
...

and it is compiled into like this?

<xsl:stylesheet>
   <xsl:use-package name="http://example.org/complex-arithmetic.xsl" version="1.0"/>
   <xsl:import href=".../src/compiler/generate-tests-utils.xsl"/>
...

Will it serve the majority of use cases?
Note that XSpec (CLI and Ant) already allows users to specify a Saxon configuration file to be used when running the compiled stylesheet.

Please be warned that I have zero experience with XSLT packages and am just looking at the given example :)

[jcronk]

Thanks for taking the time to look at this. :)

Without xsl:import, global x:param and x:variable cannot overwrite the ones in the stylesheet being tested.

Okay, that makes sense. I've periodically been going through the source code to try and see how it works, but there's a lot in there to follow, and what XSpec does is completely unlike anything I've tried to do in XSLT.

So, with the given example, what if XSpec is written like this

<x:description stylesheet="package-example.xsl">
   <x:scenario>
      <x:call function="f:complex-number">
...

and it is compiled into like this?

<xsl:stylesheet>
   <xsl:use-package name="http://example.org/complex-arithmetic.xsl" version="1.0"/>
   <xsl:import href=".../src/compiler/generate-tests-utils.xsl"/>
...

Will it serve the majority of use cases?

If I'm understanding, this would allow me to test what's in the package itself (which is a good thing!) but I don't see a way that it would allow me to test any contents of the stylesheet that is using the package. With such a simple set of examples, it's hard to come up with a convincing use case, but suppose we had something like:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:f="http://example.org/complex-arithmetic.xsl"
    exclude-result-prefixes="xs"
    version="3.0">
    <xsl:use-package name="http://example.org/complex-arithmetic.xsl" version="3.0"/>
    <xsl:template match="MyNumbers">
        <xsl:variable name="real-numbers" as="xs:double*" select="Real/nbr"/>
        <xsl:variable name="imaginary-numbers" as="xs:double*" select="Imaginary/nbr"/>
        <xsl:complex-numbers select="for $r in $real-numbers, $i in $imaginary-numbers return f:complex-number($r, $1)"/>
        <xsl:sequence select="fold-left($complex-numbers, f:complex-number(0,0), function ($result, $input) { f:add($result, $input) })"/>
    </xsl:template>
</xsl:stylesheet>

So then I'd want to create an XSpec test for this (especially since I wrote this out without a reference and it could be full of syntax errors or have other problems) that verifies that something like:

<MyNumbers>
    <Real>
        <nbr>1</nbr>
        <nbr>4</nbr>
        <nbr>9</nbr>
    </Real>
    <Imaginary>
        <nbr>8</nbr>
        <nbr>2</nbr>
        <nbr>7</nbr>
    </Imaginary>
</MyNumbers>

when given as input to the template will yield the correct sum.

I'd absolutely be willing to try to add this functionality, but I'm not sure where I'd even begin, especially here - since you need the overrides from xsl:import, it's not obvious to me how to accomplish the same thing without running into the error of importing a stylesheet that uses a package.

Maybe as you're transitioning to a 3.0 codebase (I think I saw something about there being plans to make XSpec 2.0 based completely on XSLT 3.0?) there's a solution that uses packages in XSpec's compiler code...somehow? Could the stylesheet under test maybe be recast as some sort of temporary package, allowing you to set visibility or override rules to get the same result as you can with xsl:import? Probably not practical, but all I have are half-formed ideas and speculation at this point.

[AirQuick]

What if XSpec takes the following measures when it finds xsl:use-package in the stylesheet being tested?

• Stop inserting global xsl:param (generated from global x:param) into the compiled stylesheet.
• Stop writing <xsl:import href="stylesheet-to-be-tested.xsl" /> in the compiled stylesheet. Use xsl:include instead.
• Do not run the compiled stylesheet directly. Instead, run it via fn:transform() with stylesheet-params option (generated from global x:param)

[jcronk]

That sounds like a promising approach.

[AirQuick]

fn:transform() has its own limitations and uncertainty.
For now, what I come up with is a combination of these:

• Implement xsl:use-package in XSpec
• Include your main stylesheet from a stub main package where the named mode rules are redirected to the unnamed mode rules:

  <xsl:package declared-modes="no" exclude-result-prefixes="#all" name="my-main-package" version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
     <xsl:include href="main-stylesheet.xsl" />
     <xsl:expose component="*" names="*" visibility="public" />
     <xsl:template match="document-node() | node() | attribute()" mode="my-mode">
        <xsl:apply-templates select="." />
     </xsl:template>
  </xsl:package>

• Test the stub main package from XSpec using a named mode:

  <x:description stylesheet="main-package.xsl" xmlns:x="http://www.jenitennison.com/xslt/xspec">
    <x:scenario label="using named mode">
       <x:context mode="my-mode" />
       <x:scenario label="test something">
          <!-- test.xml has <MyNumbers> -->
          <x:context href="test.xml" />
          <x:expect label="expect something" select="map{0:4.2e1,1:5.1e1}" />
       </x:scenario>
    </x:scenario>
  </x:description>

Again, my knowledge of packaging is extremely limited. There may be a better way.

[cmarchand]

I think we could ask Michael Kay for help on this particular point. He previously helped XSpec project with the Coverage java code.
Did you commit on a particular branch, on which Mike could comment ?

[AirQuick]

No, I haven't made any commit. I just tested my previous comment with the given example.

What I probably need to know first (except for learning the packages itself...) would be a best practice when you have this working A.xsl

<!-- A.xsl -->
<xsl:stylesheet ...>
   <!-- Usual xsl:param, xsl:variable, xsl:function, xsl:template etc
      including unnamed mode xsl:template -->
</xsl:stylesheet>

and working B.xsl

<!-- B.xsl -->
<xsl:stylesheet ...>
   <xsl:import href="A.xsl" />
   <!-- Use stuff in A.xsl, possibly overriding some global xsl:param -->
</xsl:stylesheet>

and when xsl:use-package is introduced in A.xsl.

<!-- A.xsl revised -->
<xsl:stylesheet ...>
   <!-- NEW: Use package -->
   <xsl:use-package ... />

   <!-- The rest intact -->
</xsl:stylesheet>

What is the best practice to rewrite A.xsl and/or B.xsl so that both continue working?

Maybe the answer is "it depends"... But I can't go further, as I haven't used the packages myself. My previous comment just came up from Note ("If existing XSLT code has been written to use template rules in the unnamed mode...") in the spec.

[cmarchand]

I copy here mail exchanges with Michael Kay, on this topic :

Interesting. It's difficult to advise because I don't really know enough about the constraints imposed by the current architecture.

I'd be inclined to say: if the stylesheet under test uses packages, then it must itself be (or be treated as) a package, which means XSpec should reference it using xsl:use-package rather than using xsl:import.

But I'd also suggest that static linking of the stylesheet under test seems an unnatural thing to do when you've got fn:transform() to achieve dynamic invocation. Under Saxon-JS a lot of our test driver for the XSLT3 test suite is written in XSLT, and it invokes the stylesheet under test using fn:transform(). This seems cleaner because there's no risk of unintended overrides, plus you can test many different stylesheets in a single run with no conflicts of global variables etc. (But I guess XSpec is focussed on testing one stylesheet...)

Static variables and parameters are also worth considering here. Although it exposes a bit of a hole in the packaging spec, where it's rather unclear how xsl:use-package is supposed to supply values of static parameters in the used stylesheet.

Just to mention another issue that may turn out to be related: static parameters. A particular challenge here is that static coupling of the test driver to the stylesheet under test (whether using xsl:import or xsl:use-package) doesn't allow testing of different values of static parameters in a single run. Dynamic invocation using fn:transform() could solve this problem, as well as the xsl:use-package problem.

Mike

On 28 Feb 2020, at 08:27, Christophe Marchand wrote:

Hello Mike,

sorry for this off-list message, but with XSpec guys, we need advices on how to implement unit tests on packages.

I've ask O'Neil and Debby last year at Prague, be we weren't ready to implement unit tests on packages - we had just discover that it'll be difficult...

Actually, when we test a XSLT, we generate a XSL that imports the one to test. It's ok, if the module to test is not a package.

To test packages, we do not know what is the best practise, to create a wrapper, to transform the package in a standard module, or another possibility.

Could you spend a small part of your spare time to give us some advices on the way to do it ? We have an issue open on github for this : #762

[AirQuick]

Thanks, @cmarchand

After experimenting xsl:include and xsl:use-package which are relatively trivial and work to some extent, I'm inclined toward fn:transform() for practical reasons. The biggest reason is that xsl:include and xsl:use-package benefit only the package users. Another reason is that we'll not be able to have a plenty of test cases instantly.

I already have an implementation of fn:transform(). I'll open its pull request after clearing housekeeping tasks, unless someone comes up with a better idea. Since it breaks compatibility here and there, introducing it as a replacement of the current version is clearly impossible. It will be an opt-in experimental feature.

[cmarchand]

Yes, I agree with you, fn:transform() should do the job, but it's an important amount of work to rewrite everything.
A question stays un-answered : is a private function in a package should be unit-tested ? On one hand, I'd be inclined to test everything ; on the other hand, a private function is not an API, and does not offer service to package end-users ; so it is a package implementation detail, and there is no imperial reason to unit-test it.
But, if everyone is Ok with non-testing private package function, this must be documented, as it will generate an error at runtime. We should clearly avoid changing visibility of functions only for unit-tests, as we sometime do in Java.

[AirQuick]

@jcronk @cmarchand , I submitted an experimental implementation: #892

[jcronk]

I tried testing the code on the pull request, but it's entirely possible that I missed a step or I'm doing something wrong. I initially tried it with several xslt packages, got an error, decided to take a step back and test it with something more minimal, and I got the same error. Here's the minimal example I tried it with:

Package:

<xsl:package name="com.company.utils.array" package-version="0.1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:array-ext="http://www.company.com/utils/array"
    xmlns:array="http://www.w3.org/2005/xpath-functions/array" xmlns:fn="http://www.w3.org/2005/xpath-functions"
    xmlns:map="http://www.w3.org/2005/xpath-functions/map" exclude-result-prefixes="#all" expand-text="true" version="3.0">
    <xsl:expose component="function" names="array-ext:*" visibility="public"/>
    <xsl:function name="array-ext:interlace" as="array(*)*">
        <xsl:param name="first-item" as="item()*"/>
        <xsl:param name="second-item" as="item()*"/>
        <xsl:sequence
            select="
                for-each-pair($first-item, $second-item, function ($x, $y) {
                    [$x, $y]
                })"
        />
    </xsl:function>
    <xsl:function name="array-ext:shallow-flatten">
        <xsl:param name="ary" as="array(*)"/>
        <xsl:sequence select="(1 to array:size($ary)) ! $ary(.)"/>
    </xsl:function>
</xsl:package>

Stylesheet:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:array-ext="http://www.company.com/utils/array" exclude-result-prefixes="#all" version="3.0">
    <xsl:output method="text" omit-xml-declaration="yes"/>
    <xsl:use-package name="com.company.utils.array" package-version="0.1.0"/>
    <xsl:variable name="nested-array" select="[1, 2, 3, 4, [5, 6], 7, [8, [9, 10]]]"/>
    <xsl:template name="output">
        <xsl:variable name="flattened-array" select="array-ext:shallow-flatten($nested-array)"/>
        <xsl:sequence select="serialize($flattened-array, map {'method': 'json'})"/>
    </xsl:template>
</xsl:stylesheet>

and config:

<configuration xmlns="http://saxon.sf.net/ns/configuration" edition="EE">
    <xsltPackages>
        <package name="com.company.utils.array" version="0.1.0" sourceLocation="array.xsl" />       
    </xsltPackages>
</configuration>

In Oxygen XML, I'm getting:
[java] Fatal error during transformation: java.lang.ClassCastException: net.sf.saxon.value.StringValue cannot be cast to net.sf.saxon.om.NodeInfo

When I run XSpec from the terminal, I get a somewhat more useful stack trace:

java.lang.RuntimeException: Internal error evaluating template rule  at line 17 in module file:/<snipped>/./xspec/Main-compiled.xsl
        at com.saxonica.ee.bytecode.ByteCodeCandidate.process(ByteCodeCandidate.java:153)
        at net.sf.saxon.expr.instruct.NamedTemplate.expand(NamedTemplate.java:265)
        at net.sf.saxon.Controller.callTemplate(Controller.java:2547)
        at net.sf.saxon.s9api.Xslt30Transformer.callTemplate(Xslt30Transformer.java:750)
        at net.sf.saxon.Transform.processFile(Transform.java:1252)
        at net.sf.saxon.Transform.doTransform(Transform.java:782)
        at net.sf.saxon.Transform.main(Transform.java:81)
Caused by: java.lang.RuntimeException: Internal error evaluating template rule  at line 30 in module file:/<snipped>/./xspec/Main-compiled.xsl
        at com.saxonica.ee.bytecode.ByteCodeCandidate.process(ByteCodeCandidate.java:153)
        at net.sf.saxon.expr.instruct.NamedTemplate.expand(NamedTemplate.java:265)
        at net.sf.saxon.expr.instruct.CallTemplate.process(CallTemplate.java:353)
        at net.sf.saxon.expr.instruct.CallTemplate.processLeavingTail(CallTemplate.java:411)
        at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:687)
        at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:151)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:337)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:284)
        at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:151)
        at net.sf.saxon.expr.instruct.ResultDocument.processInstruction(ResultDocument.java:480)
        at net.sf.saxon.Configuration.processResultDocument(Configuration.java:2273)
        at com.saxonica.config.EnterpriseConfiguration.processResultDocument(EnterpriseConfiguration.java:1938)
        at net.sf.saxon.expr.instruct.ResultDocument.process(ResultDocument.java:387)
        at net.sf.saxon.expr.instruct.ResultDocument.processLeavingTail(ResultDocument.java:373)
        at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:687)
        at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:151)
        at com.saxonica.ee.bytecode.ByteCodeCandidate.process(ByteCodeCandidate.java:141)
        ... 6 more
Caused by: java.lang.RuntimeException: Internal error evaluating template rule  at line 286 in module file:/<snipped>/xspec-testing/src/compiler/generate-tests-utils.xsl
        at com.saxonica.ee.bytecode.ByteCodeCandidate.process(ByteCodeCandidate.java:153)
        at net.sf.saxon.expr.instruct.NamedTemplate.expand(NamedTemplate.java:265)
        at net.sf.saxon.expr.instruct.CallTemplate.process(CallTemplate.java:353)
        at net.sf.saxon.expr.instruct.CallTemplate.processLeavingTail(CallTemplate.java:411)
        at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:687)
        at net.sf.saxon.expr.LetExpression.processLeavingTail(LetExpression.java:709)
        at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:687)
        at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:151)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:337)
        at net.sf.saxon.expr.instruct.ElementCreator.processLeavingTail(ElementCreator.java:284)
        at net.sf.saxon.expr.instruct.Block.processLeavingTail(Block.java:687)
        at net.sf.saxon.expr.instruct.Instruction.process(Instruction.java:151)
        at com.saxonica.ee.bytecode.ByteCodeCandidate.process(ByteCodeCandidate.java:141)
        ... 22 more

The line number in Main-compiled.xsl that it's referencing is an xsl:message containing the scenario label, and the line number in generate-tests-utils.xsl is an xsl:param declaration under the test:report-sequence template.

I'm not sure if this is an issue with the code or an issue with something I'm doing. In any case, I don't immediately see where the ClassCastException could be coming from, unless the line numbers it's giving are only approximations.

Up until the error hits, it looks like everything is working correctly - it seems to find the package and go through the initial steps just fine. @AirQuick , if you see that I'm making a mistake somewhere, please let me know.

[AirQuick]

@jcronk Thanks for testing out the pr. Can you let us know these?

• Saxon version
• .xspec file
• Corrected version of stylesheet. Looks like the current version has an error:

  C:\test>java -jar SaxonEE9-9-1-7J\saxon9ee.jar -it:output -xsl:test.xsl -config:config.xml
  Error at char 10 in xsl:sequence/@select on line 9 column 85 of test.xsl:
    SERE0023: JSON output method cannot handle a sequence of more than one item

• Did you run test/external_*.xspec successfully in the same environment?

[jcronk]

Sure, sorry about that.

Corrected stylesheet:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:array-ext="http://www.company.com/utils/array" exclude-result-prefixes="#all" version="3.0">
    <xsl:output method="text" omit-xml-declaration="yes"/>
    <xsl:use-package name="com.company.utils.array" package-version="0.1.0"/>
    <xsl:variable name="nested-array" select="[1, 2, 3, 4, [5, 6], 7, [8, [9, 10]]]"/>
    <xsl:template name="output">
        <xsl:variable name="flattened-array" select="array-ext:shallow-flatten($nested-array)"/>
        <xsl:sequence select="serialize($flattened-array, map {'method': 'adaptive'})"/>
    </xsl:template>   
</xsl:stylesheet>

Should return something like

1
2
3
4
[5,6]
7
[8,[9,10]]

From Oxygen, the version of Saxon is 9.8.0.12 (EE), and the version I have XSpec pointing to on the command line is 9.8.0.14 (EE).

Here's the xspec file:

<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec" stylesheet="Main.xsl" run-as="external">
    <x:variable name="x:saxon-config" select="'SaxonConf.xml'"/>
    <x:scenario label="Scenario for testing function output">
        <x:call template="output"/>
        <x:expect label="it does something" select="'foo'"/>
    </x:scenario>
</x:description>

I was expecting a failure here, obviously - I usually start with a test I know will fail so I can confirm what XSpec is getting for the result. I thought maybe the string in the x:expect/@select attribute might have been what it was talking about in the error message, but if I put something in the content of x:expect it does the same thing.

I didn't think to run the external_* tests, but I just tried that from the command line and they were all successful; example:

Saxon script not found, invoking JVM directly instead.

Creating Test Stylesheet...


Running Tests...
Testing with SAXON EE 9.8.0.14
Calling a public function in the package
should be possible

Formatting Report...
passed: 1 / pending: 0 / failed: 0 / total: 1
Report available at test/xspec/external_xslt-package_arith-result.html
Done.

[AirQuick]

<x:variable name="x:saxon-config" select="'SaxonConf.xml'"/>

This @select should be @href like this:

    <x:variable name="x:saxon-config" href="SaxonConf.xml"/>

or x:variable should contain a configuration element like this:

xspec/test/external_xslt-package_filter_use_static-param.xspec

Lines 6 to 16 in f922358

	<x:variable name="x:saxon-config">
	<configuration xmlns="http://saxon.sf.net/ns/configuration">
	<xsltPackages>
	<package name="http://example.org/filter.xsl"
	sourceLocation="{resolve-uri('xslt-package/filter/package/filter.xsl', $x:xspec-uri)}"
	version="1.0">
	<withParam name="filter" select="'location = ''UK'''" />
	</package>
	</xsltPackages>
	</configuration>
	</x:variable>

I pushed some changes to the pr so that this kind of error is detected:

C:\xspec>bin\xspec.bat M:\test\test.xspec
...
Running Tests...
Testing with SAXON EE 9.9.1.7
Scenario for testing function output
ERROR: Invalid $x:saxon-config

[jcronk]

Thanks! I corrected the variable to use href and now both of my basic examples work. So far so good.

[jcronk]

@AirQuick Here's a new one for you:

It looks like when I try to test higher-order functions, I get an error. Here's an example of one:

Package file:

<xsl:package name="hof.reductions" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:hof="http://www.company.com/hof"
    xmlns:math="http://www.w3.org/2005/xpath-functions/math" exclude-result-prefixes="xs math" version="3.0">
    <xsl:function name="hof:reductions" visibility="public">
        <xsl:param name="seq"/>
        <xsl:param name="init"/>
        <xsl:param name="func"/>
        <xsl:choose>
            <xsl:when test="empty($seq)">
                <xsl:sequence select="$seq"/>
            </xsl:when>
            <xsl:otherwise>
                <xsl:sequence select="$init, hof:reductions(tail($seq), $func($init, head($seq)), $func)"/>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:function>
</xsl:package>

XSpec file:

<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec" xmlns:hof="http://www.company.com/hof"
    stylesheet="reductions.xsl" run-as="external">
    <x:variable name="x:saxon-config" href="../../SaxonConf.xml"/>
    <x:scenario label="Scenario for testing function reductions">
        <x:call function="hof:reductions">
            <x:param name="seq" select="1, 2, 3, 4"/>
            <x:param name="init" select="0"/>
            <x:param name="func" select="function($res, $inp) { $res + $inp }"/>
        </x:call>
        <x:expect label="it returns a seq of each step in the reduction" select="0, 1, 3, 6"/>
    </x:scenario>
</x:description>

When I run this (same setup as before), I get this (the whole stack trace is really long, so this is just the first bit)

run-xslt-test:
     [echo] Running XSLT Tests...
     [java] Testing with SAXON EE 9.8.0.12
     [java] Scenario for testing function reductions
     [java] java.lang.IllegalStateException: A function created under one Configuration cannot be called under a different Configuration
     [java] 	at com.saxonica.functions.hof.UserFunctionReference$BoundUserFunction.makeNewContext(UserFunctionReference.java:373)
     [java] 	at net.sf.saxon.functions.SystemFunction.dynamicCall(SystemFunction.java:451)
     [java] 	at net.sf.saxon.functions.ApplyFn.call(ApplyFn.java:159)
     [java] 	at net.sf.saxon.expr.FunctionCall.iterate(FunctionCall.java:547)
     [java] 	at net.sf.saxon.value.MemoClosure.makeSequence(Unknown Source)
     [java] 	at net.sf.saxon.value.MemoClosure.iterate(Unknown Source)
     [java] 	at net.sf.saxon.expr.VariableReference.iterate(VariableReference.java:540)
     [java] 	at net.sf.saxon.expr.instruct.BlockIterator.next(BlockIterator.java:49)
     [java] 	at net.sf.saxon.expr.instruct.BlockIterator.next(BlockIterator.java:51)
     [java] 	at net.sf.saxon.value.SequenceExtent.<init>(SequenceExtent.java:86)
     [java] 	at net.sf.saxon.value.SequenceExtent.makeSequenceExtent(SequenceExtent.java:111)
     [java] 	at net.sf.saxon.om.SequenceTool.toGroundedValue(SequenceTool.java:47)
     [java] 	at net.sf.saxon.s9api.Xslt30Transformer.callFunction(Xslt30Transformer.java:842)
     [java] 	at net.sf.saxon.functions.TransformFn.call(TransformFn.java:756)
     [java] 	at net.sf.saxon.expr.FunctionCall.iterate(FunctionCall.java:547)
     [java] 	at net.sf.saxon.expr.Expression.evaluateItem(Expression.java:843)
     [java] 	at net.sf.saxon.expr.parser.Evaluator$3.evaluate(Evaluator.java:73)
     [java] 	at net.sf.saxon.expr.parser.Evaluator$3.evaluate(Evaluator.java:70)
     [java] 	at net.sf.saxon.expr.SystemFunctionCall.evaluateArguments(SystemFunctionCall.java:448)
     [java] 	at net.sf.saxon.expr.FunctionCall.iterate(FunctionCall.java:545)
     [java] 	at net.sf.saxon.expr.LetExpression.iterate(LetExpression.java:501)
     [java] 	at net.sf.saxon.value.MemoClosure.makeSequence(Unknown Source)
     [java] 	at net.sf.saxon.value.MemoClosure.iterate(Unknown Source)

There's a comment here in the Saxon documentation that

From Saxon 9.8.0.8, the vendor-options parameter allows an entry of the form "vendor-options": map {QName('http://saxon.sf.net/', 'configuration'): $config} where $config is the root node of a configuration file expressed as an XML tree. If this option is present, the target transformation is run in a different configuration from the calling application, so that settings in the configuration file do not affect the environment of the caller.

So this may not even be a bug - what I found after experimenting was that if I run XSpec from the terminal and I set SAXON_CUSTOM_OPTIONS with -config:MyConfig.xml and then remove the x:variable in the XSpec, it seems to run fine, but I'm not sure how I'd do that from inside Oxygen.

I've run into a couple of other things I wasn't expecting, but I'll need to gather more information to see whether it's worth reporting or if it's just behavior that's happening as a consequence of using packages. Or if it's something about packages that I'm doing wrong, since this is pretty new to me. So I'll update with more if I find anything else that seems noteworthy. Thanks again for putting your branch out there!

[AirQuick]

I don't have an official answer from Saxonica, but it seems that if you use vendor-options (=$x:saxon-config), the invoked stylesheet can't call a function passed as a parameter, unfortunately.
So, if you want to call the parameterized function together with Saxon configuration specified, you probably have to use SAXON_CUSTOM_OPTIONS.

if I run XSpec from the terminal and I set SAXON_CUSTOM_OPTIONS with -config:MyConfig.xml and then remove the x:variable in the XSpec, it seems to run fine, but I'm not sure how I'd do that from inside Oxygen.

SAXON_CUSTOM_OPTIONS (command line) equivalence in Oxygen Run XSpec Test transformation scenario is saxon.custom.options as in the end of parameter list of this picture and explained in Wiki.

[jcronk]

Just to update on this, @AirQuick, I've managed to get XSpec to run for a good portion of the codebase I'm converting to packages, and so far everything looks great, at least for my purposes. I ran into an issue with visibility where if I used a function in the XSpec file that was defined in the SUT, it would say it couldn't be found. But maybe I'm not supposed to do that, and in any case, I haven't run across it in a few days (but I'm still rebuilding the stylesheets and tests, since in order to get a fairly sparse example, I had to begin with a lot of what I'd previously had commented out).

I'm still working on this, but I want to reiterate my thanks for your answers to my dumb questions. :) There are a number of pieces I have yet to put together, so I'll update if I see any corner cases that look like they may be coming from the XSpec code changes. The worst issue I've seen so far turned out to be an uncaught exception in Saxon itself, where if you use xsl:expose and you put an undeclared namespace prefix in the names attribute, it throws a null pointer exception and gives you no idea where it might be coming from. Just mentioning this in case anyone else tests this branch and is as inattentive as I sometimes can be.

[AirQuick]

if I used a function in the XSpec file that was defined in the SUT, it would say it couldn't be found. But maybe I'm not supposed to do that

Right, as listed in the PR description:

• In .xspec files (x:expect/@select etc.), you can't access variables/functions defined in the tested stylesheet.

If this is going to be a really annoying limitation, we might need to consider introducing something like x:include-helper which includes a helper stylesheet (or query) into the compiled stylesheet (or query).

an uncaught exception in Saxon itself, where if you use xsl:expose and you put an undeclared namespace prefix in the names attribute, it throws a null pointer exception and gives you no idea where it might be coming from.

I remember I encountered it myself while writing a test for this PR. I didn't have time to report it to Saxonica at that time, but yes, it actually was cryptic.

[jcronk]

Right, as listed in the PR description:

• In .xspec files (x:expect/@select etc.), you can't access variables/functions defined in the tested stylesheet.

Ah, okay. I should have revisited the PR before posting about it.

If this is going to be a really annoying limitation, we might need to consider introducing something like x:include-helper which includes a helper stylesheet (or query) into the compiled stylesheet (or query).

I was actually going to ask about this separately, since it wasn't directly in the scope of the original issue: something like x:include-helper, or maybe a use-package that passes through to the compiled stylesheet, would be extremely helpful in some cases. I've been using x:import to bring in a set of variables, some of which are anonymous functions, but being able to include a stylesheet (or use a package) would make things much nicer.