Skip to content

PhoenixmlDb.XQuery 3.0.0

Choose a tag to compare

@elvogel elvogel released this 04 Nov 04:20
· 42 commits to main since this release
2bb6451

XPath mode, schema-aware function results and types, bounded schema fetches, tail calls, and a
large drop in QT3 failures. QT3: 31 failing of 31,325 (2.9.0: 207 of 31,331). The count includes
corrections to the test harness as well as engine changes, and 5 Unicode cases that depend on
another Unicode version are skipped. Requires PhoenixmlDb.Core 2.6.0.

Security: deeply nested or chained expressions with untrusted queries (GHSA-9qvr-vjmg-m3pf)

Only hosts that compile or evaluate queries from untrusted parties are affected. A deeply nested
or long chained expression compiles and evaluates within the thread's stack and in linear time, or
raises XPDY0130. In browser WebAssembly and WASI, which cannot start a thread, a deep evaluation
continues on the current stack as in 2.9.0, and the workarounds of the advisory still apply. See
GHSA-9qvr-vjmg-m3pf
for the details and workarounds.

Breaking — read this before upgrading

No public type or signature is removed or changed. These behaviour changes can affect a host:

  • E => f(A) parses to the function call f(E, A): a FunctionCallExpression, or a
    DynamicFunctionCallExpression when f is a variable or an expression. The parser makes an
    ArrowExpression only for the thin arrow ->, and its IsThinArrow is always true. Code that
    walks the AST and looks for ArrowExpression to find => must look for the call.
  • XPDY0130 for an expression nested too deeply for the thread's stack. A caller can catch it.
    A deep chain of arithmetic, comparison, and, or or || operators does not raise it: its
    evaluation continues on a new stack. Chains of predicates, ! maps, unary operators, lookups
    and FLWOR clauses can raise it on a small stack. The parser needs a stack of at least 768 KB.
  • CostModel.EstimateCardinality keeps each result per CostModel instance. Do not use one
    instance from more than one thread at the same time. A change in IContainerStatistics after
    the first estimate does not change the kept results. The library makes a new CostModel for
    each query plan.
  • XQueryExpressionRewriter.Rewrite and XQueryExpressionWalker.Walk can continue on another
    thread when the stack is low.
    A subclass that keeps per-thread state ([ThreadStatic]) in its
    Visit methods can see a different thread during a deep walk.
  • A nested load-xquery-module (depth more than 1) runs on a new 16 MB stack, so the limit of
    64 nested loads gives FOQM0003 on any thread stack size.
  • FODC0005 for a colon in the first segment of a relative fn:doc reference (#340), for
    example doc('2026-01-02T10:30.xml'). Such a name is not a valid relative URI. A stored
    document with such a name cannot be read with doc('name'); start the reference with ./
    (doc('./2026-01-02T10:30.xml')).

API

  • XPath mode: CompilationOptions.XPath and XQueryParserFacade.XPath (#328). The text is
    parsed as an XPath 3.1 expression. XQuery-only syntax (prolog, FLWOR clauses beyond one for or
    let, positional and typed bindings, switch, typeswitch, try/catch, update expressions,
    validate, extension and ordered expressions, constructors, string constructors, annotations)
    is XPST0003. < is always an operator, string literals are verbatim, line endings are not
    normalized, and the namespace axis is available. The default is off, and XQuery behaviour does
    not change.
  • XdmSequenceType.OptionalNumeric (#330).

Schema fetches and locations

  • A schema fetch is bounded, and a timeout is XQST0059 (#348). With no ResourcePolicy, a
    schema location is read through the engine's HTTP client, which gives up on a connect after
    10 s. A timeout is reported as XQST0059 from Compile, and the schema provider is released.
  • A schema location is resolved as a URI (#265): a location with a scheme that is not a URI is
    not read as a path, and a schema hint is not made absolute against the working directory.

Deep trees

  • A query that builds a deeply nested tree does not end the process (#301). A constructed
    element copies its descendants with an explicit stack. fn:serialize, the result serializer
    (xml, adaptive, json, html, xhtml), fn:deep-equal, fn:base-uri and copy-modify continue on a
    16 MB thread when the stack is short. validate {} over such a tree raises a catchable
    InsufficientExecutionStackException.

Types and function conversion

  • fn:analyze-string results carry the types of their schema (#345) when the schema provider
    can take the built-in F&O schema; otherwise they stay untyped. A carriage return in the text
    survives.
  • element(*, xs:numeric) and attribute(*, xs:numeric) match nodes of a numeric type (#346).
  • element(*, U) is a subtype of element(*, V) in a function type only if U derives from V
    (#342), when a schema provider is present.
  • A function result of a schema type casts an untyped value through the schema (#343), so a
    union tries its members in order.
  • fn:abs, ceiling, floor, round and round-half-to-even return xs:numeric? (#330).
    The two-argument forms take xs:anyAtomicType?, and the result keeps the argument's type.
  • Node arguments of atomic-typed parameters are atomized (#306). escape-html-uri,
    encode-for-uri and iri-to-uri accept nodes; escape-html-uri and iri-to-uri encode by code
    point. fn:id splits each string value into IDREFs.
  • An xs:string parameter takes a string, an untyped value or an xs:anyURI (#317, #332). A
    number, boolean, date, time, duration, g* value or QName is XPTY0004.
  • An untyped value that does not cast raises the cast's error (#321): local:f(<a/>) for an
    xs:boolean parameter is FORG0001.
  • List-typed nodes (#305): a list of xs:QName keeps each item's namespace, and a general
    comparison compares each item.
  • A string is whitespace-collapsed before a cast to a non-string type (#307).
  • An integer wider than 64 bits casts to xs:double and xs:float (#319).
  • A double zero keeps its sign, and an xs:decimal zero has none (#323). The optimizer folds
    0 * x and x + 0 only where the result is exact.
  • Value equality compares xs:QName by namespace URI (#338): index-of, distinct-values and
    the other value-equality functions.
  • fn:deep-equal treats a one-item sequence member as its item (#333).
  • A validated tree's string values leave out the dropped whitespace (#331).

Error codes

  • XPST0008 for a prolog variable in its own initializer; a cycle through a function stays XQDY0054
    (#325).
  • XPST0081 for an undeclared prefix in a literal computed-constructor name (#312) and in a declared
    variable's name (#337).
  • XQST0040 for two direct attributes with one expanded name (#322).
  • XQST0022 only for a real enclosed expression in a namespace declaration; {{ and }} are
    literal braces (#311).
  • XQST0034 for a function named like an imported schema type's constructor (#335).
  • XQST0052 for a cast to a type outside the in-scope schema types, reported at compile time (#341).
  • XQTY0105 for a function item in element or document content (#326).
  • XPST0003 for validate or an extension expression at the start of a postfix, path or simple map
    expression without parentheses (#344). Queries that write validate { … }/@a need parentheses.
  • FOAR0001 for INF idiv 0 (#313); FOAR0002 for a decimal sum out of range in fn:avg (#324);
    FOAY0002 for a negative array:subarray length, with the end check in long (#316).
  • FODC0005 for a colon in the first segment of a relative fn:doc reference (#340).
  • FOJS0005 for json-to-xml with validate and duplicates="retain"; duplicates defaults to
    reject with validate (#334).
  • FOUT1190 for undecodable fn:unparsed-text content with no encoding named, and XPTY0004 for a
    non-string encoding (#308).
  • XPTY0004 for an atomic lookup node in fn:id and fn:element-with-id (#309).

Other behaviour

  • fn:normalize-unicode supports FULLY-NORMALIZED (#339).
  • A document node without exactly one root element serializes as a fragment (#318):
    document{<a/>, <b/>}.
  • format-number in a library function uses that module's decimal format first (#336).
  • Each closing brace matches the brace it closes (#320), for an enclosed expression nested in
    document { } in element content.
  • MessagePack 2.5.307 in the language server, for GHSA-qhrr-8q5h-9q3h (#272).

Also in this release

  • A string walked by a recursive function is linear (#296, with tail calls): an argument
    substring($s, 2) to a declared function whose parameter is untyped or xs:string is passed
    as a slice of the string, not a copy of its rest. A 50,000-character walk: 909 ms and 2.5 GB
    -> 73 ms and 133 MB. Tail calls are also open to a host's own functions (the XSLT engine's
    stylesheet functions); a call never crosses from one kind to the other.

  • Tail calls of declared functions (#296). A call to a declared function in tail position of
    a declared function's body (the body; the branches of if, typeswitch and switch; the
    return of a FLWOR of let clauses only; an arrow call there) runs as the caller's next
    iteration instead of nested, when the callee has the caller's declared return type. Tail
    recursion, direct or mutual, is no longer limited to a depth of 1,000 (FOER0000): a million
    iterations run in about 1.4 s. Recursion whose result is used after the call keeps the limit.
    An infinite tail recursion now runs until it is cancelled, as a loop does. QT3
    numberformat121, numberformat122.

  • The descendant axes allocate one stack per walk, not one iterator per node (xquery#294).
    iso_svrl_for_xslt2 over NY.EMS.v350: 2,355 -> 1,696 MB, time 0.85x; output unchanged.

  • A query context can be reset and used again (internal, for hosts): ResetForReuse
    returns it to the state of a new one, and a context an inline function item keeps is marked
    and never reset. The XSLT engine uses it to evaluate its XPath expressions without a new
    context each (xquery#294). No public API changes.

  • if, arithmetic and value comparisons are evaluated synchronously where their operands
    allow it, so an expression containing one no longer leaves the synchronous path (each
    evaluation allocated an async iterator per operator). W3C evaluate-010, which evaluates
    if (position() mod 7 = 0) then $a else $b 100,000 times: 518 ms -> 350 ms warm, 1,403 MB
    -> 685 MB. Results unchanged.

  • Character map keys are one character, and are checked as the source requires. From a map, a
    key that is not a string (an xs:QName) is XPTY0004 and a longer string SEPM0016; from an
    output:serialization-parameters element, a character that is not one character makes the
    element invalid, SEPM0017. A supplementary character (a surrogate pair) is one key, and is
    mapped. QT3 serialize-xml-023, -023b, -139, -139b.

  • A constructor function item carries its signature (XPath 3.1 §3.14.2): xs:T#1 is
    function(xs:anyAtomicType?) as xs:T?, and for a list type returns its item type's sequence
    (xs:NMTOKENS#1 returns xs:NMTOKEN*; a schema list type of integers, xs:integer*). It was
    function(item()?) as item(), so let $f as function(xs:anyAtomicType) as xs:NMTOKEN* := xs:NMTOKENS#1 was refused. A list type is cast to only from one xs:string or
    xs:untypedAtomic: xs:ENTITIES(xs:anyURI("abcd")) and a sequence of two strings are
    XPTY0004. QT3 CastAs-ListType-17..19, -26, -27.

  • An unprefixed type name in element() and attribute() tests is in the default namespace for
    elements and types
    (XQuery 3.1 §3.1.1), which import schema default element namespace and
    declare default element namespace set. It was read as an xs type: element(*, restrictedType)
    raised XPST0008, or as a step matched nothing. A built-in type name keeps reading as one. QT3
    ForExprType026..041, step-expr-si-1..3.

  • fn:substring of a string held as a slice costs no copy (#287). An internal string slice
    (StringSlice) shares the characters of the string it was taken from; fn:substring,
    fn:string-length and fn:starts-with read one without making its string, and every other
    reader of a variable gets the string. Slices are made only by the XSLT engine, for a template
    parameter bound to substring(...) (see the XSLT release notes); a query never sees one. No
    public API changes.

  • count and subsequence of a range are computed from its bounds. count(1 to N),
    count(subsequence(a to b, ...)) and subsequence(a to b, ...) no longer generate the integers
    before the slice, so count(subsequence(1 to 3000000000, 2147483648)) returns 852516353 instead
    of stopping at the materialization limit (FOER0000). QT3 cbcl-subsequence-010..014.

  • A year outside 1..9999 survives fn:dateTime and the component functions. fn:dateTime
    dropped the year it holds apart from the .NET date, so -0012-12-03 combined into
    0004-12-03; and year-from-date and the other twelve component functions parsed an untyped
    argument, or a node, as a .NET date, which refused -0012-12-03. An untyped argument is now cast
    by the XSD type's own parser, and a node atomized first.

  • schema-element(N) follows XSD 1.1 substitution groups: an element in several groups is a
    member of each (with PhoenixmlDb.Core's change), an abstract element is a member of none, and a
    nilled element matches only where the declaration of its name is nillable (XPath 3.1 §2.5.5.3).
    ISchemaProvider gains IsNillableElementDeclaration, with a default that says true. QT3
    substitution-020..025.

  • A kind test as the node test of a step is schema-aware: document-node(element(N)) and
    document-node(schema-element(N)) test the document element (the inner test was not applied, so
    the step selected every document), and element(*, T) / attribute(*, T) match a validated node
    of type T or one derived from it (only the untyped types were known). The prefix of the element
    test inside document-node() is resolved as a step's is. W3C XSLT validation-1401.

  • An untyped element does not match schema-element(N), nor an untyped attribute
    schema-attribute(N)
    , even where N is declared with xs:anyType or xs:anySimpleType, from which
    the untyped types derive (spec bug 10065). They matched such a declaration. W3C XSLT
    validation-1001, with input-type-annotations="strip".

  • fn:deep-equal of two validated elements respects their types' content (F&O 3.1 §14.2.1):
    elements whose types have different content varieties (simple, element-only, mixed, empty) are
    not equal, and two of simple content compare their typed values. Their text and children were
    compared, so xs:integer 2012 equalled xs:gYear 2012. QT3 K2-SeqDeepEqualFunc-36, 38, 39, 42.

  • A cast to a schema list type whose item type is a union with an xs:QName member casts each
    token with the query's prefixes
    : 'a b xs:integer' cast as s:sensitiveList gives two NCNames and
    a QName. The list was kept whole, and a union with such a list as a member refused the value.
    SchemaSimpleType gains ListItemType. QT3 CastAs-UnionType-28..32, CastAs-ListType-21.

  • Q{uri}local names a type of the schema for that namespace, as a prefix bound to it does. It
    was looked up by its local name among the built-in types, so Q{urn:s}u was "not a recognized
    type" and Q{urn:s}string was taken for xs:string.

  • A member type derives from a union only when the union is pure (XPath 3.1 §2.5.2, §2.5.6.1):
    not from a union of list types, nor one with facets. An element of one list type matched
    element(*, U) for U the union of two list types. W3C XSLT match-197.

  • A node passed for a parameter of type xs:anyAtomicType is atomized (XPath 3.1 §3.1.5.2): an
    element validated as xs:integer gives an xs:integer, an untyped one an xs:untypedAtomic. The
    coercion of arguments skipped that type, so the node itself was bound. QT3 FunctionCall-005/007.

  • An xs:untypedAtomic value for the xs:QName parameter of fn:local-name-from-QName,
    fn:namespace-uri-from-QName and fn:prefix-from-QName is XPTY0117
    : it is not cast to a
    namespace-sensitive type. It was XPTY0004. QT3 FunctionCall-018/041, LocalNameFromQNameFunc015a,
    NamespaceURIFromQNameFunc015a.

  • document-node(element(...)) and document-node(schema-element(N)) test the document element
    as an item
    : its namespace, its type, and the substitution group of N. The inner test of
    document-node(schema-element(N)) was not applied, so every document matched it, and
    document-node(element(N, T)) compared the local name of the element only. W3C XSLT match-185,
    with the XSLT change.

  • A prefix that the host binds (CompilationOptions.StaticNamespaces) is known while the query is
    parsed.
    A name the parser resolves itself, as in schema-element(j:array), was looked for in no
    namespace when only the host bound j (XPST0008). A prolog declaration of the prefix still wins.

  • An element validated against a type has that type when the schema declares its name with
    another type.
    Validation by type does not use the element's declaration, but the annotating
    parse assessed the element by it and refused the type, so the element stayed untyped (an XSLT
    <e xsl:type="xs:string"> of a declared e). W3C XSLT match-220 and match-221, with the XSLT change.

  • Arithmetic on an xs:boolean, xs:anyURI, xs:hexBinary or xs:base64Binary is XPTY0004. A boolean
    was taken as 1 or 0 (1 + true() was 2), and the other three escaped as a .NET FormatException
    or InvalidCastException. XPath 1.0 compatibility mode still converts its operands to numbers.

  • The grammar parses the operators from or to cast as with two left-recursive rules (#401),
    not one rule per precedence level. Each operand went through seventeen nested rules, and the
    parse tree had a node for each. For a generated Schematron validator of 2.3 MB, an XSLT compile
    now allocates 316 MB, not 399 MB (-21%), and takes about 315 ms, not 375 ms. Precedence, and the
    XPST0003 for a second comparison, range, cast, castable, treat or instance of on one operand
    without parentheses, are as before.

  • A parse-tree node's child list starts with room for one child (#401). The grammar's precedence
    chain gives most nodes one child, and each list made an array of four. For a generated Schematron
    validator of 2.3 MB, an XSLT compile now allocates 399 MB, not 419 MB.

  • Queries and XPath expressions are parsed in two stages (#401): SLL prediction first, and full LL
    with error reporting only for input that SLL cannot parse. For a generated Schematron validator of
    2.3 MB, an XSLT compile now allocates 795 MB, not 1,095 MB (-27%). With the string-literal lexer
    change it is 435 MB (-60%), and the compile takes about 400 ms, not 590 ms. Syntax errors are
    reported as before.

  • String literals are lexed from the lexer's DFA cache (#401). A predicate in the string-literal
    rule ({AllowRawAmpersand}?) kept every string literal out of the cache, so each character was
    matched by ATN simulation. For a generated Schematron validator of 2.3 MB, an XSLT compile now
    allocates 735 MB, not 1,095 MB (-33%), and takes 460 ms, not 590 ms. A bare & in an XQuery
    string literal is still XPST0003; the error now gives the position of the &.

  • The input document can be validated before a query runs (#267): QueryEngine.InputValidation
    and XQueryFacade.InputValidation (strict or lax), and --validation strict|lax (also
    --val:strict, --val:lax) on the xquery command (package xquery4). The context item, when it is a document or an
    element, is validated as validate { . } would, against the schemas the query imports and the
    other schemas of the schema provider; the query and $input then see the typed node. An invalid
    input is XQDY0027; with strict validation, an element with no top-level declaration is XQDY0084.
    Without the setting the input stays untyped, as before.

  • A type derived from xs:NOTATION makes an xs:NOTATION. Its constructor function made a
    value of the type that was not an instance of xs:NOTATION. A QName or a NOTATION now casts to a
    NOTATION type (XPath 3.1 §19.1); it was "Cannot cast xs:QName to Notation". QT3 notation-cast-3;
    W3C XSLT notation-0002, 0003, 0401-0404.

  • A partial application of a resource function runs as the module that made it, as a named
    reference already did: json-doc(?) made in a library module and called from the main module
    resolved its location against the main module's base URI, and ResourceRequest.ModuleUri
    named the main module.

  • fn:transform turned off by the resource policy raises FOXT0004 ("the XSLT
    transformation has been disabled", F&O 3.1); it raised FOXT0001, which means no suitable
    XSLT processor.