Repository navigation
Full text by XQuery and XPath Full Text 3.0, with no Lucene dependency; directory collections; two
security updates for hosts that use a ResourcePolicy or a large thread stack. Requires
PhoenixmlDb.Core 2.6.1. W3C QT3: 31,294 of 31,325 (31 failing, the same cases as 3.1.0). W3C Full
Text (XQFTTS 1.0.4): 672 of the 672 applicable cases (3.1.0: 271 of 685).
Security: a file that changes between the policy check and the read (GHSA-v27x-ffg3-3wqq)
Only hosts that use a ResourcePolicy, and whose allowed roots untrusted parties can write to,
are affected. The library opens a file first, and the policy then judges the path of the file
that is open, as the operating system reports it. A symbolic link put in place of the file, or of
a directory on its path, after the check is not followed. This applies to fn:unparsed-text,
fn:json-doc, fn:doc, fn:doc-available, fn:collection, module imports, schema and DTD loads,
xi:include and output:parameter-document. Content that the host's resource resolver supplies,
and locations that the host maps, do not change. See
GHSA-v27x-ffg3-3wqq.
Security: an error deep in recursion on a large thread stack (GHSA-rq84-6mj7-vqww)
Only hosts that run queries on a thread with a large stack and set no step or depth limit are
affected. A query that fails deep in recursion returns its error in time linear in the depth. The
error code and message do not change. A query error or a cancellation does not carry a stack trace
that spans every level. See
GHSA-rq84-6mj7-vqww.
Breaking — read this when you upgrade
-
contains textuses the defaults of Full Text 3.0 (Appendix C, #384):- No stemming:
"running" contains text "run"is false. Writeusing stemming, or
declare ft-option using stemming;for a module. - No stop words, and stop words apply to the query only (§3.4.7): a stop word in the query
matches any one token."the walrus and the carpenter" contains text "the"is true. - Diacritics insensitive:
"café" contains text "cafe"is true. window N wordsmatches only words that are inside N words of each other.- Tokens do not cross element tags, except where a letter or digit touches the tag on both
sides (<b>p</b>rototypes). A paragraph ends at an element tag. A sentence ends at.,!,
?and at the end of a paragraph.
- No stemming:
-
The public full-text syntax tree types change shape (#384). Code that walks the syntax tree
must change:FtMildNotNode.IncludeandExcludebecomeOperands.FtSelectionWithFiltersis removed. The new node types areFtWithOptionsNode,
FtPositionFilterNodeandFtExtensionSelectionNode.FtMatchOptionshas option groups.FtContainsExpression.MatchOptionsbecomes options on each primary plusDefaultOptions.
FtContainsExpression.Ignoreholds thewithout contentexpression.FtContainsOperator.WordOperatorsbecomesEmbeddedOperators, and the operator hasIgnore.- A walker must visit the embedded expressions: weight, times range, window size, distance range
and ignore.FtSelectionNode.AllExpressions()lists them all.
FullTextEngine.Analyze,ContainsText,WithinWindowandFullTextAnalysisOptionskeep
their signatures. -
fn:serializewith no parameters, or with a map that names no method, uses the xml method
(F&O 3.1, #387), as Saxon does.serialize((true(), "a"))givestrue a(3.1.0:
true() "a").serialize(map{...})with no method gives SENR0001. -
A relative URI with no base URI gives an error (#374). A query compiled from a string with
no base URI does not read from the process working directory:doc()gives FODC0002,
json-doc()gives FOUT1170 (asunparsed-text()does in 3.1.0), anddoc-available()returns
false. This applies with and without aResourcePolicy. A relativedeclare base-uriwith no
base stays as written. A document that the host loaded into the store under a relative name,
host-supplied content and mapped locations do not change. To read a local file, set
CompilationOptions.BaseUrior give an absolute URI. ThexqueryCLI sets the base URI. -
QueryEngine.Compilehas overloads that take aCancellationToken(#371). The token is
checked during parsing, each analysis and rewrite pass (imported modules too) and planning. A
cancel givesOperationCanceledException.ExecuteAsync(XQueryExpression, ...)and
XQueryFacadepass their token to the compile. The overloads without a token stay. A caller
that has a token in scope and does not pass it gets analyzer warning CA2016, which stops a build
with warnings as errors. -
FullTextEngine.Analyzehas a second overload (#381). A<see cref="FullTextEngine.Analyze"/>
is ambiguous (CS0419). Name the signature in the reference.
New: full text without Lucene (#381)
PhoenixmlDb.XQuery depends on Antlr4.Runtime.Standard and PhoenixmlDb.Core only. The
built-in analyzer gives the same tokens as the Lucene 4.8 chain of 3.1.0: the term, position and
offsets of each token are equal on 15,051 files and on 2,000,000 random strings, under 7 option
sets. A full-text index made with 3.1.0 stays valid. A host that uses Lucene itself references
Lucene.Net directly and chooses its version.
IFullTextAnalyzer,FtToken,FtAnalysisRequest,FullTextAnalyzerRegistryand
StandardFullTextAnalyzer(namespacePhoenixmlDb.XQuery.FullText) let a host register an
analyzer for a language.QueryEngine.FullTextAnalyzersgives the registry tocontains text,
phx:tokenizeandphx:stem.FullTextEngine.Analyze(text, options, registry)lets an index
use the same registry.- The word boundaries follow UAX #29 over Unicode 6.3 data. The stemmer is Porter (1980).
New: XQuery and XPath Full Text 3.0 (#384, #388)
All of the Full Text 3.0 grammar:
- match options on each primary:
("dog" using stemming) ftand "cat"; weight {expr};occurs exactly|at least|at most|from … to … times;- the positional filters
ordered,window,distance,same|different sentence|paragraph,
at start,at endandentire content, in any combination; without content <expr>;using lowercaseandusing uppercase;using stop words default,at "uri"and("a", …), withunionandexcept;using thesaurus at "uri" relationship "…" N levels,thesaurus defaultand
thesaurus ( … );using option prefix:name "value";- extension selections
(# pragma #) { … }; declare ft-option using …;;for $x score $s in …andlet score $s := ….
The errors that the specification names: FTST0008, FTST0009, FTST0013, FTST0018, FTST0019,
FTDY0016, FTDY0017, FTDY0020, XQST0089, plus XPST0003, XPTY0004 and XPST0081 where they apply.
Stop word lists and thesauri by URI (#388). IFullTextResourceResolver
(QueryEngine.FullTextResources) supplies the resource for using stop words at "uri" and
using thesaurus at "uri". FullTextResources is an in-memory resolver. FullTextThesaurus loads
the W3C test format and its own format. With no resolver, a stop word URI gives FTST0008 and a
thesaurus URI gives FTST0018. using stop words default uses the resolver's default list, else the
built-in English list.
New: index support for full text
FullTextAnalysisOptions.FoldDiacritics(#385) removes diacritics from the tokens of an
index, so that the index finds the candidates of a diacritics-insensitive query. The default
tokens do not change.- A candidate-source hook for
contains text(#389):IFullTextCandidateSource,
IFullTextCandidateFilter,FtCandidateQueryandQueryEngine.FullTextCandidates. With a
source,contains textskips only the items that the source rules out, and the results do not
change. With no source, the evaluation does not change. An index that answers the hook must
give at least the candidates that the evaluator matches.
New: directory collections (#390)
collection() and uri-collection() read a local directory:
file:///data/?select=*.json;recurse=yes. The parameters are those of Saxon: select (a glob),
match (a regular expression), recurse=yes|no, on-error=fail|warning|ignore and
content-type, separated by ; or &.
collection()gives an.xmlfile as a document, a.jsonfile as a map or array (as
json-doc()), a text file as a string and any other file asxs:base64Binary.
uri-collection()gives the URIs and reads no file. The order is the path order.ServerDefaultand aResourcePolicythat denies files refuse the directory, and the
directory is not listed. A resolver withSuppliesAllContentgives no listing. A listing does not leave the
directory through..or a symbolic link, and a linked directory is not entered. Each file is
authorised and read with the limits ofdoc(),json-doc()andunparsed-text().on-error
does not hide a refusal or a limit.- A relative directory URI with no base URI gives FODC0002.
- Only the built-in
XdmDocumentStorelists a directory. A host with its own
IDocumentResolverkeeps its own collection behaviour. - Differences from Saxon-HE 12.10:
on-erroralso applies to a JSON parse error, and the order is
the path order. doc()keeps the cause of a retrieval error in its message.
Serialization
- Indented XML output has the layout of Saxon (#392, #393). With
indent=yesand the xml
method, an empty element is<e/>(3.1.0 in thexqueryCLI:<e />), and an element
serialized with its XML declaration starts on the line after the declaration. Text directly
after the declaration gets no line break. The xhtml and html methods keep<br />. A test that
compares indented output byte for byte must update its expected files. serialize()keeps" />"in text, comments, processing instructions and CDATA sections
(#393). Only the end of an empty element changes. ThroughXQueryFacade, the xhtml method gives
<br />.- The XML declaration names the encoding that the caller gives, or
UTF-8(#393), on every
path. 3.1.0 writesutf-8for a document node and in the CLI. - A space between adjacent atomic values with the xml, html and xhtml methods (#383).
Also in this release
fn:analyze-stringreturns its result element with no parent (#386), so a path such as
analyze-string($s, $re)/*works.generate-id()comes from the identity of the node (#376). Two different nodes do not get the
same id. The form of an id changes.- UCA collations apply
alternate=shiftedand the numeric strengths1to5(UTS #10, #379). fn:index-of#2uses the default collation (#375).
xquery CLI (ships on cli-v3.2.0)
- Runs on PhoenixmlDb.XQuery 3.2.0 and PhoenixmlDb.Xslt 3.2.0, so it includes the changes
above andfn:transformfrom a query gets the 3.2.0 XSLT engine. Includes the fixes for
GHSA-v27x-ffg3-3wqq and GHSA-rq84-6mj7-vqww. - JSON sources (#391): a
.jsonsource is its map or array, and it is the context item when
it is the first source. A directory source also loads its*.jsonfiles:collection()gives
the documents, then the JSON items. Input on stdin that starts with{or[is read as JSON. - A source that cannot be loaded is an error (#391):
Error: Cannot load source '<s>': <reason>on one line, exit code 1. A stack trace only with--verbose. Breaking: 3.1.0
writes a warning for a missing source and then gives XPDY0002 with exit code 3. A script that
tests for exit code 3 must change. - An empty element in indented output is
<e/>(#392).