Skip to content

[feature] Add EXPath File Module 4.0 as built-in extension module#6257

Open
joewiz wants to merge 3 commits into
eXist-db:developfrom
joewiz:feature/expath-file-extension
Open

[feature] Add EXPath File Module 4.0 as built-in extension module#6257
joewiz wants to merge 3 commits into
eXist-db:developfrom
joewiz:feature/expath-file-extension

Conversation

@joewiz
Copy link
Copy Markdown
Member

@joewiz joewiz commented Apr 24, 2026

Summary

  • Ports the EXPath File Module 4.0 (http://expath.org/ns/file) from a standalone XAR package (eXist-db/exist-file) to a built-in extension module
  • Makes the module available out of the box — no XAR installation required
  • Coexists with eXist's native file module (http://exist-db.org/xquery/file) since they use different namespace URIs

What Changed

New module: extensions/modules/expath-file/

  • 10 Java classes ported from eXist-db/exist-file (48 functions)
  • Functions: file properties, I/O (read/write/append), directory manipulation, path operations, system properties
  • All functions require DBA role (security boundary)
  • Artifact ID: exist-expath-file

Registration:

  • extensions/modules/pom.xml — added expath-file module
  • exist-distribution/pom.xml — added exist-expath-file runtime dependency
  • exist-distribution/src/main/config/conf.xml — registered http://expath.org/ns/file

Coexistence with Legacy Module

Both modules can be used simultaneously — they have different namespace URIs:

(: EXPath standard — portable across eXist, BaseX, Saxon :)
import module namespace file = "http://expath.org/ns/file";
file:read-text("/path/to/file.txt")

(: eXist-native — unchanged, fully backward compatible :)
import module namespace file = "http://exist-db.org/xquery/file";
file:read("/path/to/file.txt")

XQTS Impact

The QT4 test suite contains 190 EXPath File tests (expath-file test set) and 574 EXPath Binary tests (27 bin-* test sets) — 764 tests total across both modules. These tests currently all fail because the modules aren't available to the XQTS runner. Making them built-in extension modules resolves this. The standalone XAR achieved 96.3% compliance (183/190).

Spec References

Test Plan

  • exist-core XQSuite tests pass (no regressions)
  • Module compiles and dependency analysis passes
  • XQTS expath-file test set (after merge to integration branch)

🤖 Generated with Claude Code

Port the EXPath File Module (http://expath.org/ns/file) from a standalone
XAR package (eXist-db/exist-file) to a built-in extension module. This
makes the module available out of the box without requiring XAR installation.

The EXPath File Module implements the EXPath File Module 4.0 specification
and coexists with eXist's native file module (http://exist-db.org/xquery/file)
since they use different namespace URIs.

Source: https://github.com/eXist-db/exist-file

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The module was only registered in the distribution conf.xml but not in
any test conf.xml files. The XQTS runner uses the test conf.xml, so
all EXPath File tests fail with "unknown function" errors without this.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@duncdrum
Copy link
Copy Markdown
Contributor

@joewiz I m a bit lost with this and the companion PR. I know the xq4 includes expath binary functions, should these PRs wait until XQ4? Why the split?

@joewiz
Copy link
Copy Markdown
Member Author

joewiz commented May 12, 2026

@duncdrum The reason I returned to the idea of adding this (EXPath File) and EXPath Bin to eXist (as opposed to as separate packages) is that the QT4 tests of the file: and bin: functions expect them to be available, and I couldn't figure out how to get exist-xqts-runner to run eXist with packages. The EXPath File 4.0 spec does make use of XQuery 4.0 features like record types, and I believe the implementation in this PR expects eXist to support those—so if I confirm that, I could adjust the PR to make it 3.1-compatible (plain maps instead of records). Then, once we support 4.0 records in eXist, we could adjust this. It would be nice to get File and Bin added for eXist 7, but if push comes to shove, they could wait for a later release. Thoughts?

@joewiz
Copy link
Copy Markdown
Member Author

joewiz commented May 12, 2026

Also relevant: qt4cg/qtspecs#2603 makes EXPath Bin and File optional for implementations, though since we have them and they are well tested, with up-to-date tests, in XQTS, having these in eXist (as opposed to in packages) seems like a win-win for testing and access.

This PR sets the namespace prefix for EXPath File to exfile to avoid conflicting with eXist's longstanding file module. This status allows coexistence of the two modules and works fine with XQTS in my testing.

@duncdrum
Copy link
Copy Markdown
Contributor

Yes I would be very in favor of pulling in an xquery 3.1 implementation for 7.0.0

final StringBuilder regex = new StringBuilder();
for (int i = 0; i < glob.length(); i++) {
final char c = glob.charAt(i);
switch (c) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert to switch expression

@line-o line-o added the XQ4 xquery 4 label May 12, 2026
Addresses reinhapa's review on PR eXist-db#6257 (FileManipulation.java:511).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@joewiz
Copy link
Copy Markdown
Member Author

joewiz commented May 12, 2026

[This response was co-authored with Claude Code. -Joe]

Done in 7c6a9ed96eglobToRegex switch converted to Java 21 arrow syntax, comma-separated case labels for the escaped-character set. Compiles clean.

Separately on @duncdrum's question about whether this PR should wait until XQ 4.0: per your follow-up "Yes I would be very in favor of pulling in an xquery 3.1 implementation for 7.0.0" — that's the direction we'll take. Preparing a follow-up commit that converts the PR from XQ 4.0 record types to XQ 3.1 map types so it lands cleanly under 3.1 for the eXist 7 release. That'll be its own commit on this branch.

@joewiz
Copy link
Copy Markdown
Member Author

joewiz commented May 13, 2026

[This response was co-authored with Claude Code. -Joe]

Following up on Duncan's "should this wait until XQ 4.0?" question with concrete data: I ran the EXPath File 4.0 QT4 test suite against this PR's branch with the develop runtime (Saxon 9.9.1-8).

Result: 174 of 190 tests pass (91.6%).

Decomposing the 16 failures:

  • 8 require xquery version "4.0" in the test prolog and raise XQST0031 under develop's XQ 3.1-only parser. These will pass automatically once XQ 4.0 lands on develop — they're not implementation gaps, just version mismatches in the test fixtures.
  • 6 are real module-side impl bugs (3× file:read-text-lines blank-line semantics, 2× recursive file:copy edge cases, 1× file:append mode). Filing follow-up issues; don't block landing this PR.
  • 2 are test-infrastructure issues (sandpit relative-path / shared-state). Also follow-up.

Static analysis also confirmed: the PR uses zero XQ 4.0-specific syntax — no record types, no XQ 4.0 numeric literals, no fn{...} lambdas, no keyword arguments, no otherwise operator. Types are declared with XQ 3.1-vintage Type.MAP, Type.ITEM, etc. The illustrative record(...) notation in the EXPath File 4.0 draft spec is documentation for readers; the formal function-signatures declarations at the bottom of the spec use map(*), which is what the PR implements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

XQ4 xquery 4

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants