Skip to content

feat(sparql-anything): add SparqlAnythingConverter for chunked non-RDF to RDF conversion - #511

Merged
ddeboer merged 6 commits into
mainfrom
feat/sparql-anything-converter
Aug 31, 2026
Merged

feat(sparql-anything): add SparqlAnythingConverter for chunked non-RDF to RDF conversion#511
ddeboer merged 6 commits into
mainfrom
feat/sparql-anything-converter

Conversation

@ddeboer

@ddeboer ddeboer commented Jun 23, 2026

Copy link
Copy Markdown
Member

Adds @lde/sparql-anything, the first piece of porting geonames-rdf (shell scripts driving the SPARQL Anything CLI) to LDE.

SparqlAnythingConverter runs the SPARQL Anything jar once per input chunk to bound memory use, then streams the per-chunk N-Triples outputs into a single file. Processes are spawned through an @lde/task-runner, so the same converter works on the host, in Docker, or anywhere else a TaskRunner is implemented.

Domain-agnostic API

The originally reviewed version took a required adminCodesFile – a GeoNames concept in the constructor of a package that exists to convert anything, and one that could not express the invocations passing no --load at all. It is now an optional, generic load. Folded in before the first publish, where the rename is free. The general rule this prompted is #783.

It stays a single string rather than the string[] the review suggested: SPARQL Anything's -l takes one path, and a file (default graph) is not interchangeable with a directory (one named graph per file), so an array would have been both unsupported and semantically different.

Safeguards ported from map.sh

  • A non-zero exit aborts the whole conversion, so a crashed chunk cannot be silently dropped from the output.
  • An empty or missing chunk output aborts it too. SPARQL Anything exits successfully when it cannot read or parse an input: it logs the problem, writes nothing and stops. map.sh guards this explicitly ([ -s … ]) after a run once shipped a geonames.nt missing every ontology triple while staying green.
  • An empty chunk list is an error rather than an empty output: a chunking step that produced nothing has already failed.
  • convert() resolves only once the concatenated output is flushed and closed. pipeline(…, { end: false }) resolves when the source ends, so an un-awaited output.end() would let convert() return with writes still pending.

Paths, and where per-run files live

A review of the above found that the file handling, not the conversion, was where the remaining defects were:

  • Every interpolated path is quoted (shellQuote). Both runners execute through a shell – NativeTaskRunner spawns with shell: true, DockerTaskRunner runs sh -c – so a chunk name containing a space broke the command, and one containing ; or a backtick executed whatever followed.
  • A required workDir – the runner's cwd, or its Docker mountDir. The generated query file used to be written to the host temp directory, which a container cannot see, so Docker never actually worked despite the documented claim. Query files and chunk outputs now go into a fresh subdirectory of workDir and are named relative to it, so one command works in both places.
  • That subdirectory is removed when the run ends. Chunk outputs used to land beside the inputs and stay there, which also meant a later run whose process wrote nothing at all could pass the non-empty check on the previous run's triples.
  • Inspecting a chunk output rethrows anything but ENOENT, so a permission problem is not reported as an empty conversion.
  • A newline is written between concatenated chunks, so two triples cannot share a line if a chunk output does not end in one.

workDir is required rather than defaulted: only the caller knows what their runner can see, and a silent host-temp default is exactly what made the Docker failure so confusing.

shellQuote moves into @lde/task-runner

sparql-qlever already carried a private shellQuote, added after an apostrophe in 's-Hertogenbosch silently produced 0-triple imports. Rather than copy it, it moves to the layer both consumers share, with tests that round-trip awkward filenames through a real shell; sparql-qlever now imports it. Its coverage thresholds drop by 0.02pp because the helper it no longer holds was fully covered.

Also in this update

  • Reference documentation moved to docs/reference/sparql-anything.md and registered in the sidebar and packages overview; the README is back to the npm-page shape. The docs site became the source of truth on main after this branch was opened.
  • Manifest starts at 0.0.0, per the versioning guidance now on main – which supersedes this branch's own docs: start from 0.1.0 for new packages commit, dropped in the rebase.
  • @lde/task-runner is depended on as ^0.2.13, matching the workspace version as @nx/dependency-checks requires; nx release rewrites it when task-runner is bumped.

Remaining review findings – concurrency, JVM heap and a CLI passthrough – stay in #782 and follow after this merge.

Note for the maintainer: @lde/sparql-anything is not on npm yet, so the release run on merge will fail to publish it until it is bootstrapped manually. Bootstrap after that release run, so the manifest already carries the released @lde/task-runner range.

@ddeboer
ddeboer force-pushed the feat/sparql-anything-converter branch from 80488b3 to 9e59373 Compare June 23, 2026 12:30
@ddeboer ddeboer mentioned this pull request Aug 31, 2026
17 tasks
…F to RDF conversion

- Run the SPARQL Anything CLI once per input chunk (via an @lde/task-runner)
  to bound memory use, then stream-concatenate the per-chunk N-Triples into
  one file.
- Substitute each chunk's path into the query's `{SOURCE}` placeholder through
  a temporary `-q` file, avoiding shell-escaping a large inline SPARQL query.
- Abort the whole conversion when any chunk's process exits non-zero, so a
  crashed chunk can never be silently dropped from the output.
- Scaffold the @lde/sparql-anything package (0.1.0) and list it in the root
  README packages table and architecture diagram.
- Assert each chunk's --output file is non-empty. SPARQL Anything exits 0
  when it cannot read or parse an input: it logs, writes nothing and stops,
  so a run stayed green while its output silently missed every triple of
  that chunk.
- Await the concatenated output being flushed and closed. pipeline() with
  'end: false' resolves when the source ends, so convert() could resolve
  while output.end() was still pending.
The docs site landed on main after this branch was opened, so the package
carried its documentation in its README.

- Add docs/reference/sparql-anything.md and register it in the reference
  sidebar and the packages overview.
- Slim the README to the npm-page shape: description, installation, one
  example and a link to the docs.
- Start the manifest at 0.0.0, per the versioning guidance now on main.
- Depend on @lde/task-runner with the ^0.2.13 range the sibling packages
  use, so the workspace copy is linked rather than a nested one installed.
@ddeboer
ddeboer force-pushed the feat/sparql-anything-converter branch from 9e59373 to 7fef7e6 Compare August 31, 2026 07:50
adminCodesFile was required, singular and named after a GeoNames concept,
which a domain-agnostic package must not carry. It also could not express
the invocations that pass no --load at all.

- Rename it to an optional `load`, and pass no --load when it is unset.
- Document that SPARQL Anything reads a file into the default graph and a
  directory as one named graph per file, so the two are not interchangeable.
- Drop the remaining domain-specific names from the tests and examples.

The CLI takes a single -l path, not a repeatable one, so this stays a
string rather than the string[] the review suggested. Not marked breaking:
the package has never been published, so the whole API arrives at once.
Task runners execute their command through a shell – NativeTaskRunner
spawns with shell: true, DockerTaskRunner runs sh -c – so an unquoted path
containing a space splits into several arguments, and one containing a
semicolon or a backtick executes whatever follows.

sparql-qlever had already learnt this the hard way and carried a private
shellQuote; move it to the layer both consumers share, with tests that
round-trip awkward names through a real shell.
…ee them

A code review of the merged converter found six defects. All of them are
about files and paths rather than the conversion itself:

- Quote every interpolated path with shellQuote. Both runners execute
  through a shell, so a chunk name with a space broke the command and one
  with a semicolon or a backtick executed whatever followed.
- Take a required workDir – the runner's cwd, or its Docker mountDir – and
  write the generated query and the per-chunk outputs into a fresh
  subdirectory of it, referred to by a relative path. The query file used
  to be written to the host temp directory, invisible to a container, so
  Docker never worked despite the documented claim.
- Remove that subdirectory when the conversion ends. Chunk outputs used to
  land beside the inputs and stay there, so a later run whose process wrote
  nothing at all passed the non-empty check on stale triples.
- Reject an empty chunk list instead of writing an empty output, since a
  chunking step that produced nothing has already failed.
- Rethrow anything other than ENOENT when inspecting a chunk output, so a
  permission problem is not reported as an empty conversion.
- Write a newline between concatenated chunks, so two triples cannot share
  a line if a chunk output does not end in one.
@ddeboer
ddeboer merged commit b14e043 into main Aug 31, 2026
4 checks passed
@ddeboer
ddeboer deleted the feat/sparql-anything-converter branch August 31, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant