feat(sparql-anything): add SparqlAnythingConverter for chunked non-RDF to RDF conversion - #511
Merged
Merged
Conversation
ddeboer
force-pushed
the
feat/sparql-anything-converter
branch
from
June 23, 2026 12:30
80488b3 to
9e59373
Compare
…F to RDF conversion
- Run the SPARQL Anything CLI once per input chunk (via an @lde/task-runner)
to bound memory use, then stream-concatenate the per-chunk N-Triples into
one file.
- Substitute each chunk's path into the query's `{SOURCE}` placeholder through
a temporary `-q` file, avoiding shell-escaping a large inline SPARQL query.
- Abort the whole conversion when any chunk's process exits non-zero, so a
crashed chunk can never be silently dropped from the output.
- Scaffold the @lde/sparql-anything package (0.1.0) and list it in the root
README packages table and architecture diagram.
- Assert each chunk's --output file is non-empty. SPARQL Anything exits 0 when it cannot read or parse an input: it logs, writes nothing and stops, so a run stayed green while its output silently missed every triple of that chunk. - Await the concatenated output being flushed and closed. pipeline() with 'end: false' resolves when the source ends, so convert() could resolve while output.end() was still pending.
The docs site landed on main after this branch was opened, so the package carried its documentation in its README. - Add docs/reference/sparql-anything.md and register it in the reference sidebar and the packages overview. - Slim the README to the npm-page shape: description, installation, one example and a link to the docs. - Start the manifest at 0.0.0, per the versioning guidance now on main. - Depend on @lde/task-runner with the ^0.2.13 range the sibling packages use, so the workspace copy is linked rather than a nested one installed.
ddeboer
force-pushed
the
feat/sparql-anything-converter
branch
from
August 31, 2026 07:50
9e59373 to
7fef7e6
Compare
adminCodesFile was required, singular and named after a GeoNames concept, which a domain-agnostic package must not carry. It also could not express the invocations that pass no --load at all. - Rename it to an optional `load`, and pass no --load when it is unset. - Document that SPARQL Anything reads a file into the default graph and a directory as one named graph per file, so the two are not interchangeable. - Drop the remaining domain-specific names from the tests and examples. The CLI takes a single -l path, not a repeatable one, so this stays a string rather than the string[] the review suggested. Not marked breaking: the package has never been published, so the whole API arrives at once.
Task runners execute their command through a shell – NativeTaskRunner spawns with shell: true, DockerTaskRunner runs sh -c – so an unquoted path containing a space splits into several arguments, and one containing a semicolon or a backtick executes whatever follows. sparql-qlever had already learnt this the hard way and carried a private shellQuote; move it to the layer both consumers share, with tests that round-trip awkward names through a real shell.
…ee them A code review of the merged converter found six defects. All of them are about files and paths rather than the conversion itself: - Quote every interpolated path with shellQuote. Both runners execute through a shell, so a chunk name with a space broke the command and one with a semicolon or a backtick executed whatever followed. - Take a required workDir – the runner's cwd, or its Docker mountDir – and write the generated query and the per-chunk outputs into a fresh subdirectory of it, referred to by a relative path. The query file used to be written to the host temp directory, invisible to a container, so Docker never worked despite the documented claim. - Remove that subdirectory when the conversion ends. Chunk outputs used to land beside the inputs and stay there, so a later run whose process wrote nothing at all passed the non-empty check on stale triples. - Reject an empty chunk list instead of writing an empty output, since a chunking step that produced nothing has already failed. - Rethrow anything other than ENOENT when inspecting a chunk output, so a permission problem is not reported as an empty conversion. - Write a newline between concatenated chunks, so two triples cannot share a line if a chunk output does not end in one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
@lde/sparql-anything, the first piece of porting geonames-rdf (shell scripts driving the SPARQL Anything CLI) to LDE.SparqlAnythingConverterruns the SPARQL Anything jar once per input chunk to bound memory use, then streams the per-chunk N-Triples outputs into a single file. Processes are spawned through an@lde/task-runner, so the same converter works on the host, in Docker, or anywhere else aTaskRunneris implemented.Domain-agnostic API
The originally reviewed version took a required
adminCodesFile– a GeoNames concept in the constructor of a package that exists to convert anything, and one that could not express the invocations passing no--loadat all. It is now an optional, genericload. Folded in before the first publish, where the rename is free. The general rule this prompted is #783.It stays a single
stringrather than thestring[]the review suggested: SPARQL Anything's-ltakes one path, and a file (default graph) is not interchangeable with a directory (one named graph per file), so an array would have been both unsupported and semantically different.Safeguards ported from
map.shmap.shguards this explicitly ([ -s … ]) after a run once shipped ageonames.ntmissing every ontology triple while staying green.convert()resolves only once the concatenated output is flushed and closed.pipeline(…, { end: false })resolves when the source ends, so an un-awaitedoutput.end()would letconvert()return with writes still pending.Paths, and where per-run files live
A review of the above found that the file handling, not the conversion, was where the remaining defects were:
shellQuote). Both runners execute through a shell –NativeTaskRunnerspawns withshell: true,DockerTaskRunnerrunssh -c– so a chunk name containing a space broke the command, and one containing;or a backtick executed whatever followed.workDir– the runner'scwd, or its DockermountDir. The generated query file used to be written to the host temp directory, which a container cannot see, so Docker never actually worked despite the documented claim. Query files and chunk outputs now go into a fresh subdirectory ofworkDirand are named relative to it, so one command works in both places.ENOENT, so a permission problem is not reported as an empty conversion.workDiris required rather than defaulted: only the caller knows what their runner can see, and a silent host-temp default is exactly what made the Docker failure so confusing.shellQuotemoves into@lde/task-runnersparql-qleveralready carried a privateshellQuote, added after an apostrophe in's-Hertogenboschsilently produced 0-triple imports. Rather than copy it, it moves to the layer both consumers share, with tests that round-trip awkward filenames through a real shell;sparql-qlevernow imports it. Its coverage thresholds drop by 0.02pp because the helper it no longer holds was fully covered.Also in this update
docs/reference/sparql-anything.mdand registered in the sidebar and packages overview; the README is back to the npm-page shape. The docs site became the source of truth on main after this branch was opened.0.0.0, per the versioning guidance now on main – which supersedes this branch's owndocs: start from 0.1.0 for new packagescommit, dropped in the rebase.@lde/task-runneris depended on as^0.2.13, matching the workspace version as@nx/dependency-checksrequires;nx releaserewrites it when task-runner is bumped.Remaining review findings – concurrency, JVM heap and a CLI passthrough – stay in #782 and follow after this merge.
Note for the maintainer:
@lde/sparql-anythingis not on npm yet, so the release run on merge will fail to publish it until it is bootstrapped manually. Bootstrap after that release run, so the manifest already carries the released@lde/task-runnerrange.