Custom Sources

Custom Sources

Point QQL at your own JSON and give it a short code. No Rust, no rebuild, and it works through the CLI, the C ABI and Dart alike.

The quickest version

Drop a qql-sources.json in your data directory. It is picked up automatically, on the first query.

[
  {
    "code": "X",
    "name": "My Collection",
    "aliases": ["MINE"],
    "path": "mydata/{primary}.json",
    "items": "lines",
    "ar": "arabic",
    "en": "translation",
    "primary_key": "chapter",
    "container_metadata": { "chapter_title": "title" },
    "metadata": { "note": "note" }
  }
]

qql --data ./mydata 'X:1:2'
qql --data ./mydata --sources        # X now listed

Spec reference

Key	Meaning
code	Short code. Uppercased automatically
name	Display name, used as collection in results
aliases	Extra codes that select this source
path	Data file, relative to the data directory. {primary} → one file per chapter; omit it for a single file
items	Dotted path to the array of items. Empty means the file is the array
ar, en	Dotted paths within an item. "english.text" reaches nested fields
chapters + chapter_id	For single-file books: the chapter array, and the field matched against the primary
item_id	Match items by a field instead of by position
primary_key	Names the primary in output (surah, chapter, …). Default primary
metadata	Extra output fields from the item: output key → dotted path
container_metadata	The same, from the chapter or file
flat	Enables X::100 and unscoped search — { "path", "items", "item_id" }

The two layouts

One file per chapter, items at the top level:

{
  "code": "X",
  "name": "My Collection",
  "path": "mydata/{primary}.json",
  "items": "verses",
  "ar": "text",
  "en": "translation"
}

One file for everything, chapters selected by an id field:

{
  "code": "S",
  "name": "Single File Book",
  "path": "book.json",
  "chapters": "book.chapters",
  "chapter_id": "id",
  "items": "entries",
  "ar": "text",
  "en": "en.deep"
}

Chapters are matched by their id, never by array position — the same rule the Hisnul Muslim resolver needs, because that file stores chapter 27 first.

Book-wide numbering

Add a flat block to enable X::100 and unscoped search:

"flat": { "path": "all-lines.json", "items": "all", "item_id": "n" }

Without it, X::100 and X:"term" are refused with QQL_UNSUPPORTED naming the fix, rather than quietly searching a narrower slice.

What you get for free

Everything that applies to built-in sources applies here: query order preserved, duplicates dropped within a reference, ranges bounds-checked, groups, sticky sources, exact search, and JSON errors.

From Rust

let mut ctx = qql::Context::new("mydata");
ctx.register_spec(spec)?;                     // a SourceSpec
ctx.add_sources_from("other-sources.json")?;  // a whole manifest

Sources are searched newest-first, so registering an existing code shadows it — that is how a custom source replaces a built-in one. The data-directory manifest loads on the first query, so it lands after anything registered manually; call ctx.load_manifest()? first if you mean to override something it defines.

When a spec is not enough

Write a real impl Source when the data is too irregular for a declarative mapping to express honestly. The Hisnul Muslim resolver exists for exactly that reason: its file has two objects with duplicate keys, one misspelled field, and a UTF-8 BOM.

use qql::{Error, Record, Reference, Repository, Source};

pub struct MyCollection;

impl Source for MyCollection {
    fn code(&self) -> &str { "X" }
    fn name(&self) -> &str { "My Collection" }

    fn resolve(
        &self,
        repo: &mut Repository,
        reference: &Reference,
        out: &mut Vec<Record>,
    ) -> Result<(), Error> {
        // Reject impossible numbers, load through `repo`, push records in
        // query order. `reference.expand(total)?` handles ordering, dedup and
        // bounds in one call — do not re-implement any of the three.
    }
}

Then add the module, one line to Registry::with_defaults, and the data. Adding a source must never require touching the lexer or parser — if your diff does, the approach is wrong. See Architecture.

Search over custom sources

Exact search works immediately. The optional engines need indexes:

• vector — scripts/build-vectors.py only knows the built-in corpora today; a custom source needs its reader added there.
• fulltext — qql-index enumerates through the ordinary query path, so it indexes any registered source, custom ones included.