docs: restructure README as qmd-ja native documentation#6
Merged
Conversation
The MCP HTTP server hardcoded listen(port, "localhost"), so it is unreachable from any non-loopback address. In a container, a liveness probe connects from the pod IP rather than 127.0.0.1, and the bind refuses that connection. Add a --host flag with a QMD_HOST environment-variable fallback. The default stays "localhost", so out-of-the-box behavior is unchanged. Set --host 0.0.0.0 to accept off-host connections (e.g. k8s probes). - server.ts: startMcpHttpServer binds options.host ?? QMD_HOST ?? "localhost" - cli: --host flag, forwarded to the daemon spawn and shown in startup logs - README: document --host / QMD_HOST
…ing SQLITE_BUSY Running multiple `qmd query` invocations against the same index in parallel (e.g. an agent fanning out searches) caused N-1 of N processes to fail immediately with `SQLiteError: database is locked` from `initializeDatabase`. The first DDL statement (`DROP TRIGGER IF EXISTS documents_ai`) hits the write lock; with `busy_timeout = 0` (the default for both `bun:sqlite` and `better-sqlite3`), the loser throws on contact instead of waiting. WAL improves read-while-write concurrency but does not serialise concurrent writers. Only the timeout does that. Setting `PRAGMA busy_timeout = 5000` in `openDatabase` makes any connection wait up to 5s for the write lock before failing. Initialization runs in <100ms, so the worst-case wait for typical agent fan-out (5-10 processes) is ~1s and every process eventually succeeds. Three tests in test/db.test.ts cover the PRAGMA round-trip on a single and multiple connections, plus a behavioural check that SQLite honours the configured timeout under real lock contention.
index.yml — qmd's declarative collection config — had no README documentation at
all, despite being the single source of truth for every collection's path, glob,
ignore rules, contexts, and update command. This documents it in full, with the
per-collection `update` hook the maintainer publicly called under-documented as the
centerpiece, and overhauls example-index.yml so the repo's starter template
actually teaches the schema.
- README: new "Configuring index.yml" section — annotated YAML example plus a
key-reference table for global_context, editor_uri, models.*, and per-collection
path/pattern/ignore/update/includeByDefault/context, with file-location rules
(XDG_CONFIG_HOME, QMD_CONFIG_DIR, named {name}.yml, project-local .qmd/index.yml).
- README: "Automatic update commands" subsection — execution via bash -c in the
collection's directory, run-then-reindex order, non-zero-exit abort, and the
qmd collection update-cmd shortcut, with a sample qmd update trace.
- README: ignore documented as YAML-only and additive with the un-overridable
built-in exclusions; XDG_CONFIG_HOME/QMD_CONFIG_DIR added to the env-var table;
Model Configuration cross-references the models:/QMD_EMBED_MODEL override path.
- example-index.yml: rewritten into a fully-commented starter template where each
collection demonstrates a distinct feature; models: left commented so it can't
drift from the live defaults.
Every key verified against src/collections.ts and its consumers (src/cli/qmd.ts,
src/store.ts). example-index.yml parses as valid YAML. Companion to #715.
CLAUDE.md's command list had drifted from the CLI. Most notably it documented `qmd update --pull` as "git pull first," but `--pull` is parsed and never consumed, so it's a no-op — the same dead-code claim #715 removed from the README. This brings CLAUDE.md in line and refreshes the rest of the cheat sheet. - Fix --pull: point to the per-collection `update` hook (qmd collection update-cmd), the real mechanism that runs automatically on `qmd update`. - Add commands that now exist: qmd init, qmd doctor, qmd bench. - Add collection subcommands: show, update-cmd, include, exclude. - Lead output options with --format <kind> (cli|json|csv|md|xml|files); keep the bare --json/--csv/--md/--xml/--files booleans noted as legacy aliases. - Document `qmd get <file>[:from[:count]]` and add --intent, --no-rerank, --full-path, --no-line-numbers. Cheat-sheet scope only: Bun guidance, the Do-NOT sections, Architecture, and Releasing are untouched. Every command/flag verified against src/cli/qmd.ts.
`qmd doctor` could report a downloaded, working model as "invalid GGUF":
⚠ model cache: invalid 1: generation: hf:tobil/qmd-query-expansion-1.7B-gguf/...
(.../qmd-query-expansion-1.7B-q4_k_m.gguf.etag: not valid GGUF
(expected magic "GGUF", got "\"c38", 0 KB)). Next: run `qmd pull --refresh`
Root cause: `findCachedModelInspection()` scans the model-cache dir for every
entry whose name `includes(filename)` and inspects each as a GGUF. `qmd pull`
writes a `<filename>.etag` HTTP sidecar next to each blob, and that sidecar's
name also contains `filename`, so it gets inspected — and a ~67-byte etag has
no `GGUF` magic. The loop returns on the first *valid* GGUF it finds, so the
false positive only surfaces when `readdir` happens to yield the sidecar before
the blob; that's why it hits some models/platforms and not others, and why
`qmd pull --refresh` doesn't clear it. (It's also invisible to users who let
models download lazily, since only `qmd pull` writes the sidecar.)
Fix: skip `*.etag` entries in the scan. One line in `findCachedModelInspection`.
Adds a regression test that stages a valid blob plus a garbage `.etag` sidecar
and asserts `qmd doctor` reports the model valid. The test fails on the prior
code (reproduces `invalid 1: ... .etag: not valid GGUF`) and passes with the fix.
…Y_TIMEOUT The initial 5s ceiling was sized for init-time DDL contention (sub-100ms work). On multi-GB indexes, a single `embed` batch commit can outlast 5s, so an `update` or `query` that races a long-running `embed` still hits `SQLITE_BUSY` on the first write. Raise the default to 120000 ms, which outlasts the worst-case batch commit observed on multi-GB / multi-tens-of-thousands-of-doc indexes. Add `QMD_SQLITE_BUSY_TIMEOUT` (milliseconds) as an operator escape hatch. Unset, empty, or unparseable values fall back to the default; `0` restores upstream fail-fast behaviour for environments that prefer surfacing contention as an error. Tests expand from 3 to 6 cases: default value, multi-connection default, env-override honored, env=0 fail-fast, garbage value falls back to default, and the existing real-lock-contention timing check (unchanged). Verified under both `bun:sqlite` and `better-sqlite3` runtimes; `tsc` clean.
Runtime detection only looked for lockfiles inside the package directory. Package-manager installs never place one there: a bun global install writes $BUN_INSTALL/install/global/bun.lock while the package lands in .../install/global/node_modules/@tobilu/qmd/, so bun-installed qmd always ran under whatever node was first on PATH. The native modules were built by the installing runtime, so the next Node major upgrade breaks every store-touching command with ERR_DLOPEN_FAILED (NODE_MODULE_VERSION mismatch). When the package directory has no lockfile, apply the same npm-priority rule one level up, at the install root above node_modules/. npm and Homebrew globals keep no lockfile at either level and still default to node. Adds regression tests covering the bun-global install-root layout and install-root npm priority.
Add an optional plain-text "query" parameter to the MCP query tool, mutually exclusive with the existing "searches" array. When "query" is provided it is handed to the SDK auto-expansion path (expand -> RRF fuse -> rerank) instead of requiring the caller to hand-author typed lex/vec/hyde sub-queries. "searches" becomes optional; the handler validates that exactly one of query/searches is supplied and returns a clear error otherwise. The typed "searches" path is unchanged, and snippet extraction uses the plain query when present, else the first lex/vec sub-query as before. This makes natural-language search the recommended default for MCP clients while keeping precise typed control available.
rebuildFTSForCjkNormalization() loaded every active document body into the JS heap via Statement.all() before inserting, inside a single transaction that first cleared documents_fts. On a large index this OOMs the process at store open, and a crash mid-rebuild leaves FTS empty with the version marker unset, so every subsequent open retries the same failing migration. Stream rows with Statement.iterate() and commit in 500-row batches into a documents_fts_rebuild shadow table, then atomically swap it into place (legacy_alter_table=ON so the documents_ai/au/ad triggers re-resolve against the renamed table). The live index keeps serving until the swap succeeds; a crashed prior run leaves only the shadow table, which the next open drops and rebuilds. Adds Statement.iterate<T>() to the db wrapper interface and 6 regression tests (OOM-safe open, crash recovery, CJK + Latin search, inactive-doc exclusion, no-.all() structural).
ALTER TABLE ... RENAME triggers SQLite 3.25+ re-validation of every trigger/view body that references the renamed table. The documents_ai and documents_au triggers reference documents_fts by name; renaming documents_fts_rebuild after DROP documents_fts fails with "no such table: main.documents_fts" before the rename can complete. Replace the RENAME path with: DELETE FROM documents_fts INSERT INTO documents_fts(...) SELECT ... FROM documents_fts_rebuild DROP TABLE documents_fts_rebuild The live table and its triggers stay untouched until the transaction commits. No PRAGMA legacy_alter_table workaround needed. Update the structural test assertion to match (was checking for "RENAME TO documents_fts", now checks for "INSERT INTO documents_fts"). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
qmd embed runs inside a session capped at a hardcoded 30 minutes. When the cap is reached the remaining document batches are skipped and you re-run embed to pick up where it left off. On a large corpus that turns one index into several supervised restarts. --timeout <minutes> lets you set the cap yourself, or pass 0 to remove it, so a big index can finish in a single run. The default stays at 30 minutes and the skip-and-resume behavior is unchanged when the cap is hit. Wired through the embed CLI option, vectorIndex, and generateEmbeddings (new EmbedOptions.maxDurationMs), with input validation and a unit test covering early stop.
The tool was renamed from structured_search to query (per CHANGELOG, at/before 2.1.0) but skills/qmd/references/mcp-setup.md was never updated. The binary and skills/qmd/SKILL.md already use 'query'; this aligns the setup reference. Refs #741
Concurrent processes opening the same index could crash during store initialization with `trigger documents_ai already exists` or `database is locked`, even with busy_timeout set. The FTS sync triggers were dropped and recreated on every open as separate autocommit statements, so two connections interleaved between the DROP and the CREATE (A drops, B drops, A creates, B creates -> "already exists"); busy_timeout serialises individual statements but not the DROP/CREATE pair. Separately, `PRAGMA journal_mode = WAL` needs a brief exclusive lock to migrate a cold database and does not invoke the busy handler, so concurrent first opens threw SQLITE_BUSY. Gate the trigger rebuild behind PRAGMA user_version inside one IMMEDIATE transaction with a double-checked read, so the DROP/CREATE pair is atomic across connections and runs once per schema version. Move WAL setup into openDatabase with a bounded retry within the busy-timeout budget, alongside busy_timeout. Add a multi-process regression test (cold and existing database) that fails before and passes after.
docs: rename MCP search tool heading `structured_search` -> `query`
docs: refresh stale command reference in CLAUDE.md
docs: document the index.yml configuration schema
feat(mcp): add --host flag (and QMD_HOST env) to the HTTP server
Increase multi_get default maxBytes
…lse-positive fix(doctor): skip `.etag` sidecars when validating the model cache
fix(launcher): detect bun global installs via the install-root lockfile
fix(store): stream-batch CJK FTS rebuild to prevent OOM on large indexes
feat(mcp): expose plain query parameter for auto-expand search
Report ignored documents distinctly
feat(embed): add --timeout to override the 30-minute session cap
fix(db): guard SQLite store init against concurrent opens
…eady fix(db): make concurrent store initialization safe
chore: release v2.6.3
- Resolve merge conflicts in package.json, src/store.ts, README.md,
CHANGELOG.md against upstream/main (v2.6.3).
- Bump version to 2.6.3-ja.1; preserve qmd-ja name and ONNX/Vaporetto
configuration throughout.
- Adopt upstream embed --timeout support via options?.maxDurationMs in
generateEmbeddings().
- Fix upstream test incompatibilities with qmd-ja environment:
- store-paths: skip Git Bash path tests on non-Windows (WSL2 compat)
- store-cjk-fts / store-concurrency: import FTS_CJK_NORMALIZED_VERSION
from src/store.js instead of hardcoding "1" (qmd-ja uses "3")
- cli: comment out GGUF-specific doctor assertion (qmd-ja uses ONNX)
- Add qmd-ja-specific tests (34 tests across 2 new files):
- test/providers.test.ts: unit tests for ONNX provider routing logic
- test/store-ja-vaporetto.test.ts: Vaporetto tokenizer and Japanese
BM25 search end-to-end tests
- Replace upstream title with qmd-ja focused heading - Add architecture diagram note: Vaporetto tokenizer and Japanese models - Update What's different table to include model-level differences - Add Japanese Optimization section (Vaporetto WASM + recommended models) - Update generate model to adsholoko/qmd-query-expansion-ja in Japanese config examples - Simplify installation note to a single upstream reference line
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
# QMD - Query Markup Documents）を## qmd-ja — On-Device Search Engineに置き換えWhat's differentテーブルにモデル差分行を追加（3モデルすべて）## Japanese Optimizationセクションを新設（Quick Start の前）：Vaporetto WASM 説明 + 推奨モデル設定adsholoko/qmd-query-expansion-jaに更新（Japanese ONNX section・Model Configuration）Test plan
npm run test:unit— 913 pass / 77 skip (vitest)bun test test/store.test.ts— CJK テスト含め pass🤖 Generated with Claude Code