Complete the Diamond repository implementation contract

[kcsfelty]

Purpose

Complete Diamond in one continuous repository-owned implementation pass. This is the sole active implementation tracker for Diamond. It has no parent program issue.

Scope

• Preserve RDF/JS fidelity, atomic quad storage, prepared RDF patches, SPARQL behavior and security controls, W3C conformance, migrations, integrity, and package compatibility.
• Finish the portable SQLite/D1 capability used by downstream packages: bindings and BLOBs, integer fidelity, ordered atomic batches, rollback, connection isolation, persisted restart, migration checksums, and real workerd/native conformance.
• Add only generic low-level SQLite capabilities required for lexical or vector storage. Do not implement relevance ranking, domain authorization, search cursors, canonical records, lifecycle policy, UI, runtime assembly, or hosted deployment.
• Verify public package metadata, trusted publishing/provenance, dependency and security scanning, protected main, least-privilege workflows, and release controls.
• Preserve or supersede the useful work on the local SQLite-conformance branch.

Required proof

Run the complete package check, W3C conformance, benchmarks, real persisted native SQLite and workerd D1 conformance, rollback/fault/restart tests, Worker bundling, and exact packed-consumer verification.

Completion

Close only when Diamond's clean and packed gates pass, existing RDF/SPARQL behavior remains compatible, its portable storage contract is sufficient for consumers, repository/release security is verified, and the handoff records exact versions, commands, results, API/migration changes, performance evidence, and any external interface mismatch without editing another repository.

[kcsfelty]

Partial evidence accepted — Diamond PR #48

Merged as 4916d93 with post-merge main quality, Node 22/24, W3C conformance, benchmark-smoke, audit, secret-scan, and CodeQL checks green.

The PR provides narrowly scoped prerequisite evidence:

• one shared persisted Node SQLite + real Miniflare/workerd D1 capability suite;
• observable conditional UPDATE ... RETURNING semantics: exactly one winner and stale revision no-op;
• scalar/BLOB bind/read parity through validated readSqliteBytes() normalization, including sparse/malformed rejection;
• retained bounded child-process native lock-contention evidence.

This does not complete #46 or provisional IDs D01-D05: ranked lexical retrieval, snippets, filters, current-state/auth exclusion, index lifecycle, rebuild, and persisted-search restart remain dependent on the Taproot public/projection contract and later shared lexical conformance. Keeping the issue open.

[kcsfelty]

Ownership corrected authoritatively: #46 is now generic SQLite/D1 substrate qualification only. PR #48 is retained as legitimate cross-runtime conditional-write/BLOB evidence, but ranking, auth, public search, documents, cursors, lifecycle, rebuild, and health are explicitly Taproot-owned. No product ledger ID or D01-D05 completion is claimed, and the issue remains open.

[kcsfelty]

Dependency clarification: conditional substrate only

#46 is not an unconditional step in the approved Phase-A critical path. Taproot #15 owns lexical ranking, authorization, search documents, cursors, lifecycle, rebuild, and health. Only if a #15 implementation/review identifies a concrete missing generic SQLite/D1 capability should it hand that narrowly specified gap to Diamond, with the same low-level test against both runtimes.

PR #48 remains accepted conditional-write/BLOB substrate evidence. It does not complete D01-D05 and does not need speculative follow-on work before Taproot freezes the consumer requirement. Keep open and conditional.

[kcsfelty]

2026-07-22 conditional status

This issue remains open and conditional, not an unconditional critical-path implementation step. Taproot #15 must first freeze its lexical port and shared conformance harness. Open Diamond implementation only if that harness demonstrates a concrete generic D1/native-SQLite substrate gap.

Merged Diamond PR #48 is reusable low-level evidence but completes 0 Search Contract IDs. Diamond must not own ranking, domain projections, authorization policy, search lifecycle or product acceptance. Authoritative total remains 3/105.

[kcsfelty]

Complete implementation evidence — PR #52

Commit 3ea59cb on draft PR #52 completes the remaining Diamond-owned implementation-contract work.

Contract section evidence

• 1 — preserve Diamond product: the unchanged RDF/SPARQL surface passed 182 tests, the exact Worker bundle/runtime consumer, Wikibase example, and 490/490 applicable W3C cases. No search routing or approximate graph behavior was introduced.
• 2 — portable SQLite: native unsafe numeric binds now fail before SQL; native out-of-range integer rows remain bigint; D1's explicit bigint rejection is recorded. One shared persisted native/workerd suite now proves positional ordered results, mutation metadata, rollback at boundaries 0/1/2 and commit-time deferred-FK failure, concurrency, foreign connection rejection (native), close/reopen, BLOB fidelity, migrations, checksum drift, and interrupted recovery.
• 3 — search substrate only: added generic UTF-8/BLOB measurement and explicit 1.9 MB D1-safe payload bounding. The exact tarball consumer composes caller SQL and prepareQuadPatch() in one batch and reopens persisted BLOB, bigint, domain-row, and RDF state. No ranking, projection, authorization, cursor, lifecycle, or health semantics were added; 0 Search Contract IDs are claimed.
• 4 — security/operations: public MIT/npm metadata and npm SLSA provenance for 0.4.0 verified; pinned least-privilege workflows, Dependabot, CodeQL, secret scan, production audit, license allowlist, public-only OIDC provenance release wiring, SBOM/checksum/attestation wiring remain intact.

Clean commit verification

• npm run check — pass; 182/182 tests, 96.11% statements, 92.05% branches; build, Worker dry-run/local workerd, 61-file exact tarball, packed native/prepared-patch consumer, licenses, readiness
• npm run conformance — 490/490
• npm run benchmark:check — 10 scenarios validated
• npm run benchmark:storage:check — pass
• npm audit --omit=dev --audit-level=high — 0 vulnerabilities
• GitHub CI #79 — success: quality, Node 22, Node 24, W3C summary/EARL, benchmark
• GitHub Security #79 — success

Package remains @gnolith/diamond@0.4.0 with Unreleased changelog entries. Public additions: MAX_PORTABLE_SQLITE_BIND_BYTES, sqlitePayloadByteLength(), assertSqlitePayloadSize(), and SqlitePayload. Native integer row behavior is precision-safe. No migration changed.

External settings and residual limits

The connector confirms gnolith/diamond is public with main as default; npm confirms MIT metadata and SLSA provenance. Repository files prove least-privilege pinned Actions and OIDC/provenance release wiring. This environment cannot read the GitHub admin pages for branch protection, required checks, secret scanning/push protection, vulnerability alerts, automated fixes, or the npm trusted-publisher binding, so those settings still require an owner/admin UI check. The current public npm artifact proves provenance, but not the present trusted-publisher configuration.

Issue remains open until #52 is merged to main.

[kcsfelty]

Released — v0.4.1

The Diamond implementation contract is now merged and released.

• PR #52 protected-squash merged.
• Main/release commit: 0d3c6ceda09b7124c32835a985d7359057b2c735
• Immutable tag: v0.4.1, peeled exactly to the main commit.
• npm: @gnolith/diamond@0.4.1
• Release workflow: run 29944747445, success.
• Post-merge main CI 29944725510 and Security 29944725503, success.

Release workflow completed main ancestry verification, clean install, full package check, 61-file artifact verification, npm pack, SBOM/checksum generation, build-provenance attestation, OIDC npm publication, and GitHub Release creation.

Independent public verification:

• npm integrity: sha512-DMibXt/WDezuZM/dDFXx4G8pyFoC6bZ9uOODIfg+To0/vG63BGVfEHgxm2ZuPCuPQh6T7NU+cIe6DE/f+H5o/A==
• npm shasum: 97475a5156c44637bf619c18868ebcbc54fc9c39
• published package: 61 files, 65,968-byte tarball, 254,647 bytes unpacked
• GitHub tarball SHA-256: 084718233136619dd935b251fbc0bd5b539085bad7dd63210d2c06b915b078e4
• SBOM SHA-256: 71513860d608967ab9be61aa59ca015fc1c7ba6c6b03c3878a35ca40f9cdb2b9
• downloaded Release assets matched SHA256SUMS
• npm registry exposes both publish and SLSA provenance attestations; provenance resolves to release.yml, refs/tags/v0.4.1, run 29944747445, and commit 0d3c6ce
• fresh public-registry consumer passed persisted native SQLite, BLOB normalization, bigint fidelity, caller-owned SQL plus prepareQuadPatch(), RDF reopen, and exact public imports
• npm audit signatures: 538 verified registry signatures and 11 verified attestations

No schema migration changed. No Gnolith Site was provisioned, deployed, or accepted, and no cross-repository/full-stack compatibility claim is made here. Search semantics remain outside Diamond. Closing #46 as completed.