[Feature]: Consumer + agent documentation for using SKaiNET as a library

[fiwio-developer]

🧠 D: DOCUMENT — Problem & Opportunity

What is the problem, limitation, or opportunity? Why does this matter for SKaiNET?

SKaiNET has no documentation page that answers the first question every downstream user asks: "how do I add SKaiNET to my project and start using it?" The information exists, but only as a by-product scattered across task-specific tutorials — tutorials/java-getting-started.adoc, tutorials/data-sources-getting-started.adoc, how-to/io-readers.adoc and others each re-explain the BOM and a different subset of the sk.ainet.core:skainet-* artifacts. There is no single "Add SKaiNET to your project" page in nav.adoc, and no page that explains which artifacts a consumer actually needs for inference vs. training vs. a specific model format.

The second, less obvious problem is that SKaiNET is DSL-heavy and therefore unusually hostile to AI coding assistants. sequential { }, dag { }, tensor { }, pipeline<T> { }, the DType/Phase/ExecutionContext type parameters — none of this is inferable from a class listing, and an agent that guesses will produce code that looks plausible and does not compile. Agents fall back on generic PyTorch/Keras patterns or on stale web results, and the failure is silent.

We already have a partial answer to this: the SKaiNET-developers/skainet-coding-skills marketplace ships seven consumer skills (skainet-consumer-setup, skainet-data-dsl, skainet-nn-dsl, skainet-inference, skainet-model-loading, skainet-java-consumer, skainet-android-integration). But those skills are (a) Claude Code specific, (b) hand-written prose that drifts from the real API as it evolves, and (c) entirely undocumented in the Antora site — a consumer reading our docs has no idea they exist.

Library-Insight shows the complementary half of the answer. It is a JVM API explorer that runs as both a CLI and an MCP server: it scans the actual JAR/AAR on your classpath, reads .class files and Kotlin @Metadata, and answers version-correct questions about classes, signatures, nullability, generics and DSL markers. Its stated purpose is exactly our problem — stopping AI assistants from inventing API calls from outdated documentation. It even calls out Kotlin DSL support and @DslMarker mapping specifically.

The opportunity is to combine both: hand-written intent documentation (what the DSL means, which artifact to pick, what the idioms are) with machine-generated fact lookup (what signatures actually exist in the version you depend on).

Summary (1–2 sentences):

Add a consumer-facing "Using SKaiNET as a library" documentation track to Antora, covering dependency setup, artifact selection and the DSL idioms that agents get wrong; and evaluate a Library-Insight-style MCP/CLI so that any AI agent — not just Claude Code — can answer SKaiNET API questions against the exact published version rather than guessing.

---

🔍 A: ASSESS — Feasibility & Impact

✔️ Feasibility

The documentation half is straightforward and needs no new code:

• The Antora structure and the Diátaxis split (Tutorials / How-to / Reference / Explanation) already exist; this is new pages plus nav.adoc entries.
• The content largely exists but is scattered — much of it can be consolidated from the tutorials and from the seven consumer skills in skainet-coding-skills, which are already written and reviewed.
• Dokka already publishes a full API reference at /api/, and reference/ops-status-matrix.adoc / reference/kernel-support-matrix.adoc are already generated from build output. There is precedent for generated reference pages.

The MCP/CLI half needs a decision before it needs code. Three broad options, in increasing cost:

1. Document only — publish an "AI-assisted development" page pointing at skainet-coding-skills and at the generated matrices. Zero new code.
2. Generate an agent context artifact — emit a compact, machine-readable API digest (llms.txt-style, or JSON) from the existing KSP operator metadata plus Dokka output, publish it with the docs site, and let any agent fetch it. Moderate cost, reuses GenerateDocumentationTask machinery in build-logic.
3. Ship an MCP server — a skainet-mcp module (or a Library-Insight configuration) exposing tools like skainet_find_operator, skainet_dsl_example, skainet_backend_support. Highest cost, highest payoff, and a maintenance commitment.

Note that Library-Insight can likely serve option 3 without us writing anything — it scans published JARs generically. That should be tested before we consider building our own.

✔️ Expected Impact

• Removes the single biggest onboarding obstacle for new users: there is currently no canonical answer to "which artifacts do I need".
• Makes agent-assisted SKaiNET development reliable rather than a coin flip. Given how DSL-heavy the API is, this disproportionately affects us compared to a conventional imperative library.
• Surfaces skainet-coding-skills, which is currently invisible to anyone who has not stumbled across the marketplace.
• A machine-readable API digest also benefits humans — it is the same data that powers the operator coverage matrix.

✔️ Risks / Constraints

• Documentation drift. A hand-written consumer setup page listing artifact coordinates will go stale as modules are added or renamed. Prefer generating the artifact table from skainet-bom (the sk.ainet.transformers.bom-coverage plugin already enumerates every published module) over maintaining it by hand.
• Duplication with skainet-coding-skills. Two homes for the same guidance will diverge. Decide which is canonical — the working assumption should be that Antora is canonical prose and the skills link to it, not the reverse.
• MCP is a moving target. Committing to a hosted MCP server is a long-lived maintenance obligation. Options 1 and 2 have no such tail.
• Third-party dependency. Recommending Library-Insight means recommending a tool we do not control. Worth pinning a version and noting it as a suggestion rather than a supported integration, at least initially.
• No numerical-stability or API-consistency concerns; this is documentation and tooling only.

✔️ Dependencies

• Antora docs component (docs/), nav.adoc, and the docs.yml / documentation.yml workflows.
• skainet-bom and sk.ainet.transformers.bom-coverage for an authoritative published-artifact list.
• The KSP operator metadata pipeline (operators.json → GenerateDocumentationTask) if we generate an agent digest.
• Dokka aggregate output at /api/.
• skainet-coding-skills — cross-repo coordination.
• Library-Insight (evaluation only at this stage).

---

📚 R: RESEARCH — What Must Be Understood First?

Research Tasks

• Run Library-Insight against the published sk.ainet.core:* artifacts and record what it gets right and wrong. Specifically: does it resolve sequential<T, V> { } / dag { } builder scopes usefully, and does it surface @DslMarker receivers in a way an agent can act on?
• Inventory which consumer questions are actually answerable from bytecode (signatures, nullability, generics) versus which need hand-written intent (when to use dag over sequential, why ExecutionContext is threaded through forward). The split determines what the MCP layer is worth.
• Audit the seven consumer skills in skainet-coding-skills for content that belongs in Antora, and identify where they already disagree with the current API.
• Review how comparable Kotlin-first projects document consumption and agent usage — Ktor, Compose Multiplatform, kotlinx-* — and whether any publish an llms.txt or equivalent.
• Determine whether the published-artifact table can be generated from skainet-bom at docs build time.
• Check whether Dokka 2.x can emit a machine-readable API index alongside HTML, avoiding a bespoke extractor.

Open Questions

• Is Antora or skainet-coding-skills the canonical home for consumer guidance? (Recommend Antora; skills become thin pointers.)
• Do we ship an MCP server ourselves, or document Library-Insight as the recommended third-party route?
• Should the agent-facing artifact be a static file published with the docs site (cheap, cacheable, works with any agent) or a live server (richer, but a service to run)?
• Which SKaiNET version does agent tooling target — latest release, or whatever the consumer has on their classpath? Library-Insight's whole premise is the latter.
• Does this need a SKEEP? Per CONTRIBUTING.md a docs-structure change for a long-lived feature area arguably does; a pure docs addition does not. Worth a maintainer call before the MCP half starts.

---

🛠️ C: CODE — Implementation Plan

Deliberately staged so the documentation lands independently of any tooling decision.

Development Tasks

Stage 1 — consumer documentation (no new code)

• how-to/add-skainet-to-your-project.adoc — BOM setup for Gradle KMP / Gradle JVM / Android / Maven, and which sk.ainet.core:skainet-* artifact solves which problem
• reference/published-artifacts.adoc — the full artifact table, generated from skainet-bom rather than hand-maintained
• explanation/dsl-idioms.adoc — the DSL entry points (tensor, sequential, dag, pipeline) and the mistakes an unfamiliar reader (or agent) reliably makes
• Consolidate the dependency-setup preambles duplicated across the existing tutorials to xref the new page
• Add all of the above to docs/modules/ROOT/nav.adoc under Using SKaiNET

Stage 2 — agent support, documented

• how-to/ai-assisted-development.adoc — using SKaiNET with coding agents: the skainet-coding-skills marketplace, what the generated matrices provide, and the failure modes to watch for
• Cross-link from explanation/skainet-for-ai.adoc
• Open a companion issue in skainet-coding-skills so the skills point back at Antora instead of restating it

Stage 3 — machine-readable API surface (gated on research)

• Decide between static digest and MCP server based on the Library-Insight evaluation
• If static: extend the existing build-logic documentation task to emit the digest, and publish it with the site
• If MCP: separate design issue — do not fold it into this one

Acceptance Criteria

• A new user can go from an empty Gradle project to a running model.forward(x, ctx) using only the Antora site, without reading the SKaiNET source
• The published-artifact table is generated, not hand-maintained, and cannot silently go stale
• skainet-coding-skills is discoverable from the docs site
• The DSL idioms page covers every public entry point exercised by the consumer skills
• Docs build clean (./gradlew generateDocs, Antora build in .github/workflows/docs.yml) with no broken xrefs
• Reviewed and approved by maintainers
• Stage 3 either delivered or explicitly deferred with a linked follow-up issue

---

💬 Additional Notes

Reference: Library-Insight — JVM API explorer and MCP server; scans the exact JAR/AAR on the classpath, reads .class files and Kotlin @Metadata, exposes version-correct signatures, nullability, generics and DSL markers to AI IDEs. Its motivation section is a precise statement of the problem this issue addresses.

Prior art inside SKaiNET worth reusing rather than reinventing:

• sk.ainet.documentation (build-logic/convention/.../DocumentationPlugin.kt) already turns KSP-emitted operators.json into Antora pages — the same pipeline could emit an agent digest
• sk.ainet.transformers.bom-coverage (BomCoveragePlugin.kt) already enumerates every published module and is the natural source for the artifact table
• reference/kernel-support-matrix.adoc is precedent for a generated reference page derived from a test that introspects the runtime registry

Scope note: this issue covers documentation and the evaluation of agent tooling. Building and operating an MCP server is explicitly out of scope here and should get its own DARC issue if the research stage recommends it.