docs: onchain AI guide

[marc0olo]

Summary

• Introduces the LLM canister (w36hm-eqaaa-aaaal-qr76a-cai) as an onchain AI inference service
• Covers Prompt API (one-shot) and Chat API (multi-turn) with Motoko and Rust examples using language-synced tabs
• Explains determinism guarantees (random beacon seeding) and how this differs from HTTPS outcalls to external AI providers
• Documents current model support (Llama 3.1 8B), output limit (1000 tokens), streaming not currently supported, and free rollout period
• Uses .mdx for Motoko/Rust language-synced tab components

Note: npm run build reports a pre-existing failure in https-outcalls.mdx (references the examples submodule not initialized in worktrees); this page does not cause or worsen that failure.

Sync recommendation

informed by dfinity/examples — motoko/llm_chatbot, rust/llm_chatbot; limits verified against dfinity/llm; no dedicated icskill exists for on-chain AI

[marc0olo]

Review: Onchain AI

Must fix

• icp deploy --network ic is an invalid command: icp deploy has no --network flag — verified against .sources/icp-cli/docs/reference/cli.md. The only targeting flag for icp deploy is --environment / -e. The correct command is icp deploy -e ic. This will error for any developer who copies it. Also fix the follow-up icp canister call --network ic ... to use -e ic for consistency with the official deploying-to-mainnet guide pattern (icp canister call my-canister greet '("World")' -e ic). Note: --network does technically exist for icp canister call (it takes a URL or network name), but the standard guide uses -e ic, and mixing patterns on the same page is confusing.

• Open Needs human verification flag must be resolved before merge: Line 249 contains {/* Needs human verification: confirm the exact limits (10 messages, 10 KiB, 200 tokens) and whether streaming is supported in the current release */}. This flag is unresolved and cannot be removed by an agent (the limits aren't in .sources/), but it must not ship to users — MDX comments are stripped in production by the agent-docs plugin, so it won't appear on the site. However, per review guidelines, the flag should be investigated and either resolved or escalated. The values (10 messages, 10 KiB, 200 tokens) match what's in the Limitations table, so the table contents likely came from the author's knowledge. Human confirmation is needed before these numbers are presented as authoritative, especially since the LLM canister is in rollout.

Suggestions

• Local testing mock is Motoko-only and missing wiring instructions: The mock at lines 257-265 only shows Motoko, with no Rust equivalent. The rest of the page uses Motoko/Rust tabs throughout. Additionally, the mock is a separate actor but the page doesn't explain how to configure your canister to route to the mock instead of the real LLM canister — a developer following this guide won't know how to wire it up. Consider either: (a) adding a Rust mock and explaining that you'd swap the canister ID in your icp.yaml or pass it as an init argument, or (b) replacing this mock section with a brief note pointing to a testing guide. As written, the mock is not actionable.

• Streaming not covered: The content brief explicitly lists "(5) Streaming responses" as a required topic. The Needs human verification comment asks whether streaming is supported. If streaming is not yet available, the page should explicitly say so (e.g., "Streaming is not currently supported — the LLM canister returns the full response once inference completes"). If streaming is available, it should be documented. Either way, the gap should be addressed rather than left as a verification flag.

• ic-cdk version in Cargo.toml is underspecified: Line 69 shows ic-cdk = "0.17" alongside ic-llm = "1.1.0". The examples repo pin is ic-cdk = "0.17.1". While 0.17 resolves to the latest 0.17.x, the examples repo is more specific. Consider pinning to 0.17.1 to match the verified source.

• Response determinism claim may be misleading: Line 25 states "Inference is seeded from ICP's random beacon, making results deterministic per execution round and verifiable by the subnet." Deterministic AI inference is a notable property. If it is not guaranteed at the query level (e.g., only at replicated update level), the current framing could mislead developers. Worth a human review of this specific claim.

Verified

• All internal links resolve correctly: https-outcalls.md resolves to https-outcalls.mdx; data-persistence.md resolves to data-persistence.mdx; ../../concepts/app-architecture.md exists. All three links are valid in Astro.
• LLM canister ID verified: w36hm-eqaaa-aaaal-qr76a-cai confirmed against both .sources/examples/rust/llm_chatbot/dfx.json and .sources/examples/motoko/llm_chatbot/dfx.json.
• Dependency versions verified: ic-llm = "1.1.0" and ic-cdk = "0.17.1" confirmed in .sources/examples/rust/llm_chatbot/backend/Cargo.toml; llm = "2.1.0" confirmed in .sources/examples/motoko/llm_chatbot/mops.toml.
• Prompt and Chat API code verified: Both Motoko and Rust examples verified against .sources/examples/motoko/llm_chatbot/backend/app.mo and .sources/examples/rust/llm_chatbot/backend/lib.rs. Function names, argument types, and builder chains match exactly.
• Array.concat verified: Exists in .sources/motoko-core/src/Array.mo with correct signature. Used correctly in the conversation-building example.
• Tab order and content rules compliance: Motoko then Rust throughout; .mdx justified by language-synced tabs; no dfx references; no banned URLs (internetcomputer.org/docs or docs.rs); JSX comments ({/* */}) used correctly; onchain spelled without hyphen; no headings inside <TabItem> blocks.
• Frontmatter complete: title, description, and sidebar.order present and consistent with page content.
• Examples repo links: GitHub links to dfinity/examples/tree/master/rust/llm_chatbot and motoko/llm_chatbot — both directories exist in .sources/examples/.
• Upstream comment format: {/* Upstream: informed by dfinity/examples ... */} correctly formatted for .mdx at end of file.

[marc0olo]

Feedback addressed:

• Fixed all --network ic occurrences in deploy and canister call commands — replaced with -e ic (the correct --environment flag for icp deploy; consistent form used for icp canister call too)
• Pinned ic-cdk = "0.17.1" (exact version from .sources/examples/rust/llm_chatbot/backend/Cargo.toml)
• Verification flag ({/* Needs human verification: limits and streaming */}) intentionally left in place — no source material in .sources/ documents the 10-message, 10 KiB, 200-token limits or streaming support; human confirmation required before removing

[marc0olo]

Feedback addressed:

• Corrected output token limit: ~200 tokens → 1000 tokens (verified against dfinity/llm repo — MAX_OUTPUT_TOKENS = 1000)
• Added Max output tokens | 1000 row to limitations table
• Added streaming row to limitations table: Streaming | Not supported
• Added explicit prose note: "Streaming is not currently supported — the LLM canister returns the complete response when inference finishes."
• Removed {/* Needs human verification */} comment — all values are now verified from source
• Updated <!-- Upstream: --> comment to credit dfinity/llm for the verified limits