Roadmap #10 AI scene assistant + #11 generative 3D - #28
Merged
Conversation
- src/lib/ai/providers.js: local provider config store (Grok/Gemini/custom
vLLM presets); a list of saved {baseUrl,apiKey,model} configs + one active,
JSON-blob persisted in localStorage like avatarConfig
- setters addAiProvider/updateAiProvider/removeAiProvider/setAiActiveProvider/
setAiEnabled + activeAiConfig()/aiReady() helpers
- keys are the first credentials the app stores: plaintext localStorage,
documented in the file header (Reset settings wipes them)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- src/lib/ai/client.js: plain-fetch client (no SDK) for any OpenAI-compatible endpoint; SSE streaming with a line buffer carried across chunks and index-keyed tool_call fragment accumulation - runChat() tool-loop: any accumulated tool_calls count as a tool turn (Gemini-compat defensiveness), malformed args feed an error result back so the model self-corrects, abort threaded through - testConnection() (GET /models -> 1-token fallback) + describeAiError() mapping 401/429/404/5xx and naming CORS for self-hosted vLLM Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… A3) - history.js: beginHistoryBatch/endHistoryBatch collect entries into one composite 'aibatch' step so a whole AI prompt is a single undo; the batch branch in recordEntry returns BEFORE redoStack is cleared (redo only clears when the batch commits) - aibatch applier replays each sub-entry through the existing applyState (reverse order on undo) so create/move/color/delete sub-ops replicate their own peer messages for free - materialsHandler.setObjectColor(): the missing exported color trio (color.set + peer color message + recordMaterialChange); the live gesture- debounced picker sites keep their inline path on purpose Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- src/lib/ai/tools.js: OpenAI-format tools (list_scene, create_objects, update_objects, delete_objects, group_objects, clear_scene), batch-first - executeAiTool maps each tool onto the existing REPLICATED mutation surface (createGeometry/createLight/createGroup with an explicit uuid = no selection steal, move/color/name/group/delete messages, renameObject/moveObjectToGroup/ deleteObjectsByUuid/setObjectColor/setMaterialParam); never throws, returns results so the model self-corrects; skips objects locked by other peers - summarizeScene() walks objectsGroup into a compact uuid/name/transform/color list; buildSystemPrompt() gives Y-up ground-plane guidance Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- src/lib/ai/assistant.js: module-level conversation state (survives closing the window); runPrompt() seeds system prompt + a fresh scene summary each turn (multi-turn refine), streams the model, runs tool calls - wraps the whole run in beginHistoryBatch/endHistoryBatch (in finally) so a partial run on abort/error still commits as one undo step - guards no-provider by toasting + deep-linking Settings; stopAi() aborts Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- AiAssistant.svelte: floating window modeled on Chat (dragWindow/focusStack/ tabbable), streaming message list by role, provider picker, Send/Stop - quick prompt pill hidden by default, toggled by the backquote (`) shortcut (registered via the shortcuts registry so it binds AND lists in Settings); submitting the pill opens the window and runs the prompt - appStore: aiAssistantHidden + aiPromptBarOpen (session-only) stores; mount AiAssistant in Menu.svelte next to Chat Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Settings.svelte: new AI accordion item between Scene and Shortcuts (legacy- mode file, on:/bind: directives) - enable toggle, saved-provider rows with active radio + Edit/Remove, add/edit form (preset prefills baseUrl+model, password key field), Test connection button, plaintext-key warning - settingsSection 'ai' deep-links + expands the section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests/e2e/ai-assistant.test.cjs: drives the assistant against a mocked OpenAI-compatible endpoint (page.route -> canned SSE tool_calls); asserts a prompt creates 3 boxes locally, broadcasts the standard create/color/move replication messages (captured via a send stub), one undo removes the whole batch (aibatch) + broadcasts 3 deletes, and redo restores + re-broadcasts - App.svelte: expose ai/providers, ai/tools, ai/assistant on the __stores hook - receive-path replication rides existing message handlers covered by other two-peer suites; the public cloud is too flaky to gate this suite on Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tools/agent: new Node package (own package.json, ESM) for a console agent
that joins a session as a peer; deps peerjs + @roamhq/wrtc + ws
- polyfills.js installs RTCPeerConnection/WebSocket/navigator globals BEFORE
peerjs imports; RTC impl isolated so it is swappable
- spike.js proves viability: @roamhq/wrtc installs + loads on win32/Node 20
(the main risk), and the peerjs WebSocket handshake reaches the signaling
server (public cloud currently 429-rate-limits repeated connects, which
itself proves the server was reached). GO: no Playwright-page fallback needed
- peerjs Peer constructor is at (await import('peerjs')).default.Peer in ESM
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- messages.js: pure protocol builders + catalogs (create/light/move/color/name/ group/delete/objectParameters + minimal handshake); no peerjs/THREE, unit-tested - peerBridge.js: connection state machine + approval dance (connect -> host closes unknown conn -> human Approves -> host connects back + gossips hosts -> agent reopens a fresh outgoing conn + sends minimal handshake -> connected); broadcasts to every open conn (full mesh); OMITS the modules handshake (would toast module mismatches); send queue gated on conn.open; signaling retry with backoff + a process guard that swallows peerjs's transient uncaught ws 429 crash - registry.js: scene tracking from own sends + observed peer messages; GLTF object syncs become uuid stubs (tracked:'stub') - cli.js --smoke: scripted create/move/color/delete to prove the pipe - test/messages.test.mjs: 10 pure unit tests (builders + registry), all green Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tools.js: same tool names/shapes as the in-app assistant (list_scene, create_objects, update_objects, delete_objects, group_objects, get_status) mapped onto peer messages via messages.js + bridge.broadcast; never throws, returns JSON results; systemPrompt() for the REPL/MCP drivers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- llmClient.js: minimal OpenAI-compatible client (non-streaming tool loop) for any endpoint (vLLM/Hermes/Grok/Gemini) - repl.js: readline loop driving the scene via tool calls; slash commands /status /list /raw /quit; config via --api-url/--api-key/--model or env Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mcpServer.js: stdio MCP server exposing the scene tools + a connect tool so any MCP host (Claude Code) can drive the session; starts immediately and connects in the background, tools report "awaiting approval" until connected so registration never blocks on the human step - verified with a real MCP client: lists all 7 tools, get_status + create route correctly (create reports not-connected without hanging) - add @modelcontextprotocol/sdk dependency Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…p 10 B6) - test/bridge.test.mjs: deterministic mock-peer test of the FULL approval dance (connect -> host closes unknown -> awaiting-approval -> host connects back + gossips hosts -> agent reopens conn -> handshake + flush queue -> connected), send-queue flush, handshake shape (locked/hosts/userdata, modules OMITTED), registry, and shutdown. 16/16 green with no network - tests/e2e/agent-bridge.test.cjs: real browser round-trip (agent joins a session, human Approve, create/move/color/delete land on the host). Rides the public PeerJS cloud + Node WebRTC, so it is flaky/rate-limited (429) like every two-peer suite; re-run after a cooldown. The deterministic bridge/messages/registry tests + the MCP client test cover the same logic offline - README.md: install, modes (repl/mcp/smoke), flags, architecture, limitations - cli.js: drop the redundant uncaughtException guard (peerBridge installs it) - npm test runs both offline unit suites Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2e (roadmap 10) - peerBridge: real WebRTC rarely signals the host's pre-approval conn close, so the post-approval reopen was blocked. Now the inbound conn from the host (the approve-connects-back signal) drives a retrying reopen of our outgoing conn, and _openOutgoing replaces a stale never-opened conn after an 8s grace (mirrors peerHandler's "Restoring connection"). Whitelist inbound peers. Server opts gain a key + default path /peerjs to match the app's self-hosted convention - cli --peer-server defaults to 443 + /peerjs + optional --peer-key - agent-bridge.test.cjs: reads VITE_PEER_* from .env so the agent joins the SAME self-hosted signaling server the browser uses; Users-list check is now polled. FULL round-trip now PASSES (connect + create/color/delete land on the host) - ai-assistant.test.cjs: reload before the pill check to dodge the dev-server HMR dual-module-instance split (the hook + the component's static import can diverge); 12/12 PASS Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lits The AI prompt-pill check flaked because the debugStores hook reaches singletons via dynamic import(), which on an HMR-churned dev server can bind a SECOND module instance — so a value set through window.__stores.<mod> is not the store the component reads via its static import, and the component renders stale. - helpers.freshReload(peer): reload + wait for the hook, collapsing to one module graph. Seed localStorage-persisted state first, reload, then assert component UI; set session-only stores after the reload. - ai-assistant.test.cjs uses it for the pill check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- src/lib/ai/meshProviders.js: LOCAL mesh-backend provider store mirroring ai/providers.js — kinds comfyui (self-hosted TRELLIS) + meshy (hosted); list + active + meshGenEnabled; meshGenReady() gates the feature (comfyui needs a workflow, meshy needs a key). Plaintext keys, same caveat as LLM keys - Settings.svelte: a "Mesh generation" block in the AI section — enable toggle, provider rows, add/edit form (ComfyUI workflow-JSON paste + output node, or Meshy key + preview/refine mode) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- meshAdapters/: one small async interface (submit/poll/fetchResult) per backend.
comfyui.js = the self-hosted TRELLIS path (substitute {{PROMPT}}/{{SEED}} into an
API-format workflow -> POST /prompt -> poll /history -> GET /view GLB; bearer +
--enable-cors-header). meshy.js = hosted (preview->refine task poll -> model GLB)
- meshJobs.js: runner — submit, poll to completion (10min cap, cancel), download,
import; 40MB guard; progress via the meshJobs store; best-effort cache into the
Explorer "Generated" folder
- fileHandler.importGeneratedGlb(): parse GLB bytes, stamp userData.aiGen
provenance, place + replicate + record undo via the EXISTING addImported path
(standard object sync — no new message types)
- MeshJobsCard.svelte: progress cards (bar, cancel, dismiss); App debugStores hook
exposes meshProviders + meshJobs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…map 11 G6) - tools.js: generate_mesh LONG tool (async fire-and-forget so the chat loop is not blocked for minutes; the job runner places the mesh when ready). getAiTools() includes it only when a mesh provider is ready; buildSystemPrompt mentions it - assistant.js uses getAiTools() per turn - addObjects.buildAddChildren: "Generate 3D model" viewport-menu entry when ready - MeshGenModal.svelte: direct (no-LLM) generate dialog (provider + prompt + name, hosted-credit note); meshGenModalOpen store; mounted in Menu with MeshJobsCard Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- exports a real GLB from a box in-page, mocks ComfyUI (/prompt, /history, /view) and Meshy (task + download) via page.route, and asserts: provider ready after freshReload, ComfyUI generate -> import with aiGen provenance, Meshy generate -> import, and undo removes the last generated mesh. 7/7 PASS Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- meshGen.js: compact Node port of the ComfyUI + Meshy adapters (submit/poll/ download GLB bytes) - tools.js generate_mesh: async fire-and-forget; on completion pushes the GLB via the objectfile raw-bytes message (the agent can't run three.js). Registry stub - cli.js: mesh backend from --mesh-kind/--mesh-url/--mesh-key/--mesh-workflow(file)/ --mesh-mode or AGENT_MESH_* env; messages.objectFileMsg builder Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap #10 (AI scene assistant) + #11 (generative 3D)
Ships the AI feature line built on
feature/ai-scene-assistant. Linear history on top ofmain(22 commits, no divergence).Roadmap #10 — AI scene assistant
Track A — in-app assistant
src/lib/ai/providers.js— Grok / Gemini / custom vLLM presets via one OpenAI-compatible client (config list + active,aiReady()).ai/client.js— plain-fetch OpenAI-compatible SSE streaming, index-keyed tool_call accumulation,runChattool loop,testConnection, CORS-aware error naming.history.js—beginHistoryBatch/endHistoryBatch+aibatchcomposite kind (one undo per prompt).ai/tools.js—list_scene/create_objects/update_objects/delete_objects/group_objects/clear_sceneover the replicated mutation surface (explicit uuids = no selection steal).ai/assistant.js—runPrompt,aiMessages/aiBusystores.AiAssistant.svelte— chat window + backquote-toggled prompt pill (hidden by default), Settings AI section.ai-assistant.test.cjs(single page + broadcast-capture stub, 12/12).Track B — headless console agent (
tools/agent/, own package)@roamhq/wrtcpeerjs-in-Node,peerBridge.jsapproval-dance state machine (omits the modules handshake), REPL + MCP modes,mcpServer.jsstdio server.Roadmap #11 — generative 3D ("AI v2")
Text/image to textured mesh in-scene via a provider-adapter architecture:
ai/meshProviders.js+meshAdapters/{comfyui,meshy}+meshJobs.jsrunner (poll, cancel, 40MB guard, progress store).{{PROMPT}}/{{SEED}}workflow template →/prompt→/history→/viewGLB); Meshy adapter = hosted REST task poll.fileHandler.importGeneratedGlb(standard replication,userData.aiGenprovenance, Explorer "Generated" cache).generate_meshlong-tool, Add-menuMeshGenModal,MeshJobsCard; console-agent parity via objectfile bytes.mesh-generation.test.cjs7/7 (mock ComfyUI + Meshy). Setup guide indocs/ai/generation.md.Verification
npm run buildgreen; svelte-check held at 502/77 baseline.docs/plan,docs/ai) intentionally uncommitted / gitignored.🤖 Generated with Claude Code