Add Cactus (needle-rs) local tool-calling provider by sroussey · Pull Request #524 · workglow-dev/libs

sroussey · 2026-05-21T20:48:23Z

Summary

Introduces @workglow/cactus, a new AI provider that wraps the needle-rs WASM runtime and the Needle 26M model for local, browser-compatible tool-calling. This provider enables specialized tool-routing inference without requiring external API calls or large model downloads.

Key Changes

New provider package (providers/cactus/):
- CactusProvider and CactusQueuedProvider classes for worker and inline registration
- Model schema and catalog for the Needle 26M model (22 MB INT4 SafeTensors)
- Runtime management: lazy SDK loading, in-memory engine caching, asset fetching with browser Cache Storage and Node/Bun filesystem support
- Task implementations:
  - Cactus_ToolCalling — tool-routing inference with streaming support
  - Cactus_Download / Cactus_DownloadRemove — model asset management
  - Cactus_ModelSearch / Cactus_ModelInfo — model discovery and metadata
- Comprehensive test coverage (unit + integration tests with RUN_CACTUS_TESTS=1)
Core infrastructure:
- Asset fetching abstraction supporting both browser (Cache Storage API) and Node/Bun (filesystem with atomic writes)
- Engine lifecycle management with deduplication of concurrent loads
- Session management (stateless, no-op for needle-rs)
- Resource eviction (browser cache deletion, Node directory cleanup)
Capability system:
- Declares tool-use, model.download, model.download-remove, model.search, model.info capabilities
- Constant capability inference (all Cactus models support the same set)
Integration:
- Registered in test harness (scripts/test.ts)
- Added needle-rs dependency to root package.json
- Updated AI task tests to verify capability requirements

Implementation Details

Dual-environment support: Browser uses Cache Storage for model persistence; Node/Bun uses configurable filesystem directory (default: ~/.cache/cactus-models)
Streaming: Supports run_stream callback for real-time token emission if available in needle-rs
Tool call parsing: Flexible JSON parsing handles both array and single-object formats, with fallback to empty array on parse failure
Atomic writes: Node/Bun uses .tmp suffix pattern to prevent partial downloads
In-flight deduplication: Concurrent requests for the same model share a single load promise
Resource cleanup: removeCachedAssets() disposes engines, clears config cache, and removes persisted assets

All source files include Apache 2.0 license headers with 2026 copyright year per project conventions.

https://claude.ai/code/session_015FARxMpsffnqxqXg59MHX8

ToolCallingTask.requires and StructuredGenerationTask.requires no longer include text.generation. Specialty tool/json-only models can now advertise honest capabilities. Existing chat-LLM providers continue to match via the strict-superset dispatch rule.

… inference

…ctus/src/__tests__

…e cache, eviction

…on eviction, atomic disk writes

…ove, ToolCalling, JobRunFns

…ESTS

pkg-pr-new · 2026-05-21T20:50:22Z

Open in StackBlitz

@workglow/cli

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/cli@524

@workglow/ai

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/ai@524

@workglow/browser-control

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/browser-control@524

@workglow/indexeddb

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/indexeddb@524

@workglow/javascript

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/javascript@524

@workglow/job-queue

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/job-queue@524

@workglow/knowledge-base

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/knowledge-base@524

@workglow/mcp

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/mcp@524

@workglow/storage

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/storage@524

@workglow/task-graph

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/task-graph@524

@workglow/tasks

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/tasks@524

@workglow/util

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/util@524

workglow

npm i https://pkg.pr.new/workglow-dev/libs/workglow@524

@workglow/anthropic

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/anthropic@524

@workglow/bun-webview

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/bun-webview@524

@workglow/cactus

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/cactus@524

@workglow/chrome-ai

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/chrome-ai@524

@workglow/electron

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/electron@524

@workglow/google-gemini

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/google-gemini@524

@workglow/huggingface-inference

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/huggingface-inference@524

@workglow/huggingface-transformers

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/huggingface-transformers@524

@workglow/node-llama-cpp

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/node-llama-cpp@524

@workglow/ollama

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/ollama@524

@workglow/openai

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/openai@524

@workglow/playwright

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/playwright@524

@workglow/postgres

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/postgres@524

@workglow/sqlite

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/sqlite@524

@workglow/supabase

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/supabase@524

@workglow/tf-mediapipe

npm i https://pkg.pr.new/workglow-dev/libs/@workglow/tf-mediapipe@524

commit: 90e3cda

github-actions · 2026-05-21T20:52:22Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	62.24%	23246 / 37344
🔵	Statements	62.11%	24063 / 38739
🔵	Functions	63.21%	4426 / 7002
🔵	Branches	50.7%	11285 / 22254

File Coverage

No changed files found.

Generated in workflow #2362 for commit 90e3cda by the Vitest Coverage Report Action

Copilot

Pull request overview

Introduces a new @workglow/cactus local AI provider that wraps the needle-rs WASM runtime to enable lightweight, browser-compatible tool-calling, and relaxes core task capability requirements so tool-only / json-only models can run relevant tasks.

Changes:

Added providers/cactus package with model catalog/schema, runtime (engine + asset caching), provider registrations, and tests.
Relaxed ToolCallingTask.requires to ["tool-use"] and StructuredGenerationTask.requires to ["json-mode"], plus updated tests and capability descriptions.
Integrated cactus tests into the repo test runner and added needle-rs dependency/lockfile entries.

Reviewed changes

Copilot reviewed 37 out of 38 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
scripts/test.ts	Adds a `cactus` test section pointing at the provider’s test folder.
providers/cactus/tsconfig.json	New TS project config + references for the cactus provider package.
providers/cactus/src/ai/runtime.ts	Runtime entry re-exports for worker/inline registration + runtime helpers.
providers/cactus/src/ai/registerCactusWorker.ts	Worker registration helper for the cactus provider.
providers/cactus/src/ai/registerCactusInline.ts	Inline (main-thread) registration helper for cactus.
providers/cactus/src/ai/registerCactus.ts	Worker-backed registration helper for cactus.
providers/cactus/src/ai/index.ts	Public exports + `_testOnly` re-exports for `@workglow/test`.
providers/cactus/src/ai/common/Cactus_ToolCalling.ts	Implements cactus tool-calling run-fn (streaming + non-streaming).
providers/cactus/src/ai/common/Cactus_Runtime.ts	Lazy SDK loading, asset fetch/cache, engine caching, eviction/disposal.
providers/cactus/src/ai/common/Cactus_ModelSearch.ts	Model search task implementation over a static catalog.
providers/cactus/src/ai/common/Cactus_ModelSchema.ts	Model config/record schemas for cactus provider config.
providers/cactus/src/ai/common/Cactus_ModelInfo.ts	Model info implementation (cached/loaded reporting).
providers/cactus/src/ai/common/Cactus_ModelCatalog.ts	Static catalog entry + HF asset URL builder.
providers/cactus/src/ai/common/Cactus_JobRunFns.ts	Registers cactus run-fns and re-exports runtime internals for tests.
providers/cactus/src/ai/common/Cactus_DownloadRemove.ts	Implements model download-remove (cache eviction).
providers/cactus/src/ai/common/Cactus_Download.ts	Implements model download (asset fetch + progress phases).
providers/cactus/src/ai/common/Cactus_Constants.ts	Provider/model constants (ids, cache name, default cache dir).
providers/cactus/src/ai/common/Cactus_CapabilitySets.ts	Declares cactus capability sets for run-fn registrations.
providers/cactus/src/ai/common/Cactus_Capabilities.ts	Capability inference + worker run-fn spec export.
providers/cactus/src/ai/CactusQueuedProvider.ts	Main-thread queued provider shell.
providers/cactus/src/ai/CactusProvider.ts	Worker-side provider implementation.
providers/cactus/src/ai.ts	`./ai` entrypoint shim.
providers/cactus/src/ai-runtime.ts	`./ai-runtime` entrypoint shim.
providers/cactus/src/tests/*	Unit + integration tests for cactus capabilities, schema, model tasks, download/toolcalling.
providers/cactus/README.md	Provider README describing capabilities and entrypoints.
providers/cactus/package.json	New published package config + exports.
packages/ai/src/task/ToolCallingTask.ts	Relaxes required capabilities to only `tool-use`.
packages/ai/src/task/StructuredGenerationTask.ts	Relaxes required capabilities to only `json-mode`.
packages/ai/src/task/index.test.ts	Adds assertions for the new `requires` semantics.
packages/ai/src/capability/Capabilities.ts	Updates capability descriptions for `tool-use` and `json-mode`.
package.json	Adds `needle-rs` dependency.
bun.lock	Adds provider workspace entry + `needle-rs` catalog/version updates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+  if (validToolCalls.length > 0) {
+    emit({ type: "object-delta", port: "toolCalls", objectDelta: [...validToolCalls] });
+  }
+  emit({ type: "finish", data: {} as ToolCallingTaskOutput });
+};


+  const is_loaded = isModelLoaded(model_id);
+  const is_cached = is_loaded || cactusConfigJson.has(model_id);
+


+/** Node/Bun on-disk cache root. */
+export const CACTUS_DEFAULT_MODELS_DIR = "./.cache/cactus";


+async function fetchAssetBytesNode(
+  url: string,
+  models_dir: string,
+  model_id: string,
+  filename: string
+): Promise<Uint8Array> {
+  const fs = await import("node:fs/promises");
+  const path = await import("node:path");


+  "exports": {
+    "./ai": {
+      "types": "./dist/ai.d.ts",
+      "import": "./dist/ai.js"
+    },
+    "./ai-runtime": {
+      "types": "./dist/ai-runtime.d.ts",
+      "import": "./dist/ai-runtime.js"
+    }
+  },


…mment, fix entry-point doc

…e dir, browser-safe runtime split, browser export condition Agent-Logs-Url: https://github.com/workglow-dev/libs/sessions/6503bd73-a51a-4cfe-8ead-df9e8985123e Co-authored-by: sroussey <127349+sroussey@users.noreply.github.com>

…ges/test/src/test/ai-provider-cactus to match sibling provider convention Migrates the 7 cactus test files to the cross-package test location used by every other provider, importing from `@workglow/cactus/ai` and `@workglow/cactus/ai-runtime` instead of relative source paths. The Capabilities test is renamed to CactusProvider.test.ts and now exercises `new CactusQueuedProvider().inferCapabilities(model)` (the public provider surface) rather than the internal helper. `_testOnly` now also re-exports `cactusEngines` and `cactusConfigJson`. The `./ai` and `./ai-runtime` entry points are bundled separately, so their copies of `Cactus_Runtime.ts` are distinct module instances. Tests that seed runtime state for run-fns bundled into `./ai` must read that same instance, which is only reachable through `_testOnly`. `scripts/test.ts` renames the `cactus` section to `provider-cactus` (matching `provider-hft`, `provider-nodellama`, `provider-api`) and points it at the new directory.

claude added 10 commits May 21, 2026 19:16

feat(cactus): scaffold @workglow/cactus provider package

a10ae0b

feat(cactus): constants, catalog, schema, capability sets, capability…

fe690a9

… inference

chore(scripts): register cactus test section pointing at providers/ca…

6aac515

…ctus/src/__tests__

feat(cactus): runtime — SDK loader, asset fetch (browser+Node), engin…

bd2e0f1

…e cache, eviction

fix(cactus/runtime): dedup concurrent engine loads, free WASM memory …

c4c468a

…on eviction, atomic disk writes

feat(cactus): run-fns — ModelSearch, ModelInfo, Download, DownloadRem…

e85740d

…ove, ToolCalling, JobRunFns

test(cactus): isolate DownloadRemove test with afterEach cleanup

7c9e492

feat(cactus): provider classes, registration entry points, barrels

b5d94cf

test(cactus): add real-download integration tests behind RUN_CACTUS_T…

6c8085d

…ESTS

sroussey self-assigned this May 21, 2026

sroussey requested a review from Copilot May 21, 2026 20:48

Copilot started reviewing on behalf of sroussey May 21, 2026 20:49 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

docs(cactus): mention in root README, add runtime-state separation co…

5d0f72e

…mment, fix entry-point doc

Copilot started work on behalf of sroussey May 21, 2026 21:56 View session

Copilot finished work on behalf of sroussey May 21, 2026 22:05

sroussey merged commit bf68ff2 into main May 21, 2026
16 checks passed

sroussey deleted the claude/brainstorming-p2V9v branch May 21, 2026 23:16

This was referenced May 22, 2026

fix(cactus): SHA-256 integrity verification + cross-bundle runtime singleton #529

Closed

feat(cactus): SHA-256 integrity verification for fetched model assets #530

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Cactus (needle-rs) local tool-calling provider#524

Add Cactus (needle-rs) local tool-calling provider#524
sroussey merged 13 commits into
mainfrom
claude/brainstorming-p2V9v

sroussey commented May 21, 2026

Uh oh!

pkg-pr-new Bot commented May 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		const is_loaded = isModelLoaded(model_id);
		const is_cached = is_loaded \|\| cactusConfigJson.has(model_id);

		/** Node/Bun on-disk cache root. */
		export const CACTUS_DEFAULT_MODELS_DIR = "./.cache/cactus";

Conversation

sroussey commented May 21, 2026

Summary

Key Changes

Implementation Details

Uh oh!

pkg-pr-new Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pkg-pr-new Bot commented May 21, 2026 •

edited

Loading

github-actions Bot commented May 21, 2026 •

edited

Loading