feat(ai): Tool call loop, built-in data tools, agent runtime & chat API by Copilot · Pull Request #1016 · objectstack-ai/framework

Copilot · 2026-03-31T08:00:45Z

Implements the core infrastructure for an Airtable Copilot-style data conversation chatbot — users ask natural language questions against business data, the AI resolves them via tool calls to the data engine.

Contract changes (`@objectstack/spec`)

AIResult.toolCalls?: AIToolCall[] — adapters can now return tool call requests from the LLM
IAIService.chatWithTools?() + ChatWithToolsOptions — optional contract method for tool-augmented chat with maxIterations safety
Export Agent type from @objectstack/spec index

Tool call loop (`AIService.chatWithTools`)

Multi-round LLM ↔ tool execution loop:

Auto-injects ToolRegistry definitions into LLM options
Executes returned toolCalls via registry, appends results as role: 'tool' messages
Loops until model returns final text or maxIterations (default 10) exhausted
Forces a tool-free final call if loop limit hit
Strips maxIterations from options before forwarding to the adapter to prevent unknown-key errors in provider SDKs

const result = await aiService.chatWithTools(
  [{ role: 'user', content: 'How many active accounts?' }],
  { maxIterations: 5 },
);

Built-in data tools

Five platform-level tools registered via registerDataTools(registry, { dataEngine, metadataService }):

Tool	Purpose
`list_objects`	List all available tables
`describe_object`	Schema introspection (fields, types, labels)
`query_records`	Filter/sort/paginate records (limit capped at 200)
`get_record`	Single record by ID
`aggregate_data`	count/sum/avg/min/max with groupBy

Data tool hardening:

query_records validates limit/offset as finite positive integers, clamping negatives/NaN to defaults
aggregate_data validates aggregation function names at runtime against the allowed set (count, sum, avg, min, max, count_distinct), returning a structured error for invalid values

Agent runtime & chat route

AgentRuntime — loads agent metadata from IMetadataService with AgentSchema.safeParse() validation (returns undefined for malformed metadata), builds system prompt from instructions + UI context (objectName, recordId, viewName), resolves agent tool references against the registry
POST /api/v1/ai/agents/:agentName/chat — agent lookup, active-check, context injection, delegates to chatWithTools
- Restricts inbound message roles to user and assistant only — system and tool messages are rejected to prevent client-side guardrail overrides
- Whitelists safe caller option overrides (temperature, maxTokens, stop); silently ignores sensitive keys (tools, toolChoice, model) to prevent tool-definition injection
DATA_CHAT_AGENT — built-in data_chat agent spec with guardrails, planning config, and all 5 data tool references

Plugin wiring

AIServicePlugin.start() auto-registers data tools and the data_chat agent when IDataEngine + IMetadataService are present in the kernel, before firing ai:ready. Agent auto-registration is non-destructive — checks exists() first to preserve user customizations.

Tests

51 new test cases covering the tool call loop (sequential, parallel, max iterations, error propagation, maxIterations stripping), all 5 data tool handlers (including limit/offset validation and aggregation function validation), agent runtime (loading, validation of malformed metadata, prompt building, tool resolution), agent route (validation, 404/403, role restriction, options whitelisting, happy path), and agent spec validation. All 130 service-ai tests and 6747 spec tests pass.

vercel · 2026-03-31T08:00:52Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
objectstack-play	Ready	Preview, Comment	Mar 31, 2026 8:45am
spec	Ready	Preview, Comment	Mar 31, 2026 8:45am

…ent chat route - Add toolCalls to AIResult and chatWithTools to IAIService contract - Implement chatWithTools() on AIService with multi-round tool calling loop - Define 5 built-in data tools: list_objects, describe_object, query_records, get_record, aggregate_data - Create AgentRuntime for agent metadata loading, system prompt injection, model/tools mapping - Add agent chat API route: POST /api/v1/ai/agents/:agentName/chat - Define data_chat agent spec with role, instructions, guardrails - Update AIServicePlugin to auto-register data tools and agent routes via ai:ready hook - Export all new modules from service-ai index Agent-Logs-Url: https://github.com/objectstack-ai/spec/sessions/10c08b4e-c31c-42cd-a9d4-27917289db5b Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

- Add comprehensive test suite for chatWithTools loop, data tools, agent runtime, agent routes - Export Agent type from @objectstack/spec index for use in service-ai - Update CHANGELOG.md with data chatbot feature entry Agent-Logs-Url: https://github.com/objectstack-ai/spec/sessions/10c08b4e-c31c-42cd-a9d4-27917289db5b Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

…ny types, enhance JSDoc Agent-Logs-Url: https://github.com/objectstack-ai/spec/sessions/10c08b4e-c31c-42cd-a9d4-27917289db5b Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot

Pull request overview

This PR adds end-to-end “tool-augmented chat” infrastructure to support an Airtable Copilot-style data chatbot: contract updates in @objectstack/spec, a tool-call execution loop in AIService, built-in data tools, an agent runtime, and a new agent chat API route (with tests and changelog entry).

Changes:

Extends the AI contracts to support tool calling (AIResult.toolCalls, IAIService.chatWithTools, ChatWithToolsOptions) and re-exports the Agent type.
Implements a multi-iteration tool-call loop (AIService.chatWithTools) plus registers 5 built-in data tools (registerDataTools) and a built-in data_chat agent.
Adds AgentRuntime and POST /api/v1/ai/agents/:agentName/chat route wiring, with comprehensive test coverage and changelog update.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
packages/spec/src/index.ts	Re-exports `Agent` type for downstream consumers.
packages/spec/src/contracts/ai-service.ts	Adds tool-call fields + `chatWithTools` contract/options.
packages/services/service-ai/src/tools/index.ts	Exposes data-tool registration + types from the tools module.
packages/services/service-ai/src/tools/data-tools.ts	Introduces 5 built-in data tools and registers them into `ToolRegistry`.
packages/services/service-ai/src/routes/agent-routes.ts	Adds an agent chat HTTP route that loads agents, builds system messages, and calls `chatWithTools`.
packages/services/service-ai/src/plugin.ts	Auto-registers data tools + `data_chat` agent and mounts agent routes when services are available.
packages/services/service-ai/src/index.ts	Re-exports new runtime/tools/agent/route APIs for library consumers.
packages/services/service-ai/src/ai-service.ts	Implements the tool-call loop (`chatWithTools`) on the service.
packages/services/service-ai/src/agents/index.ts	Adds agent barrel export.
packages/services/service-ai/src/agents/data-chat-agent.ts	Defines the built-in `DATA_CHAT_AGENT` spec.
packages/services/service-ai/src/agent-runtime.ts	Adds runtime to load agents, build system prompts, and resolve tool definitions.
packages/services/service-ai/src/tests/chatbot-features.test.ts	Adds tests for loop behavior, data tools, agent runtime, and agent routes.
CHANGELOG.md	Documents the feature set in Unreleased notes.

Copilot · 2026-03-31T08:29:59Z

+    const maxIterations = options?.maxIterations ?? AIService.DEFAULT_MAX_ITERATIONS;
+    const registeredTools = this.toolRegistry.getAll();
+
+    // Merge registered tools with any explicitly provided tools
+    const mergedTools = [
+      ...registeredTools,
+      ...(options?.tools ?? []),
+    ];
+
+    // Build the options that will be sent to every LLM call in the loop
+    const chatOptions: AIRequestOptions = {
+      ...options,
+      tools: mergedTools.length > 0 ? mergedTools : undefined,
+      toolChoice: mergedTools.length > 0 ? (options?.toolChoice ?? 'auto') : undefined,
+    };


chatWithTools() builds chatOptions via { ...options, ... }, which includes maxIterations and will pass it down to this.adapter.chat(...). Adapters that forward options to a provider SDK (OpenAI/Anthropic/etc.) may reject unknown keys or serialize them unexpectedly. Strip maxIterations before building AIRequestOptions (e.g., destructure { maxIterations, ...requestOptions }).

Copilot · 2026-03-31T08:30:00Z

+    const safeLimit = Math.min(limit ?? DEFAULT_QUERY_LIMIT, MAX_QUERY_LIMIT);
+
+    const records = await ctx.dataEngine.find(objectName, {
+      where,
+      fields,
+      orderBy,
+      limit: safeLimit,
+      offset,
+    });


safeLimit is computed with Math.min(limit ?? DEFAULT_QUERY_LIMIT, MAX_QUERY_LIMIT), but this will produce a negative limit (or NaN) if the caller supplies a negative/non-finite limit. That can propagate to dataEngine.find(...) and lead to undefined behavior or driver errors. Validate limit/offset as finite integers and clamp to a minimum of 1 (and 0 for offset) before calling the data engine.

Copilot · 2026-03-31T08:30:00Z

+/** Valid message roles accepted by the agent routes. */
+const VALID_ROLES = new Set<string>(['system', 'user', 'assistant', 'tool']);
+
+function validateMessage(raw: unknown): string | null {
+  if (typeof raw !== 'object' || raw === null) {
+    return 'each message must be an object';
+  }
+  const msg = raw as Record<string, unknown>;
+  if (typeof msg.role !== 'string' || !VALID_ROLES.has(msg.role)) {
+    return `message.role must be one of ${[...VALID_ROLES].map(r => `"${r}"`).join(', ')}`;
+  }
+  if (typeof msg.content !== 'string') {
+    return 'message.content must be a string';
+  }
+  return null;


This agent chat endpoint accepts caller-supplied system and tool role messages, and then prepends the agent's own system message (fullMessages = [...systemMessages, ...rawMessages]). Because caller system messages come after the agent system prompt, a client can override the agent instructions/guardrails. Consider restricting inbound roles for this route to user (and optionally assistant for transcript replay) and rejecting/stripping system + tool messages from the request body.

Copilot · 2026-03-31T08:30:00Z

+        const {
+          messages: rawMessages,
+          context: chatContext,
+          options: extraOptions,
+        } = (req.body ?? {}) as {
+          messages?: unknown[];
+          context?: AgentChatContext;
+          options?: Record<string, unknown>;
+        };
+
+        if (!Array.isArray(rawMessages) || rawMessages.length === 0) {
+          return { status: 400, body: { error: 'messages array is required' } };
+        }
+
+        for (const msg of rawMessages) {
+          const err = validateMessage(msg);
+          if (err) return { status: 400, body: { error: err } };
+        }
+
+        // Load agent definition
+        const agent = await agentRuntime.loadAgent(agentName);
+        if (!agent) {
+          return { status: 404, body: { error: `Agent "${agentName}" not found` } };
+        }
+        if (!agent.active) {
+          return { status: 403, body: { error: `Agent "${agentName}" is not active` } };
+        }
+
+        try {
+          // Build system messages from agent instructions + UI context
+          const systemMessages = agentRuntime.buildSystemMessages(agent, chatContext);
+
+          // Resolve agent model/tools → request options
+          const agentOptions = agentRuntime.buildRequestOptions(
+            agent,
+            aiService.toolRegistry.getAll(),
+          );
+
+          // Merge agent options with any caller overrides
+          const mergedOptions = { ...agentOptions, ...extraOptions };
+
+          // Prepend system messages then user conversation
+          const fullMessages: AIMessage[] = [
+            ...systemMessages,
+            ...(rawMessages as AIMessage[]),
+          ];
+
+          // Use chatWithTools for automatic tool resolution
+          const result = await aiService.chatWithTools(fullMessages, {
+            ...mergedOptions,
+            maxIterations: agent.planning?.maxIterations,
+          });


extraOptions is taken as an unvalidated Record<string, unknown> and shallow-merged into agentOptions, then passed into chatWithTools. This allows clients to override sensitive request settings (e.g. tools, toolChoice, model, very large maxTokens) and can cause tool-definition injection / DoS loops (LLM calling tools that aren't actually registered). Prefer whitelisting only safe AIRequestOptions keys for caller overrides and explicitly ignoring tools/toolChoice (and/or enforcing server-side caps).

Copilot · 2026-03-31T08:30:00Z

+  async loadAgent(agentName: string): Promise<Agent | undefined> {
+    const raw = await this.metadataService.get('agent', agentName);
+    return raw as Agent | undefined;
+  }


AgentRuntime.loadAgent() currently returns raw as Agent without validating it, even though the class docstring says it loads & validates agent metadata. If metadata is malformed (missing instructions, wrong types), buildSystemMessages() and buildRequestOptions() can throw at runtime. Consider validating with defineAgent(...) / AgentSchema.parse(...) (or at least checking required fields) and returning undefined / throwing a clear error when validation fails.

Copilot · 2026-03-31T08:30:01Z

+        // Register the built-in data_chat agent
+        await metadataService.register('agent', DATA_CHAT_AGENT.name, DATA_CHAT_AGENT);
+        ctx.logger.info('[AI] data_chat agent registered');


AIServicePlugin.start() unconditionally registers DATA_CHAT_AGENT on every startup via metadataService.register('agent', ...). The in-memory metadata manager implementation overwrites existing entries, so this can clobber a user-customized data_chat agent definition. Consider checking exists('agent', DATA_CHAT_AGENT.name) first, or making auto-registration configurable / non-destructive (e.g. register only if absent).

Suggested change

// Register the built-in data_chat agent

await metadataService.register('agent', DATA_CHAT_AGENT.name, DATA_CHAT_AGENT);

ctx.logger.info('[AI] data_chat agent registered');

// Register the built-in data_chat agent only if it does not already exist

const agentExists =

typeof metadataService.exists === 'function'

? await metadataService.exists('agent', DATA_CHAT_AGENT.name)

: false;

if (!agentExists) {

await metadataService.register('agent', DATA_CHAT_AGENT.name, DATA_CHAT_AGENT);

ctx.logger.info('[AI] data_chat agent registered');

} else {

ctx.logger.debug('[AI] data_chat agent already exists, skipping auto-registration');

}

Copilot · 2026-03-31T08:30:01Z

+    const result = await ctx.dataEngine.aggregate(objectName, {
+      where,
+      groupBy,
+      aggregations: aggregations.map(a => ({
+        function: a.function as AggFn,
+        field: a.field,
+        alias: a.alias,
+      })),
+    });


aggregate_data maps a.function with a type cast (a.function as AggFn) but does not validate it at runtime. If the LLM (or a caller) provides an unexpected function string, this will be passed to dataEngine.aggregate(...) and may trigger driver errors. Prefer validating a.function against the allowed set (and returning a structured tool error) instead of casting.

hotlong · 2026-03-31T08:31:53Z

@copilot fix all

1. Strip maxIterations from chatOptions before passing to adapter (prevents unknown-key errors) 2. Validate limit/offset in query_records (clamp negatives, reject NaN, floor to integers) 3. Restrict agent route to user/assistant roles only (reject system/tool to prevent guardrail overrides) 4. Whitelist safe caller overrides in agent route (only temperature/maxTokens/stop; block tools/toolChoice/model) 5. Validate agent with AgentSchema.safeParse() in loadAgent() (returns undefined for malformed metadata) 6. Only register data_chat agent if not already present (preserves user customizations) 7. Validate aggregation function at runtime against allowed set (returns structured error for invalid functions) 9 new tests covering all fixes. 130 total tests pass. Agent-Logs-Url: https://github.com/objectstack-ai/spec/sessions/ad6459f1-27d7-4eda-b0d3-00887c47d5de Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot · 2026-03-31T08:41:24Z

@copilot fix all

All 7 review comments addressed in e68ef7d:

Strip maxIterations — destructured out before building chatOptions, never forwarded to adapter
Validate limit/offset — query_records now clamps negatives/NaN to defaults, floors to integers
Restrict agent roles — only user/assistant accepted; system/tool rejected with 400
Whitelist caller overrides — only temperature/maxTokens/stop allowed; tools/toolChoice/model blocked
Validate agent metadata — loadAgent() now uses AgentSchema.safeParse(), returns undefined on failure
Non-destructive agent registration — checks exists() before registering data_chat agent
Validate aggregation functions — runtime check against allowed set, returns structured error for invalid functions

9 new tests added for these fixes. 130 total tests pass, build clean, 0 CodeQL alerts.

Initial plan

1838110

Copilot AI assigned Copilot and hotlong Mar 31, 2026

Copilot started work on behalf of hotlong March 31, 2026 08:00 View session

Copilot AI linked an issue Mar 31, 2026 that may be closed by this pull request

实现类似 Airtable 的数据对话 Chatbot 全栈开发任务清单 #1015

Closed

vercel Bot deployed to Preview – spec March 31, 2026 08:03 View deployment

vercel Bot deployed to Preview – objectstack-play March 31, 2026 08:07 View deployment

vercel Bot deployed to Preview – objectstack-play March 31, 2026 08:13 View deployment

vercel Bot deployed to Preview – spec March 31, 2026 08:15 View deployment

Copilot AI and others added 2 commits March 31, 2026 08:17

fix: address code review feedback — fix maxIterations logic, remove a…

23b5335

…ny types, enhance JSDoc Agent-Logs-Url: https://github.com/objectstack-ai/spec/sessions/10c08b4e-c31c-42cd-a9d4-27917289db5b Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

vercel Bot deployed to Preview – objectstack-play March 31, 2026 08:21 View deployment

Copilot AI changed the title ~~[WIP] Implement chatbot for data dialogue similar to Airtable~~ feat(ai): Tool call loop, built-in data tools, agent runtime & chat API Mar 31, 2026

vercel Bot deployed to Preview – spec March 31, 2026 08:23 View deployment

Copilot finished work on behalf of hotlong March 31, 2026 08:23

Copilot AI requested a review from hotlong March 31, 2026 08:23

hotlong marked this pull request as ready for review March 31, 2026 08:24

Copilot AI review requested due to automatic review settings March 31, 2026 08:24

Copilot started reviewing on behalf of hotlong March 31, 2026 08:25 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Copilot started work on behalf of hotlong March 31, 2026 08:32 View session

Copilot AI changed the title ~~feat(ai): Tool call loop, built-in data tools, agent runtime & chat API~~ feat(ai): Tool call loop, built-in data tools, agent runtime & chat API Mar 31, 2026

Copilot finished work on behalf of hotlong March 31, 2026 08:42

vercel Bot deployed to Preview – spec March 31, 2026 08:45 View deployment

hotlong merged commit dc1a20c into main Mar 31, 2026
2 of 3 checks passed

vercel Bot deployed to Preview – objectstack-play March 31, 2026 08:45 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ai): Tool call loop, built-in data tools, agent runtime & chat API#1016

feat(ai): Tool call loop, built-in data tools, agent runtime & chat API#1016
hotlong merged 5 commits intomainfrom
copilot/implement-chatbot-data-dialog

Copilot AI commented Mar 31, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Mar 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

Copilot AI Mar 31, 2026

Uh oh!

hotlong commented Mar 31, 2026

Uh oh!

Copilot AI commented Mar 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-        // Register the built-in data_chat agent
-        await metadataService.register('agent', DATA_CHAT_AGENT.name, DATA_CHAT_AGENT);
-        ctx.logger.info('[AI] data_chat agent registered');
+        // Register the built-in data_chat agent only if it does not already exist
+        const agentExists =
+          typeof metadataService.exists === 'function'
+            ? await metadataService.exists('agent', DATA_CHAT_AGENT.name)
+            : false;
+        if (!agentExists) {
+          await metadataService.register('agent', DATA_CHAT_AGENT.name, DATA_CHAT_AGENT);
+          ctx.logger.info('[AI] data_chat agent registered');
+        } else {
+          ctx.logger.debug('[AI] data_chat agent already exists, skipping auto-registration');
+        }

Conversation

Copilot AI commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contract changes (@objectstack/spec)

Tool call loop (AIService.chatWithTools)

Built-in data tools

Agent runtime & chat route

Plugin wiring

Tests

Uh oh!

vercel Bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

hotlong commented Mar 31, 2026

Uh oh!

Copilot AI commented Mar 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 31, 2026 •

edited

Loading

Contract changes (`@objectstack/spec`)

Tool call loop (`AIService.chatWithTools`)

vercel Bot commented Mar 31, 2026 •

edited

Loading