Feature suggestion: Improve customizability of handling tool input parsing error handling

[RunOrVeith]

This ticket is related to #2710 in the sense that I stumbled into that issue when trying to implement what I am proposing here, and some of the improvements go hand in hand.

I was trying to build an error correction for structured tool inputs that fixes common errors that the model does over and over automatically, without calling the model again.
Since each tool has different such errors, it needs to happen within StructuredTool.call so that I can customize it per tool.

At the moment it looks like this:

async call(
    arg: (z.output<T> extends string ? string : never) | z.input<T>,
    configArg?: Callbacks | RunnableConfig,
    /** @deprecated */
    tags?: string[]
  ): Promise<string> {
    let parsed;
    try {
      parsed = await this.schema.parseAsync(arg);
    } catch (e) {
      throw new ToolInputParsingException(
        `Received tool input did not match expected schema`,
        JSON.stringify(arg)
      );
    }
    // ... more code

My first proposal is to extract the parsing part into its own method that can be overwritten in subclasses, and only pass that to call afterward. That would improve the typing issues in #2710.

Then comes the next issue:

Inside AgentExecutor._call, we aggregate the AgentSteps that get returned to the user using returnIntermediateSteps=true.
We can not modify these steps at the moment. So if it is possible to auto-correct the input to the tool, I would like this to be reflected in the toolInput field of AgentAction.
For this we have two options: Either pass the current step into the parsing method to be modified in-place, or overwrite the the field inside AgentExecutor._call with the return value of the parsing method. This would also help to solve the typing issue described in #2710.

In case we detect that we can not use the auto-fixing in a specific error case, we need to fall back to the default error handling, which is currently done inside AgentExecutor._call as well. For this, we need to be able to throw ToolInputParsingException, which can currently not be imported. It is marked as export in tools/base, but when you try to import it, it only semi-works from the dist-folder and results in an immediate crash:

import { ToolInputParsingException } from "langchain/dist/tools/base";

throw new ToolInputParsingException("foo");

Will result in

node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './dist/tools/base' is not defined by "exports" in /home/veith/projects/automagically/ts/apps/next-automagically/node_modules/langchain/package.json
    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)
    at new NodeError (node:internal/errors:399:5)
    at exportsNotFound (node:internal/modules/esm/resolve:361:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:697:9)
    at resolveExports (node:internal/modules/cjs/loader:567:36)
    at Module._findPath (node:internal/modules/cjs/loader:636:31)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1063:27)
    at u.default._resolveFilename (/home/veith/projects/automagically/ts/node_modules/.pnpm/@esbuild-kit+cjs-loader@2.4.2/node_modules/@esbuild-kit/cjs-loader/dist/index.js:1:1519)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at <anonymous> (/home/veith/projects/automagically/ts/apps/next-automagically/debug/debug.ts:20:43)
    at Object.<anonymous> (/home/veith/projects/automagically/ts/apps/next-automagically/debug/debug.ts:165:1)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Object.F (/home/veith/projects/automagically/ts/node_modules/.pnpm/@esbuild-kit+cjs-loader@2.4.2/node_modules/@esbuild-kit/cjs-loader/dist/index.js:1:941)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:169:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

So this needs to be exposed so that it can actually be used by the user. This would also be beneficial for AgentExecutor.handleParsingErrors. At the moment, the callback you can pass in for that receives the the error, so you can do something different depending on whether it is an output parser error, or a tool input error. But you can not use instanceof ToolInputParsingException as a switch, because you can't import the error type.
You also can not do anything meaningful here, because you only receive the error, but not the input that caused the error.

So another option to implement these kind of custom error fixes would be to change the input arguments for handleParsingErrors to

const handleParsingErrors: (error: ToolInputParsingError | OutputParserError, causingData: Json | string, action?: AgentAction) => Json | string;

Please provide some feedback, I would be willing to contribute here. This should be solved together with #2710 to get the typing correct.

[dosubot]

Hi, @RunOrVeith,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. It looks like you've proposed improvements to the handling of tool input parsing errors, suggesting the extraction of the parsing part into its own method for customization in subclasses. Additionally, you've requested the ability to modify the tool input field of AgentAction and to expose the ToolInputParsingException for user use. These changes aim to address typing issues and provide more flexibility for custom error handling.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution!