Skip to content

Conversation

@valtzu
Copy link
Contributor

@valtzu valtzu commented Sep 7, 2025

Q A
Bug fix? no
New feature? yes
Docs? yes
Issues Fix #255
License MIT

Make it possible to throw custom exceptions that expose the error to the LLM when using FaultTolerantToolbox.

Example use case: an event listener for ToolCallArgumentsResolved that validates the tool call arguments and throws if any violations found. Then the LLM can auto-fix the input.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be part of the fixtures in agent component

use Symfony\AI\Agent\Toolbox\ToolFactory\ChainFactory;
use Symfony\AI\Agent\Toolbox\ToolFactory\MemoryToolFactory;
use Symfony\AI\Agent\Toolbox\ToolFactory\ReflectionToolFactory;
use Symfony\AI\Fixtures\Tool\ToolCustomException;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, same for the other fixtures?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean all fixtures in Symfony\AI\Fixtures\Tool namespace? I think they were only recently moved to root fixtures 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check where they are used? If they are only used in agent then we should move them.

cc @chr-hertel

Copy link
Contributor Author

@valtzu valtzu Sep 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

70% of Symfony\AI\Fixtures\Tool usage is in Agent component and the rest 30% is in Platform

I'd probably keep them as they are now instead of having them in several locations. Or remove all these from the root (and move to the packages), and just duplicate the ones that are needed in several components. I think it is better DX-wise if a developer can add the new fixture where there already are existing ones, instead of needing to figure out which directory you should choose (in case we had both, root + in the package).

@valtzu valtzu force-pushed the expose-custom-exceptions branch from 205131d to a06e551 Compare September 7, 2025 13:33
@OskarStark

This comment was marked as outdated.

@valtzu valtzu changed the title [Agent] Allow exposing exceptions to AI [Agent] Allow exposing tool exceptions to AI Sep 8, 2025
@OskarStark OskarStark changed the title [Agent] Allow exposing tool exceptions to AI [Agent] Allow exposing tool exceptions Sep 8, 2025
@OskarStark OskarStark changed the title [Agent] Allow exposing tool exceptions [Agent] Allow exposing tool exceptions to LLM Sep 8, 2025
@OskarStark OskarStark changed the title [Agent] Allow exposing tool exceptions to LLM [Agent] Allow exposing tool exceptions to AI Sep 8, 2025
}
}

#[AsTool('get_user_age', 'Get age by user id')]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if possible, but wouldn't it be better if you would be able to provide this info via callable in the attribute to not couple you code with the 3rd party exception?

Suggested change
#[AsTool('get_user_age', 'Get age by user id')]
#[AsTool('get_user_age', 'Get age by user id', exceptions: [EntityNotFoundException::class => static fn(THE TOOL CALL ARGUMENTS HERE)....])]

Or is this not possible? Or is it a bad idea?

Copy link
Contributor Author

@valtzu valtzu Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting idea,

but my reasoning for this feature is actually about throwing from an event listener of ToolCallArgumentsResolved – therefore I don't see this as a good place for it as you'd have to put it into every tool definition then.

Maybe this is just a bad example in the docs?


Another possibility to implement what I need could be an ability to define a custom error handler service for particular toolbox (or maybe even globally for all agents). How that could be implemented, f.e. instead of current fault_tolerant_toolbox: false/true you could set fault_tolerant_toolbox: 'app.my_abstract_fault_tolerant_toolbox' which would then use your service definition instead of builtin FaultTolerantToolbox.

I find the exception interface more convenient, however, it does have the downside of not being able to use/inject services to produce the result

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with the interface, the error handler seems to complicated for such a "normal" use case, WDYT @chr-hertel ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with that interface as well - valid use-case too me - and opt-in 👍

@OskarStark OskarStark added Agent Issues & PRs about the AI Agent component Feature New feature labels Sep 8, 2025
@OskarStark
Copy link
Contributor

Thank you @valtzu.

@OskarStark OskarStark merged commit a709f6d into symfony:main Sep 8, 2025
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Agent Issues & PRs about the AI Agent component Feature New feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Agent] Allow exposing some tool exceptions to the LLM

3 participants