You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
jamesbraza opened this issue
Jun 8, 2024
· 1 comment
Labels
Ɑ: agentRelated to agents module🤖:bugRelated to a bug, vulnerability, unexpected error with an existing feature🔌: openaiPrimarily related to OpenAI integrations
I searched the LangChain documentation with the integrated search.
I used the GitHub search to find a similar question and didn't find it.
I am sure that this is a bug in LangChain rather than my code.
The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
Example Code
fromlangchain.agentsimportAgentExecutorfromlangchain.agents.openai_functions_agent.baseimportOpenAIFunctionsAgentfromlangchain_community.callbacksimportOpenAICallbackHandlerfromlangchain_community.toolsimportSleepToolfromlangchain_core.runnablesimportRunnableConfigfromlangchain_openaiimportChatOpenAI# We should incur some OpenAI costs here from agent planningcost_callback=OpenAICallbackHandler()
tools= [SleepTool()]
agent_instance=AgentExecutor.from_agent_and_tools(
tools=tools,
agent=OpenAIFunctionsAgent.from_llm_and_tools(
ChatOpenAI(model="gpt-4", request_timeout=15.0), tools# type: ignore[call-arg]
),
return_intermediate_steps=True,
max_execution_time=10,
callbacks=[cost_callback], # "Local" callbacks
)
# NOTE: intentionally, I am not specifying the callback to invoke, as that# would make the cost_callback be considered "inheritable" (which I don't want)outputs=agent_instance.invoke(
input={"input": "Sleep a few times for 100-ms."},
# config=RunnableConfig(callbacks=[cost_callback]), # "Inheritable" callbacks
)
assertlen(outputs["intermediate_steps"]) >0, "Agent should have slept a bit"assertcost_callback.total_cost>0, "Agent planning should have been accounted for"# Fails
Error Message and Stack Trace (if applicable)
Traceback (most recent call last):
File "/Users/user/code/repo/app/agents/a.py", line 28, in
assert cost_callback.total_cost > 0, "Agent planning should have been accounted for"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Agent planning should have been accounted for
Description
LangChain has a useful concept of "inheritable" callbacks vs "local" callbacks, all managed by CallbackManger (source reference 1 and 2)
Inheritable callback: callback is automagically reused by nested invoke
Local callback: no reuse by nested invoke
Yesterday I discovered AgentExecutor does not use local callbacks for its planning step. I consider this a bug, as planning (e.g BaseSingleActionAgent.plan) is a core behavior of AgentExecutor.
The fix would be supporting AgentExecutor's local callbacks during planning
The text was updated successfully, but these errors were encountered:
dosubotbot
added
Ɑ: agent
Related to agents module
🔌: openai
Primarily related to OpenAI integrations
🤖:bug
Related to a bug, vulnerability, unexpected error with an existing feature
labels
Jun 8, 2024
fromtypingimportcastfromunittest.mockimportpatchfromlangchain.agents.openai_functions_agent.baseimportOpenAIFunctionsAgentfromlangchain_core.agentsimportAgentAction, AgentFinishfromlangchain_core.callbacksimportBaseCallbackHandler, BaseCallbackManager, Callbacksorig_plan=OpenAIFunctionsAgent.plan
...
defplan_with_injected_callbacks(
self,
intermediate_steps: list[tuple[AgentAction, str]],
callbacks: Callbacks=None,
**kwargs
) ->AgentAction|AgentFinish:
ifself==agent_instance.agent:
# Work around https://github.com/langchain-ai/langchain/issues/22703forcallbackincast(list[BaseCallbackHandler], agent_instance.callbacks):
cast(BaseCallbackManager, callbacks).add_handler(callback, inherit=False)
returnorig_plan(self, intermediate_steps, callbacks, **kwargs)
# NOTE: intentionally, I am not specifying the callback to invoke, as that# would make the cost_callback be considered "inheritable" (which I don't want)withpatch.object(OpenAIFunctionsAgent, "plan", plan_with_injected_callbacks):
outputs=agent_instance.invoke(
input={"input": "Sleep a few times for 100-ms."},
# config=RunnableConfig(callbacks=[cost_callback]), # "Inheritable" callbacks
)
Ɑ: agentRelated to agents module🤖:bugRelated to a bug, vulnerability, unexpected error with an existing feature🔌: openaiPrimarily related to OpenAI integrations
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
Traceback (most recent call last):
File "/Users/user/code/repo/app/agents/a.py", line 28, in
assert cost_callback.total_cost > 0, "Agent planning should have been accounted for"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Agent planning should have been accounted for
Description
LangChain has a useful concept of "inheritable" callbacks vs "local" callbacks, all managed by
CallbackManger
(source reference 1 and 2)invoke
invoke
Yesterday I discovered
AgentExecutor
does not use local callbacks for its planning step. I consider this a bug, as planning (e.gBaseSingleActionAgent.plan
) is a core behavior ofAgentExecutor
.The fix would be supporting
AgentExecutor
's local callbacks during planningSystem Info
langchain==0.2.3
langchain-community==0.2.4
langchain-core==0.2.5
langchain-openai==0.1.8
The text was updated successfully, but these errors were encountered: