get_openai_callback does not return token usage when used with openai tools agent

[rafboh]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

class OpenAItool:
    def __init__(self, profile) -> None:
        self.user_profile = profile
        prompt = ChatPromptTemplate.from_messages(
            [
                ("system", "You are a AI agent having a conversation with a human"),
                ("placeholder", "{chat_history}"),
                ("placeholder", "{agent_scratchpad}"),
                ("human", "{question}"),
            ]
        )
        llm = ChatOpenAI(
            model_name=profile.model,
            temperature=settings.OPEN_AI["OPEN_AI_TEMPERATURES"]["CHAT"],
        )
        self.memory = ConversationBufferMemory(
            memory_key="chat_history",
            return_messages=True,
            window=profile.memory_buffer,
        )
        self.chat_history = MemoryAgent()

        # tool objects
        dummy = Dummy()
        news = News()

        # searchTool = SerpAPIWrapper(serpapi_api_key=settings.SERP_API["API"])
        searchTool = TavilySearchResults(max_results=1)
        wikiTool = WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper())
        dummyTool = StructuredTool.from_function(
            dummy.run,
            description="This is dummy tool, returns empty string, disregard",
        )
        exRatesTool = StructuredTool.from_function(
            get_exchange_rate,
            description="This tool gets the exchange rate between two currencies.",
        )
        weatherReportTool = StructuredTool.from_function(
            getWeather,
            description="This tool gets weather reports and forecasts.",
        )
        dateTimeTool = StructuredTool.from_function(
            getTime,
            description="This tool gets local date and time",
        )
        newsTool = StructuredTool.from_function(
            news.get,
            description="This tool gets News",
        )

        all_tools = {
            "webSearchTool": searchTool,
            "wikiSearchTool": wikiTool,
            "exRatesTool": exRatesTool,
            "weatherReportTool": weatherReportTool,
            "dateTimeTool": dateTimeTool,
            "newsTool": newsTool,
        }
        tools_status = profile.agent_tools
        active_tools = [
            all_tools[tool_name] for tool_name, status in tools_status.items() if status
        ]
        if len(active_tools) == 0:
            active_tools = [
                dummyTool,
            ]
       

        agent = create_openai_tools_agent(llm, active_tools, prompt)
        self.agent_executor = AgentExecutor(
            agent=agent, tools=active_tools, verbose=True
        )

    def run(self, query):
        ###########LOAD CONVERSATION MEMORY#############
        retrieved_chat_history = self.chat_history.load_chat_from_db_mod(
            self.user_profile
        )

        #######################################################
        with get_openai_callback() as cb:
            answer = self.agent_executor.invoke(
                {
                    "question": query,
                    "chat_history": retrieved_chat_history,
                },
                callback=cb,
            )
            print(cb)
            print(f"Total Tokens: {cb.total_tokens}")
            print(f"Prompt Tokens: {cb.prompt_tokens}")
            print(f"Completion Tokens: {cb.completion_tokens}")
            print(f"Total Cost (USD): ${cb.total_cost}")
            print(self.user_profile.model)

            ############# SAVE CONVERSATION MEMORY ############
            new_message = ChatMessages(
                user=self.user_profile,
                sender_message=query,
                ai_message=answer["output"],
            )
            new_message.save()
            new_cost = TokenCost(
                user=self.user_profile, cost=cb.total_cost, tokens=cb.total_tokens
            )
            new_cost.save()
            ####################################################

    response = {
           "answer": answer,
            }
    return response

Error Message and Stack Trace (if applicable)

Entering new AgentExecutor chain...
05/18/2024 12:37:09 PM - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Hi! How can I help you today?

Finished chain.
Tokens Used: 0
Prompt Tokens: 0
Completion Tokens: 0
Successful Requests: 0
Total Cost (USD): $0.0
Total Tokens: 0
Prompt Tokens: 0
Completion Tokens: 0
Total Cost (USD): $0.0
gpt-4o

Description

get_openai_callback does not returns token usage when used with openai tools agent

System Info

System Information

OS: Linux
OS Version: #1 SMP Debian 5.10.216-1 (2024-05-03)
Python Version: 3.10.11 (main, May 14 2023, 09:02:31) [GCC 10.2.1 20210110]

Package Information

langchain_core: 0.2.0
langchain: 0.2.0
langchain_community: 0.2.0
langsmith: 0.1.59
langchain_experimental: 0.0.59
langchain_openai: 0.1.7
langchain_text_splitters: 0.2.0

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[vishyarjun]

I'm facing the same issue as well. I keep getting 0 as the token count when using get_openai_callback, irrespective of any agent_type, like tool_calling or openai-tools

[rafboh]

Solved ! So easy, self.agent_executor = AgentExecutor(
agent=agent, tools=active_tools, verbose=True, stream_runnable=False
)
stream_runnable must be set to False to make it working in my case

[eyurtsev]

Closing as this is a duplicate