High CPU usage in Kubernetes when using Teams

[mbmallinath]

High CPU usage in Kubernetes when using Teams

Hi Team,

We are seeing high CPU utilization in Kubernetes pods when using the Team feature.

• With a single agent, CPU behaves normally (increases during request and then drops).
• With Teams, CPU stays normal during execution, but after request completion it keeps increasing and does not come down.

Is there any configuration or parameter we might be missing for proper cleanup after Team execution? Or is this a known issue?

Happy to share more details if needed.

Thanks!

[omni-front]

This issue might be related to lingering sessions or resources not being cleaned up properly after Team execution. Ensure that session cleanup is properly configured in your FastAPI backend:

from fastapi import FastAPI, Request

app = FastAPI()

@app.middleware("http")
async def cleanup_middleware(request: Request, call_next):
    response = await call_next(request)
    # Perform necessary cleanup here
    # Example: Close connections, clear caches, etc.
    return response

The middleware ensures cleanup operations run after each request, helping manage resource usage.

[user1500177]

Hi @mbmallinath , did this solve ur issue ?, what version of Agno were you on

[user1500177]

Hi Agno Team and Community,

@pandego , @mbmallinath , @omni-front , @kausmeows can anyone help

I am deploying a complex multi-agent system in production on Kubernetes, and I am experiencing severe CPU spikes and sustained high resource utilization across my pods.

🛠️ Our Architecture Stack:Orchestration Layer: AgentOS using multi-agent TeamsFrontend/Streaming: Connected via the agui
interfaceTooling Infrastructure: 10 to 30 complex tools per agent (May be mre some time like 50 plus for some agents as the tools are frequesntly updated in the MCP servers and can increase inthe Futrue also , ther eare many parameters for each of these TOOLS also )

completely powered via MCP ServersConfiguration: refresh_connection=True is enabled to ensure tools are dynamically synced ad to sustain frequest onnection drops (which i took as seen in the #7073 , #7810)

So how many cpu should be allocated and mem usage we should be expecting

[omni-front]

It sounds like your system complexity might be contributing to resource strain. Consider profiling your agents to identify bottlenecks and optimize resource allocation or limit the number of active tool instances per agent.

[user1500177]

hey got one update , when i usually connect at the startup itself the scaling from 2 to 6 pods - is somtimes seen with no requests send , this was identified in those cases when the connection made to the MCP tools is unsuccessful (form what i observed) ; so whenever its wrong the MCP tool kit initialization failed error came up and the resource usage spiked up.

tested with the psutils for mesarung the cpu usage ; found in the cases when the connection to the mcp successful the same doesnt occur; so this could be a valid reason ryt ? (made a issue #8156 ) - This is my finding do you think that this could be valid ?

Now seond case is when the request are send ntemporarily the CPU spikes nad then goes down after 1 seond (i profiled in each second the CPU usage)

Now i am planning like to perform a load test - but one thing i faced when i did it locally was the rate limit hit on the open ai after first 20 or so requests itslef - so i am a little stuck on how to test like 100 or 1000 requests is that possible any help or comment ?

And when using teams is this CPU spikes expected genrally ; just asking cuz from profiling the CPU , memory ,time is taken majory by agno or internal lib package like request and response from the LLM etc

[user1500177]

@omni-front - i have only one or max two tool instace per agent