[Improvement] Modular middleware stack + agent/prompt caching + subagent resilience + unit tests

[CREDO23]

Hardening pass on top of #1326. Behind MULTI_AGENT_CHAT_ENABLED.

• Modular middleware stack (shared/, main_agent/, subagent/).
• Compile-time agent graph cache + per-thread LiteLLM prompt cache.
• Subagent resilience: scoped model fallback, retry, model/tool call limits.
• Single _build_main_agent_for_thread helper across all rebuild paths (prevents factory drift on 429 recovery).
• Degrade to builtins-only on MCP/registry/connector discovery failure.

High-level PR Summary

This PR refactors the multi-agent chat middleware into a modular, layered architecture organized into shared/, main_agent/, and subagent/ folders. It introduces compile-time agent graph caching combined with per-thread LiteLLM prompt caching to improve performance. The changes enhance subagent resilience through scoped model fallback (only on provider/network errors), retry logic, and model/tool call limits. A unified _build_main_agent_for_thread helper ensures consistent agent factory behavior across initial build, preflight, and 429 recovery paths. The system now degrades gracefully to builtin-only mode when MCP, registry, or connector discovery fails, preventing transient errors from blocking user responses. The main agent factory is renamed from create_surfsense_deep_agent to create_multi_agent_chat_deep_agent for clarity.

⏱️ Estimated Review Time: 30-90 minutes

💡 Review Order Suggestion

Order	File Path
1	surfsense_backend/app/agents/multi_agent_chat/__init__.py
2	surfsense_backend/app/agents/multi_agent_chat/main_agent/__init__.py
3	surfsense_backend/app/agents/multi_agent_chat/main_agent/runtime/__init__.py
4	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/flags.py
5	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/todos.py
6	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/memory.py
7	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/anthropic_cache.py
8	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/patch_tool_calls.py
9	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/compaction.py
10	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/file_intent.py
11	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/filesystem.py
12	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/resilience/retry.py
13	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/resilience/fallback.py
14	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/resilience/model_call_limit.py
15	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/resilience/tool_call_limit.py
16	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/resilience/bundle.py
17	surfsense_backend/app/agents/new_chat/middleware/scoped_model_fallback.py
18	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/permissions/context.py
19	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/permissions/middleware.py
20	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/permissions/__init__.py
21	surfsense_backend/app/agents/multi_agent_chat/middleware/subagent/extras.py
22	surfsense_backend/app/agents/multi_agent_chat/subagents/builtins/general_purpose/agent.py
23	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/busy_mutex.py
24	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/otel.py
25	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/anonymous_doc.py
26	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/knowledge_tree.py
27	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/knowledge_priority.py
28	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/kb_persistence.py
29	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/skills.py
30	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/selector.py
31	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/context_editing.py
32	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/noop_injection.py
33	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/repair.py
34	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/doom_loop.py
35	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/action_log.py
36	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/dedup_hitl.py
37	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/plugins.py
38	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/checkpointed_subagent_middleware/task_tool.py
39	surfsense_backend/app/agents/multi_agent_chat/middleware/stack.py
40	surfsense_backend/app/agents/multi_agent_chat/middleware/__init__.py
41	surfsense_backend/app/agents/multi_agent_chat/main_agent/graph/compile_graph_sync.py
42	surfsense_backend/app/agents/multi_agent_chat/main_agent/runtime/agent_cache.py
43	surfsense_backend/app/agents/multi_agent_chat/main_agent/runtime/factory.py
44	surfsense_backend/app/tasks/chat/stream_new_chat.py
45	surfsense_backend/app/agents/new_chat/chat_deepagent.py
46	surfsense_backend/tests/unit/agents/multi_agent_chat/middleware/checkpointed_subagent_middleware/test_resume_helpers.py
47	surfsense_backend/tests/unit/agents/multi_agent_chat/middleware/checkpointed_subagent_middleware/test_pending_interrupt.py
48	surfsense_backend/tests/unit/agents/multi_agent_chat/middleware/checkpointed_subagent_middleware/test_hitl_bridge.py
49	surfsense_backend/tests/unit/agents/multi_agent_chat/subagents/shared/test_subagent_builder.py
50	surfsense_backend/tests/unit/agents/new_chat/middleware/test_scoped_model_fallback.py
51	surfsense_backend/tests/unit/middleware/test_knowledge_search.py

[vercel]

@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b552370d-2e1c-4f54-be1a-0c8262a69efd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}