refactor(chat): unify streaming state flow and improve chat viewport + mention UX

[AnishSarkar22]

Description

• Refactored chat streaming flow by introducing shared stream handling utilities and simplifying event processing across the new chat stack.
• Consolidated frontend chat error and interrupt handling into clearer utilities, improving retry/cancel behavior and maintainability.
• Added turn cancellation and status management in new chat routes to better handle in-flight operations.
• Updated thread reset state to include shareToken and contentType so report/thread context resets correctly.
• Reworked chat UI structure with ChatViewport/NestedScroll, then simplified layout and scroll behavior for more consistent UX.
• Improved viewport interaction with auto-scroll and top-fade behavior, plus className cleanup for stable scrolling.
• Upgraded mention handling in both composer/editor and user message rendering for more reliable parsed segment display.
• Fixed assistant message comment trigger slot and visibility handling for more predictable message actions.
• Added/updated backend unit coverage for busy mutex and stream contract behavior alongside refactors.
• Added TruncatedNameWithTooltip component in model selector.

Motivation and Context

FIX #

Screenshots

API Changes

• This PR includes API changes

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactoring
• Documentation
• Dependency/Build system
• Breaking change
• Other (specify):

Testing Performed

• Tested locally
• Manual/QA verification

Checklist

• Follows project coding standards and conventions
• Documentation updated as needed
• Dependencies updated as needed
• No lint/build errors or new warnings
• All relevant tests are passing

High-level PR Summary

This PR implements a comprehensive chat state unification system that introduces graceful turn cancellation with exponential backoff retry logic. The backend now tracks cancel requests with timestamps and attempt counts, exposes new /cancel-active-turn and /turn-status endpoints, and emits real-time turn-status SSE events (idle, busy, cancelling). The frontend refactors stream processing into shared pipelines, adds automatic retry logic with TURN_CANCELLING error handling, and reorganizes UI components for consistent viewport behavior across chat, free-chat, and public-chat contexts. The changes ensure concurrent request blocking is handled smoothly with client-side retry hints rather than hard failures.

⏱️ Estimated Review Time: 1-3 hours

💡 Review Order Suggestion

Order	File Path
1	surfsense_backend/app/agents/new_chat/middleware/busy_mutex.py
2	surfsense_backend/app/schemas/new_chat.py
3	surfsense_backend/app/routes/new_chat_routes.py
4	surfsense_backend/app/services/new_streaming_service.py
5	surfsense_backend/app/tasks/chat/stream_new_chat.py
6	surfsense_backend/tests/unit/agents/new_chat/test_busy_mutex.py
7	surfsense_backend/tests/unit/test_stream_new_chat_contract.py
8	surfsense_web/lib/chat/streaming-state.ts
9	surfsense_web/lib/chat/chat-error-classifier.ts
10	surfsense_web/lib/chat/chat-request-errors.ts
11	surfsense_web/lib/chat/stream-flush.ts
12	surfsense_web/lib/chat/stream-pipeline.ts
13	surfsense_web/lib/chat/stream-side-effects.ts
14	surfsense_web/app/dashboard/[search_space_id]/new-chat/[[...chat_id]]/page.tsx
15	surfsense_web/components/assistant-ui/chat-viewport.tsx
16	surfsense_web/components/assistant-ui/nested-scroll.tsx
17	surfsense_web/components/assistant-ui/thread.tsx
18	surfsense_web/components/assistant-ui/tool-fallback.tsx
19	surfsense_web/components/free-chat/free-thread.tsx
20	surfsense_web/components/public-chat/public-thread.tsx
21	surfsense_web/atoms/chat/current-thread.atom.ts
22	surfsense_web/components/assistant-ui/thread-scroll-to-bottom.tsx

⚠️ Inconsistent Changes Detected

File Path	Warning
surfsense_web/atoms/chat/current-thread.atom.ts	The reset function now initializes additional report panel fields (shareToken, contentType) which appears unrelated to the core chat state unification and turn cancellation logic.
surfsense_web/components/assistant-ui/thread-scroll-to-bottom.tsx	File was removed entirely, which seems to be a refactoring/cleanup rather than core turn-cancellation functionality.

[vercel]

@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ca234827-ef69-49bf-97ea-bf53820acb61

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}