feat: OpenTelemetry integration

[AnishSarkar22]

Description

• Integration of OpenTelemetry with Grafana setup.
• Some UI fixes.

Motivation and Context

FIX #

Screenshots

API Changes

• This PR includes API changes

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactoring
• Documentation
• Dependency/Build system
• Breaking change
• Other (specify):

Testing Performed

• Tested locally
• Manual/QA verification

Checklist

• Follows project coding standards and conventions
• Documentation updated as needed
• Dependencies updated as needed
• No lint/build errors or new warnings
• All relevant tests are passing

High-level PR Summary

This PR integrates OpenTelemetry distributed tracing and metrics into the SurfSense backend to enable production observability. It adds comprehensive instrumentation across the entire application stack, including traces for LLM calls, tool invocations, database queries, ETL pipelines, knowledge-base searches, and chat requests, while also collecting custom metrics for token usage, request durations, error rates, and system resource utilization. The integration includes auto-instrumentation for FastAPI, SQLAlchemy, Redis, HTTPX, Celery, and Psycopg, with a new Grafana LGTM stack container for local development. A few frontend UI fixes are also included for context menus and button styling.

⏱️ Estimated Review Time: 1-3 hours

💡 Review Order Suggestion

Order	File Path
1	surfsense_backend/pyproject.toml
2	surfsense_backend/.env.example
3	docker/docker-compose.dev.yml
4	surfsense_backend/app/observability/__init__.py
5	surfsense_backend/app/observability/bootstrap.py
6	surfsense_backend/app/observability/metrics.py
7	surfsense_backend/app/observability/otel.py
8	surfsense_backend/main.py
9	surfsense_backend/app/app.py
10	surfsense_backend/app/celery_app.py
11	surfsense_backend/app/tasks/celery_tasks/__init__.py
12	surfsense_backend/app/agents/new_chat/middleware/otel_span.py
13	surfsense_backend/app/utils/perf.py
14	surfsense_backend/app/retriever/chunks_hybrid_search.py
15	surfsense_backend/app/retriever/documents_hybrid_search.py
16	surfsense_backend/app/indexing_pipeline/indexing_pipeline_service.py
17	surfsense_backend/app/etl_pipeline/etl_pipeline_service.py
18	surfsense_backend/app/tasks/chat/stream_new_chat.py
19	surfsense_backend/app/agents/multi_agent_chat/middleware/main_agent/checkpointed_subagent_middleware/task_tool.py
20	surfsense_backend/app/agents/multi_agent_chat/middleware/shared/permissions/ask/request.py
21	surfsense_backend/app/agents/new_chat/middleware/compaction.py
22	surfsense_backend/app/agents/new_chat/middleware/doom_loop.py
23	surfsense_backend/app/agents/new_chat/middleware/permission.py
24	surfsense_backend/app/routes/search_source_connectors_routes.py
25	surfsense_backend/app/tasks/celery_tasks/connector_tasks.py
26	surfsense_backend/app/tasks/celery_tasks/document_tasks.py
27	surfsense_backend/tests/unit/observability/test_otel.py
28	surfsense_backend/tests/unit/observability/test_retriever_otel.py
29	surfsense_backend/tests/unit/agents/new_chat/test_otel_span.py
30	surfsense_web/app/(home)/login/GoogleLoginButton.tsx
31	surfsense_web/components/assistant-ui/connector-popup.tsx
32	surfsense_web/components/assistant-ui/document-upload-popup.tsx
33	surfsense_web/components/layout/ui/icon-rail/SearchSpaceAvatar.tsx
34	surfsense_web/components/layout/ui/tabs/DocumentTabContent.tsx
35	surfsense_web/components/ui/button.tsx

⚠️ Inconsistent Changes Detected

File Path	Warning
surfsense_web/app/(home)/login/GoogleLoginButton.tsx	Contains merge conflict markers and unrelated refactoring of Google logo component in a PR focused on OpenTelemetry integration
surfsense_web/components/assistant-ui/connector-popup.tsx	UI styling changes for Alert variant and button appearance are unrelated to observability integration
surfsense_web/components/assistant-ui/document-upload-popup.tsx	UI styling changes for Alert variant and button appearance are unrelated to observability integration
surfsense_web/components/layout/ui/icon-rail/SearchSpaceAvatar.tsx	Refactoring of context menu handling from ContextMenu to DropdownMenu is unrelated to OpenTelemetry integration
surfsense_web/components/layout/ui/tabs/DocumentTabContent.tsx	Removing duplicate import statement is unrelated cleanup
surfsense_web/components/ui/button.tsx	Secondary button variant styling change is unrelated to observability

[vercel]

@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 80bbfe45-098e-46d8-8660-ae0f221697ba

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}