[Feat] Vision-based autocomplete with KB grounding#1130
Conversation
…ionText, updateSuggestionText
|
@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 92d75ad..a180bf5
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (34)
• surfsense_backend/alembic/versions/119_add_vision_llm_id_to_search_spaces.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/__init__.py
• surfsense_backend/app/routes/autocomplete_routes.py
• surfsense_backend/app/routes/search_spaces_routes.py
• surfsense_backend/app/schemas/new_llm_config.py
• surfsense_backend/app/services/llm_service.py
• surfsense_backend/app/services/vision_autocomplete_service.py
• surfsense_desktop/.npmrc
• surfsense_desktop/electron-builder.yml
• surfsense_desktop/package.json
• surfsense_desktop/pnpm-lock.yaml
• surfsense_desktop/scripts/build-electron.mjs
• surfsense_desktop/src/ipc/channels.ts
• surfsense_desktop/src/ipc/handlers.ts
• surfsense_desktop/src/main.ts
• surfsense_desktop/src/modules/autocomplete/index.ts
• surfsense_desktop/src/modules/autocomplete/screenshot.ts
• surfsense_desktop/src/modules/autocomplete/suggestion-window.ts
• surfsense_desktop/src/modules/permissions.ts
• surfsense_desktop/src/modules/platform.ts
• surfsense_desktop/src/modules/window.ts
• surfsense_desktop/src/preload.ts
• surfsense_web/app/dashboard/[search_space_id]/user-settings/components/DesktopContent.tsx
• surfsense_web/app/desktop/permissions/page.tsx
• surfsense_web/app/desktop/suggestion/layout.tsx
• surfsense_web/app/desktop/suggestion/page.tsx
• surfsense_web/app/desktop/suggestion/suggestion.css
• surfsense_web/components/editor-panel/editor-panel.tsx
• surfsense_web/components/layout/ui/tabs/DocumentTabContent.tsx
• surfsense_web/components/settings/llm-role-manager.tsx
• surfsense_web/components/settings/user-settings-dialog.tsx
• surfsense_web/contracts/types/new-llm-config.types.ts
• surfsense_web/types/window.d.ts
2 participants
Description
Add vision-based autocomplete that captures the user's screen on
Cmd+Shift+Space, analyzes it with a Vision LLM, searches the knowledge base for relevant context, and streams a writing suggestion in a floating tooltip at the cursor position. Accepting a suggestion pastes it into the active app via the clipboard.API Changes
POST /api/v1/autocomplete/vision/stream— new SSE endpoint for vision autocompletevision_llm_idadded to search space LLM preferences (read/update)Change Type
Testing Performed
Checklist
High-level PR Summary
This PR introduces a vision-based autocomplete feature that captures screenshots when the user presses
Cmd+Shift+Space, analyzes them using a Vision LLM to understand context, searches the knowledge base for relevant information, and streams intelligent writing suggestions in a floating tooltip at the cursor position. The implementation spans backend API endpoints for vision autocomplete with SSE streaming, database schema updates to store vision LLM preferences per search space, desktop app integration with native screenshot capture and macOS permissions handling, and UI components for managing autocomplete settings and displaying suggestions. The feature requires macOS accessibility and screen recording permissions to inject accepted suggestions into the active application via clipboard.⏱️ Estimated Review Time: 1-3 hours
💡 Review Order Suggestion
surfsense_backend/alembic/versions/119_add_vision_llm_id_to_search_spaces.pysurfsense_backend/app/db.pysurfsense_backend/app/schemas/new_llm_config.pysurfsense_backend/app/routes/search_spaces_routes.pysurfsense_backend/app/services/llm_service.pysurfsense_backend/app/routes/autocomplete_routes.pysurfsense_backend/app/services/vision_autocomplete_service.pysurfsense_backend/app/routes/__init__.pysurfsense_desktop/package.jsonsurfsense_desktop/.npmrcsurfsense_desktop/electron-builder.ymlsurfsense_desktop/pnpm-lock.yamlsurfsense_desktop/scripts/build-electron.mjssurfsense_desktop/src/modules/permissions.tssurfsense_desktop/src/modules/platform.tssurfsense_desktop/src/ipc/channels.tssurfsense_desktop/src/ipc/handlers.tssurfsense_desktop/src/modules/autocomplete/screenshot.tssurfsense_desktop/src/modules/autocomplete/suggestion-window.tssurfsense_desktop/src/modules/autocomplete/index.tssurfsense_desktop/src/modules/window.tssurfsense_desktop/src/main.tssurfsense_desktop/src/preload.tssurfsense_web/types/window.d.tssurfsense_web/contracts/types/new-llm-config.types.tssurfsense_web/app/desktop/suggestion/suggestion.csssurfsense_web/app/desktop/suggestion/layout.tsxsurfsense_web/app/desktop/suggestion/page.tsxsurfsense_web/app/desktop/permissions/page.tsxsurfsense_web/app/dashboard/[search_space_id]/user-settings/components/DesktopContent.tsxsurfsense_web/components/settings/llm-role-manager.tsxsurfsense_web/components/settings/user-settings-dialog.tsxsurfsense_web/components/editor-panel/editor-panel.tsxsurfsense_web/components/layout/ui/tabs/DocumentTabContent.tsxsurfsense_web/components/editor-panel/editor-panel.tsxsurfsense_web/components/layout/ui/tabs/DocumentTabContent.tsx