feat(0.0.12): bumped version, added composio connectors and various fixes #740
Conversation
- Introduced new enum values for Composio connectors: COMPOSIO_GOOGLE_DRIVE_CONNECTOR, COMPOSIO_GMAIL_CONNECTOR, and COMPOSIO_GOOGLE_CALENDAR_CONNECTOR. - Updated database migration to add these new enum values to the relevant types. - Refactored Composio integration logic to handle specific connector types, improving the management of connected accounts and indexing processes. - Enhanced frontend components to support the new Composio connector types, including updated UI elements and connector configuration handling. - Improved backend services to manage Composio connected accounts more effectively, including deletion and indexing tasks.
… handling - Updated the list_gmail_messages method to support pagination with page tokens, allowing for more efficient message retrieval. - Modified the return structure to include next_page_token and result_size_estimate for better client-side handling. - Improved error handling and logging throughout the Gmail indexing process, ensuring better visibility into failures. - Implemented batch processing for Gmail messages, committing changes incrementally to prevent data loss. - Ensured consistent timestamp updates for connectors, even when no documents are indexed, to maintain accurate UI states. - Refactored the indexing logic to streamline message processing and enhance overall performance.
…selection - Added a new endpoint to list folders and files in a user's Composio Google Drive, supporting hierarchical structure. - Implemented UI components for selecting specific folders and files to index, improving user control over indexing options. - Introduced indexing options for maximum files per folder and inclusion of subfolders, allowing for customizable indexing behavior. - Enhanced error handling and logging for Composio Drive operations, ensuring better visibility into issues during file retrieval and indexing. - Updated the Composio configuration component to reflect new selection capabilities and indexing options.
- Enhanced the handling of file content from Composio, supporting both binary and text files with appropriate processing methods. - Introduced robust error logging and handling for file content extraction, ensuring better visibility into issues during processing. - Updated the indexing logic to accommodate new content processing methods, improving overall reliability and user feedback on errors. - Added temporary file handling for binary files to facilitate text extraction using the ETL service.
- Added support for extracting connected account IDs from query parameters in the Composio callback, accommodating both camelCase and snake_case formats. - Improved logging for received query parameters in the Composio callback to enhance debugging. - Updated Google Drive folder listing logs to remove unnecessary emoji for consistency. - Expanded the connector dialog to include Composio Google Drive as a recognized connector type, improving user interface clarity.
…a sync support - Added methods to retrieve the starting page token and list changes in Google Drive, enabling delta sync capabilities. - Updated Composio service to handle file download directory configuration. - Modified indexing tasks to support delta sync, improving efficiency by processing only changed files. - Adjusted date handling in connector tasks to allow optional start and end dates. - Improved error handling and logging throughout the Composio indexing process.
…r handling - Added user-friendly re-authentication messages for expired or revoked tokens in both Google Calendar and Gmail connectors. - Updated error handling in indexing tasks to log specific authentication errors and provide clearer feedback to users. - Enhanced the connector UI to handle indexing failures more effectively, improving overall user experience.
…eractions - Refactored Composio connector configuration components to improve modularity and maintainability. - Simplified the ComposioCalendarConfig, ComposioGmailConfig, and ComposioDriveConfig components by removing unnecessary state management and UI elements. - Added functionality to remove selected folders and files in the Google Drive and Composio Drive configurations, enhancing user experience. - Updated connector display names for better clarity in the UI. - Improved the overall structure of the connector edit view for better readability and usability.
- Updated the Composio connector naming logic to dynamically generate user-friendly names based on existing connectors. - Introduced new utility functions for counting connectors and retrieving base names for specific connector types. - Enhanced the UI components to improve accessibility and focus management, ensuring a better user experience when interacting with connector dialogs.
(Based on discussion with Sid)
- Added normalization for "undefined" strings to None in date parameters to prevent parsing errors. - Improved date range validation to ensure start_date is strictly before end_date, adjusting end_date if necessary. - Updated Google Calendar and Composio connector indexing logic to handle duplicate content more effectively, logging warnings for skipped events. - Enhanced error handling during final commits to manage integrity errors gracefully. - Refactored date handling in various connector indexers for consistency and reliability.
- Enhanced Google Calendar and Composio connector indexing to track and log duplicate content, preventing re-indexing of already processed events. - Implemented robust error handling during final commits to manage integrity errors gracefully, ensuring successful indexing despite potential duplicates. - Updated notification service to differentiate between actual errors and warnings for duplicate content, improving user feedback. - Refactored date handling to ensure valid date ranges and adjusted end dates when necessary for better indexing accuracy.
This additionally fixes a bug where the New Chat button wasn't working properly after creating a new chat
- Improved indexing state management by refining the logic for handling notifications, ensuring accurate updates for in-progress, completed, and failed states. - Introduced a new utility function to format inbox counts, displaying numbers up to 999 and using "k+" for larger counts, enhancing user interface clarity. - Updated sidebar components to utilize the new inbox count formatting, improving the overall user experience.
- Enhanced the logic for determining success cases during indexing by distinguishing between duplicate warnings and empty results. - Updated notification messages to provide clearer feedback for empty results, improving user understanding of indexing outcomes. - Ensured that notifications reflect accurate statuses, maintaining consistency in user feedback during the indexing process.
- Updated error handling in the indexing functions for BookStack, Confluence, Google Calendar, Jira, Linear, and Luma connectors to log specific error messages when failures occur. - Enhanced logging for cases where no pages or events are found, providing clearer informational messages instead of treating them as critical errors. - Ensured consistent error reporting across all connector indexers, improving debugging and user feedback during indexing operations.
…ta queries - Modified the Google Drive indexer to use SQLAlchemy's cast function for querying document metadata, ensuring proper type handling for file IDs. - Improved the consistency of metadata queries across the indexing functions, enhancing reliability in document retrieval and processing.
feat: Enhance Composio connectors & some fixes
Updated README.md, reorganized and fixed sidebar
Updated video link in README.
…wledge base tool - Added functionality to dynamically discover available connectors and document types for the knowledge base tool, enhancing its flexibility and usability. - Introduced new mapping functions and updated existing search methods to accommodate Composio connectors, improving integration with external services. - Enhanced error handling and logging for connector discovery processes, ensuring better feedback during failures.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on df6943b..555df90
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (50)
• README.md
• surfsense_backend/alembic/versions/79_add_composio_connector_enums.py
• surfsense_backend/app/agents/new_chat/chat_deepagent.py
• surfsense_backend/app/agents/new_chat/tools/__init__.py
• surfsense_backend/app/agents/new_chat/tools/knowledge_base.py
• surfsense_backend/app/agents/new_chat/tools/registry.py
• surfsense_backend/app/connectors/composio_connector.py
• surfsense_backend/app/connectors/composio_gmail_connector.py
• surfsense_backend/app/connectors/composio_google_calendar_connector.py
• surfsense_backend/app/connectors/composio_google_drive_connector.py
• surfsense_backend/app/connectors/google_calendar_connector.py
• surfsense_backend/app/connectors/google_gmail_connector.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/composio_routes.py
• surfsense_backend/app/routes/google_drive_add_connector_route.py
• surfsense_backend/app/routes/search_source_connectors_routes.py
• surfsense_backend/app/services/composio_service.py
• surfsense_backend/app/services/connector_service.py
• surfsense_backend/app/services/notification_service.py
• surfsense_backend/app/tasks/celery_tasks/connector_tasks.py
• surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py
• surfsense_backend/app/tasks/composio_indexer.py
• surfsense_backend/app/tasks/connector_indexers/base.py
• surfsense_backend/app/tasks/connector_indexers/bookstack_indexer.py
• surfsense_backend/app/tasks/connector_indexers/confluence_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_calendar_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_drive_indexer.py
• surfsense_backend/app/tasks/connector_indexers/google_gmail_indexer.py
• surfsense_backend/app/tasks/connector_indexers/jira_indexer.py
• surfsense_backend/app/tasks/connector_indexers/linear_indexer.py
• surfsense_backend/app/tasks/connector_indexers/luma_indexer.py
• surfsense_backend/app/utils/connector_naming.py
• surfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/RowActions.tsx
• surfsense_web/components/assistant-ui/connector-popup.tsx
• surfsense_web/components/assistant-ui/connector-popup/components/composio-connector-card.tsx
• surfsense_web/components/assistant-ui/connector-popup/config/connector-status-config.json
• surfsense_web/components/assistant-ui/connector-popup/connect-forms/components/obsidian-connect-form.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-calendar-config.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-config.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-drive-config.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-gmail-config.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/google-drive-config.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/index.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/views/connector-edit-view.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/views/indexing-configuration-view.tsx
• surfsense_web/components/assistant-ui/connector-popup/constants/connector-constants.ts
• surfsense_web/components/assistant-ui/connector-popup/constants/connector-popup.schemas.ts
• surfsense_web/components/assistant-ui/connector-popup/hooks/use-connector-dialog.ts
• surfsense_web/components/assistant-ui/connector-popup/hooks/use-indexing-connectors.ts
• surfsense_web/components/assistant-ui/connector-popup/tabs/active-connectors-tab.tsx
⏭️ Files skipped (23)
| Locations |
|---|
surfsense_web/components/assistant-ui/connector-popup/tabs/all-connectors-tab.tsx |
surfsense_web/components/assistant-ui/connector-popup/utils/connector-document-mapping.ts |
surfsense_web/components/assistant-ui/connector-popup/views/composio-toolkit-view.tsx |
surfsense_web/components/connectors/composio-drive-folder-tree.tsx |
surfsense_web/components/layout/providers/LayoutDataProvider.tsx |
surfsense_web/components/layout/ui/sidebar/AllPrivateChatsSidebar.tsx |
surfsense_web/components/layout/ui/sidebar/AllSharedChatsSidebar.tsx |
surfsense_web/components/layout/ui/sidebar/InboxSidebar.tsx |
surfsense_web/components/layout/ui/sidebar/NavSection.tsx |
surfsense_web/components/settings/llm-role-manager.tsx |
surfsense_web/components/settings/model-config-manager.tsx |
surfsense_web/components/settings/prompt-config-manager.tsx |
surfsense_web/components/ui/dialog.tsx |
surfsense_web/contracts/enums/connector.ts |
surfsense_web/contracts/enums/connectorIcons.tsx |
surfsense_web/contracts/types/connector.types.ts |
surfsense_web/contracts/types/document.types.ts |
surfsense_web/hooks/use-composio-drive-folders.ts |
surfsense_web/lib/apis/connectors-api.service.ts |
surfsense_web/lib/connectors/utils.ts |
surfsense_web/lib/query-client/cache-keys.ts |
surfsense_web/messages/en.json |
surfsense_web/public/connectors/obsidian.svg |
Description
Motivation and Context
FIX #
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This PR adds Composio-based Google integration (Drive, Gmail, Calendar) as new connectors, enabling users to connect via managed OAuth without app verification. Key changes include splitting the generic Composio connector into three specific connector types with dedicated document types, implementing toolkit-specific indexers with delta sync support for Drive, adding dynamic connector discovery for AI tools, improving error handling with re-authentication messages for expired tokens, enhancing UI with folder/file selection for Drive and better indexing feedback, and fixing various bugs including date range handling, notification state management, and Google Drive file removal logic. The PR also includes general improvements like better loading states, inbox count formatting, and sidebar animations.
⏱️ Estimated Review Time: 30-90 minutes
💡 Review Order Suggestion
surfsense_backend/alembic/versions/79_add_composio_connector_enums.pysurfsense_backend/app/db.pysurfsense_backend/app/services/composio_service.pysurfsense_backend/app/connectors/composio_connector.pysurfsense_backend/app/connectors/composio_google_drive_connector.pysurfsense_backend/app/connectors/composio_gmail_connector.pysurfsense_backend/app/connectors/composio_google_calendar_connector.pysurfsense_backend/app/tasks/composio_indexer.pysurfsense_backend/app/tasks/connector_indexers/base.pysurfsense_backend/app/routes/composio_routes.pysurfsense_backend/app/routes/search_source_connectors_routes.pysurfsense_backend/app/agents/new_chat/chat_deepagent.pysurfsense_backend/app/agents/new_chat/tools/knowledge_base.pysurfsense_backend/app/services/connector_service.pysurfsense_backend/app/agents/new_chat/tools/__init__.pysurfsense_backend/app/agents/new_chat/tools/registry.pysurfsense_backend/app/tasks/celery_tasks/connector_tasks.pysurfsense_backend/app/tasks/celery_tasks/schedule_checker_task.pysurfsense_backend/app/tasks/connector_indexers/google_calendar_indexer.pysurfsense_backend/app/tasks/connector_indexers/google_drive_indexer.pysurfsense_backend/app/tasks/connector_indexers/google_gmail_indexer.pysurfsense_backend/app/routes/google_drive_add_connector_route.pysurfsense_backend/app/services/notification_service.pysurfsense_backend/app/connectors/google_calendar_connector.pysurfsense_backend/app/connectors/google_gmail_connector.pysurfsense_backend/app/tasks/connector_indexers/confluence_indexer.pysurfsense_backend/app/tasks/connector_indexers/jira_indexer.pysurfsense_backend/app/tasks/connector_indexers/linear_indexer.pysurfsense_backend/app/tasks/connector_indexers/luma_indexer.pysurfsense_backend/app/tasks/connector_indexers/bookstack_indexer.pysurfsense_backend/app/utils/connector_naming.pysurfsense_web/contracts/enums/connector.tssurfsense_web/contracts/types/connector.types.tssurfsense_web/contracts/types/document.types.tssurfsense_web/contracts/enums/connectorIcons.tsxsurfsense_web/components/assistant-ui/connector-popup/constants/connector-constants.tssurfsense_web/components/assistant-ui/connector-popup/connector-configs/index.tsxsurfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-drive-config.tsxsurfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-gmail-config.tsxsurfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-calendar-config.tsxsurfsense_web/components/connectors/composio-drive-folder-tree.tsxsurfsense_web/hooks/use-composio-drive-folders.tssurfsense_web/lib/apis/connectors-api.service.tssurfsense_web/lib/query-client/cache-keys.tssurfsense_web/components/assistant-ui/connector-popup/hooks/use-connector-dialog.tssurfsense_web/components/assistant-ui/connector-popup/hooks/use-indexing-connectors.tssurfsense_web/components/assistant-ui/connector-popup/tabs/all-connectors-tab.tsxsurfsense_web/components/assistant-ui/connector-popup/tabs/active-connectors-tab.tsxsurfsense_web/components/assistant-ui/connector-popup/connector-configs/views/connector-edit-view.tsxsurfsense_web/components/assistant-ui/connector-popup/connector-configs/views/indexing-configuration-view.tsxsurfsense_web/components/assistant-ui/connector-popup.tsxsurfsense_web/components/assistant-ui/connector-popup/utils/connector-document-mapping.tssurfsense_web/components/layout/providers/LayoutDataProvider.tsxsurfsense_web/components/layout/ui/sidebar/InboxSidebar.tsxsurfsense_web/components/layout/ui/sidebar/NavSection.tsxREADME.mdsurfsense_backend/app/connectors/google_drive_connector.pysurfsense_web/components/assistant-ui/connector-popup/connector-configs/components/google-drive-config.tsxsurfsense_web/lib/connectors/utils.tssurfsense_web/components/assistant-ui/connector-popup/config/connector-status-config.jsonsurfsense_web/components/layout/ui/sidebar/AllPrivateChatsSidebar.tsxsurfsense_web/components/layout/ui/sidebar/AllSharedChatsSidebar.tsxsurfsense_web/public/connectors/obsidian.svgsurfsense_web/components/ui/dialog.tsxsurfsense_web/components/settings/llm-role-manager.tsxsurfsense_web/components/settings/model-config-manager.tsxsurfsense_web/components/settings/prompt-config-manager.tsxsurfsense_web/messages/en.jsonsurfsense_web/app/dashboard/[search_space_id]/documents/(manage)/components/RowActions.tsx