feat: Enhance Composio connectors & some fixes #728
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced new enum values for Composio connectors: COMPOSIO_GOOGLE_DRIVE_CONNECTOR, COMPOSIO_GMAIL_CONNECTOR, and COMPOSIO_GOOGLE_CALENDAR_CONNECTOR. - Updated database migration to add these new enum values to the relevant types. - Refactored Composio integration logic to handle specific connector types, improving the management of connected accounts and indexing processes. - Enhanced frontend components to support the new Composio connector types, including updated UI elements and connector configuration handling. - Improved backend services to manage Composio connected accounts more effectively, including deletion and indexing tasks.
|
@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by RecurseML
🔍 Review performed on 2065009..0cd65e9
✨ No bugs found, your code is sparkling clean
✅ Files analyzed, no issues (19)
• surfsense_backend/alembic/versions/74_add_composio_connector_enums.py
• surfsense_backend/app/db.py
• surfsense_backend/app/routes/composio_routes.py
• surfsense_backend/app/routes/search_source_connectors_routes.py
• surfsense_backend/app/services/composio_service.py
• surfsense_backend/app/tasks/celery_tasks/connector_tasks.py
• surfsense_backend/app/tasks/composio_indexer.py
• surfsense_web/components/assistant-ui/connector-popup.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-config.tsx
• surfsense_web/components/assistant-ui/connector-popup/connector-configs/index.tsx
• surfsense_web/components/assistant-ui/connector-popup/constants/connector-constants.ts
• surfsense_web/components/assistant-ui/connector-popup/constants/connector-popup.schemas.ts
• surfsense_web/components/assistant-ui/connector-popup/hooks/use-connector-dialog.ts
• surfsense_web/components/assistant-ui/connector-popup/tabs/all-connectors-tab.tsx
• surfsense_web/components/assistant-ui/connector-popup/utils/connector-document-mapping.ts
• surfsense_web/contracts/enums/connector.ts
• surfsense_web/contracts/enums/connectorIcons.tsx
• surfsense_web/contracts/types/connector.types.ts
• surfsense_web/contracts/types/document.types.ts
… handling - Updated the list_gmail_messages method to support pagination with page tokens, allowing for more efficient message retrieval. - Modified the return structure to include next_page_token and result_size_estimate for better client-side handling. - Improved error handling and logging throughout the Gmail indexing process, ensuring better visibility into failures. - Implemented batch processing for Gmail messages, committing changes incrementally to prevent data loss. - Ensured consistent timestamp updates for connectors, even when no documents are indexed, to maintain accurate UI states. - Refactored the indexing logic to streamline message processing and enhance overall performance.
…selection - Added a new endpoint to list folders and files in a user's Composio Google Drive, supporting hierarchical structure. - Implemented UI components for selecting specific folders and files to index, improving user control over indexing options. - Introduced indexing options for maximum files per folder and inclusion of subfolders, allowing for customizable indexing behavior. - Enhanced error handling and logging for Composio Drive operations, ensuring better visibility into issues during file retrieval and indexing. - Updated the Composio configuration component to reflect new selection capabilities and indexing options.
- Enhanced the handling of file content from Composio, supporting both binary and text files with appropriate processing methods. - Introduced robust error logging and handling for file content extraction, ensuring better visibility into issues during processing. - Updated the indexing logic to accommodate new content processing methods, improving overall reliability and user feedback on errors. - Added temporary file handling for binary files to facilitate text extraction using the ETL service.
- Added support for extracting connected account IDs from query parameters in the Composio callback, accommodating both camelCase and snake_case formats. - Improved logging for received query parameters in the Composio callback to enhance debugging. - Updated Google Drive folder listing logs to remove unnecessary emoji for consistency. - Expanded the connector dialog to include Composio Google Drive as a recognized connector type, improving user interface clarity.
…a sync support - Added methods to retrieve the starting page token and list changes in Google Drive, enabling delta sync capabilities. - Updated Composio service to handle file download directory configuration. - Modified indexing tasks to support delta sync, improving efficiency by processing only changed files. - Adjusted date handling in connector tasks to allow optional start and end dates. - Improved error handling and logging throughout the Composio indexing process.
…r handling - Added user-friendly re-authentication messages for expired or revoked tokens in both Google Calendar and Gmail connectors. - Updated error handling in indexing tasks to log specific authentication errors and provide clearer feedback to users. - Enhanced the connector UI to handle indexing failures more effectively, improving overall user experience.
…eractions - Refactored Composio connector configuration components to improve modularity and maintainability. - Simplified the ComposioCalendarConfig, ComposioGmailConfig, and ComposioDriveConfig components by removing unnecessary state management and UI elements. - Added functionality to remove selected folders and files in the Google Drive and Composio Drive configurations, enhancing user experience. - Updated connector display names for better clarity in the UI. - Improved the overall structure of the connector edit view for better readability and usability.
- Updated the Composio connector naming logic to dynamically generate user-friendly names based on existing connectors. - Introduced new utility functions for counting connectors and retrieving base names for specific connector types. - Enhanced the UI components to improve accessibility and focus management, ensuring a better user experience when interacting with connector dialogs.
- Added normalization for "undefined" strings to None in date parameters to prevent parsing errors. - Improved date range validation to ensure start_date is strictly before end_date, adjusting end_date if necessary. - Updated Google Calendar and Composio connector indexing logic to handle duplicate content more effectively, logging warnings for skipped events. - Enhanced error handling during final commits to manage integrity errors gracefully. - Refactored date handling in various connector indexers for consistency and reliability.
- Enhanced Google Calendar and Composio connector indexing to track and log duplicate content, preventing re-indexing of already processed events. - Implemented robust error handling during final commits to manage integrity errors gracefully, ensuring successful indexing despite potential duplicates. - Updated notification service to differentiate between actual errors and warnings for duplicate content, improving user feedback. - Refactored date handling to ensure valid date ranges and adjusted end dates when necessary for better indexing accuracy.
- Improved indexing state management by refining the logic for handling notifications, ensuring accurate updates for in-progress, completed, and failed states. - Introduced a new utility function to format inbox counts, displaying numbers up to 999 and using "k+" for larger counts, enhancing user interface clarity. - Updated sidebar components to utilize the new inbox count formatting, improving the overall user experience.
- Enhanced the logic for determining success cases during indexing by distinguishing between duplicate warnings and empty results. - Updated notification messages to provide clearer feedback for empty results, improving user understanding of indexing outcomes. - Ensured that notifications reflect accurate statuses, maintaining consistency in user feedback during the indexing process.
- Updated error handling in the indexing functions for BookStack, Confluence, Google Calendar, Jira, Linear, and Luma connectors to log specific error messages when failures occur. - Enhanced logging for cases where no pages or events are found, providing clearer informational messages instead of treating them as critical errors. - Ensured consistent error reporting across all connector indexers, improving debugging and user feedback during indexing operations.
…ta queries - Modified the Google Drive indexer to use SQLAlchemy's cast function for querying document metadata, ensuring proper type handling for file IDs. - Improved the consistency of metadata queries across the indexing functions, enhancing reliability in document retrieval and processing.
AnishSarkar22
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
COMPOSIO_GOOGLE_DRIVE_CONNECTOR,COMPOSIO_GMAIL_CONNECTOR, andCOMPOSIO_GOOGLE_CALENDAR_CONNECTOR.model-config-manager.tsxDescription
Motivation and Context
FIX #
Screenshots
API Changes
Change Type
Testing Performed
Checklist
High-level PR Summary
This PR refactors the Composio integration from using a single generic connector type to having three specific connector types:
COMPOSIO_GOOGLE_DRIVE_CONNECTOR,COMPOSIO_GMAIL_CONNECTOR, andCOMPOSIO_GOOGLE_CALENDAR_CONNECTOR. The changes include database migrations to add new enum values, updates to backend services to handle connector-specific logic including proper cleanup of old connected accounts when reconnecting, and corresponding frontend changes to display and manage these connectors individually rather than through a toolkit selector. The refactor also improves duplicate connector detection by checking for existing connectors by type/space/user and updating them during reconnection rather than creating duplicates.⏱️ Estimated Review Time: 30-90 minutes
💡 Review Order Suggestion
surfsense_backend/app/db.pysurfsense_backend/app/services/composio_service.pysurfsense_backend/alembic/versions/74_add_composio_connector_enums.pysurfsense_backend/app/routes/composio_routes.pysurfsense_backend/app/tasks/composio_indexer.pysurfsense_backend/app/routes/search_source_connectors_routes.pysurfsense_backend/app/tasks/celery_tasks/connector_tasks.pysurfsense_web/contracts/enums/connector.tssurfsense_web/contracts/types/connector.types.tssurfsense_web/contracts/types/document.types.tssurfsense_web/contracts/enums/connectorIcons.tsxsurfsense_web/components/assistant-ui/connector-popup/constants/connector-constants.tssurfsense_web/components/assistant-ui/connector-popup/utils/connector-document-mapping.tssurfsense_web/components/assistant-ui/connector-popup/hooks/use-connector-dialog.tssurfsense_web/components/assistant-ui/connector-popup/tabs/all-connectors-tab.tsxsurfsense_web/components/assistant-ui/connector-popup/constants/connector-popup.schemas.tssurfsense_web/components/assistant-ui/connector-popup/connector-configs/index.tsxsurfsense_web/components/assistant-ui/connector-popup/connector-configs/components/composio-config.tsxsurfsense_web/components/assistant-ui/connector-popup.tsx