Improve offline sync + fix sync system issues

[mdmohsin7]

ScreenRecording_12-13-2025.14-07-22_1.MP4

[gemini-code-assist]

Code Review

This pull request introduces a new SyncBottomSheet widget. While visually well-structured, there are opportunities to improve the architecture by moving business logic and service calls out of the UI widget and into the SyncProvider to enhance maintainability and testability. Additionally, a duplicated calculation should be refactored to improve efficiency and reduce redundancy, aligning with principles of consistent value usage.

[gemini-code-assist]

To maintain a clear separation of concerns and improve testability, business logic and service calls should be handled within the SyncProvider rather than directly in the widget's build method.

Currently, ServiceManager is accessed directly to get information about orphaned files (lines 38-39), calculate timeAgo (lines 41-53), and to trigger uploads (line 320). This logic should be encapsulated within the SyncProvider.

Please consider the following refactoring:

1. Add getters for hasOrphanedFiles and orphanedCount to SyncProvider.
2. Add a method uploadOrphanedFiles() to SyncProvider.
3. Move the timeAgo calculation logic into a getter in SyncProvider. Also, the check on line 43 if (hasPendingData && pendingFlashPages.isNotEmpty) is redundant, as hasPendingData is already pendingFlashPages.isNotEmpty.

In SyncProvider:

bool get hasOrphanedFiles => _walService.getSyncs().flashPage.hasOrphanedFiles;
int get orphanedFilesCount => _walService.getSyncs().flashPage.orphanedFilesCount;

Future<void> uploadOrphanedFiles() async {
  await _walService.getSyncs().flashPage.uploadOrphanedFiles();
  // You might need to refresh state and notify listeners after this.
  await refreshWals(); 
}

String get oldestPendingWalTimeAgo {
  final pendingFlashPages = allWals.where((w) => w.storage == WalStorage.flashPage && w.status == WalStatus.miss).toList();
  if (pendingFlashPages.isEmpty) return '';

  final oldestWal = pendingFlashPages.reduce((a, b) => a.timerStart < b.timerStart ? a : b);
  final nowInSeconds = DateTime.now().millisecondsSinceEpoch ~/ 1000;
  final minutesAgo = (nowInSeconds - oldestWal.timerStart) ~/ 60;

  if (minutesAgo < 60) {
    return '$minutesAgo minutes ago';
  } else if (minutesAgo < 1440) { // 24 * 60
    return '${minutesAgo ~/ 60} hours ago';
  } else {
    return '${minutesAgo ~/ 1440} days ago';
  }
}

Then, in SyncBottomSheet, you can access this data from syncProvider and simplify the build method significantly. This makes the widget more declarative and easier to maintain.

References

1. When separating UI and provider logic, provider methods (e.g., clearFilters, removeFilter) should handle state management, while the UI layer is responsible for invoking subsequent asynchronous actions (e.g., applyFilters for server-side updates) after the state has been modified.

[gemini-code-assist]

The calculation for the total duration of pending recordings, pendingFlashPages.fold(0, (sum, w) => sum + w.seconds), is performed twice in the build method (here on line 148 and again on line 232).

To avoid redundant computation and improve code clarity and maintainability, you should calculate this value once at the beginning of the build method and store it in a variable.

For example, you can add this after line 35:

final totalPendingSeconds = pendingFlashPages.fold<int>(0, (sum, w) => sum + w.seconds);

Then you can use _formatDuration(totalPendingSeconds) in both places. This makes the code more efficient and ensures that if the calculation logic changes, it only needs to be updated in one spot.

References

1. Ensure consistency in constant values used for calculations across different parts of the codebase, especially when those calculations affect user-facing metrics like progress reporting. Define a single, well-defined constant and reuse it to avoid inaccuracies.

[mdmohsin7]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request delivers a comprehensive overhaul of the offline synchronization system, primarily targeting Limitless devices. It provides users with a clearer view of their sync status through a new dedicated bottom sheet and significantly improves the reliability of data transfer from the device to the cloud. The changes ensure that recordings are safely stored locally before upload, with mechanisms for recovering from interruptions, and that backend processing accurately reflects the conversation timelines and origins.

Highlights

• New Sync UI: Introduced a new SyncBottomSheet widget that provides a dedicated interface for users to monitor and manage the synchronization status of their Limitless device, including pending recordings and orphaned files. This replaces a simpler card widget on the conversations page.
• Robust Offline Sync Logic: Refactored the flash page synchronization process to be more resilient. This includes saving extracted audio frames to local files first, tracking these files for recovery in case of app crashes, and then uploading them to the backend. Incremental acknowledgements are sent to the device during this process.
• Orphaned File Management: Implemented a mechanism to detect and upload 'orphaned files' – recordings saved locally on the phone from previous sync attempts that might not have been fully uploaded. These files are now managed via SharedPreferences and automatically uploaded when the app starts.
• Improved Device Communication: Enhanced the LimitlessDeviceConnection to include better error handling during initialization and to clear internal data buffers when switching between batch and real-time streaming modes, preventing data contamination.
• Backend Sync Enhancements: Updated backend endpoints to correctly handle conversation sources (e.g., 'limitless' vs. 'omi'), ensure proper timezone handling for timestamps, and allow for dynamic updates of conversation finished_at times based on incoming segments.

Changelog

• app/lib/pages/conversations/conversations_page.dart
  • Removed LimitlessSyncCardWidget.
• app/lib/pages/home/page.dart
  • Added imports for bt_device.dart, sync_provider.dart, and sync_bottom_sheet.dart.
  • Integrated a new sync icon (cloud icon) into the home page, which uses a Consumer2 widget to display sync status and opens the SyncBottomSheet on tap.
• app/lib/pages/home/widgets/sync_bottom_sheet.dart
  • New file added, implementing the SyncBottomSheet widget.
  • Displays current sync status (e.g., 'Catching Up', 'Recordings Available', 'All Synced').
  • Shows progress indicators and action buttons (Sync, Upload) based on sync state.
  • Provides information on pending recordings, orphaned files, and device connection status.
  • Includes utility functions for formatting duration and getting sync status text.
• app/lib/services/devices/limitless_connection.dart
  • Added try-catch block to _initialize for robust error handling during device setup.
  • Ensured _rawDataBuffer is cleared before enabling batch mode and when disabling batch mode to prevent data mixing.
  • Refactored enableRealTimeMode to internally call disableBatchMode for consistent state management.
  • Introduced extractFramesWithSessionInfo to combine opus frame extraction with session marker parsing, providing richer data.
  • Removed a debug print statement from extractOpusFramesFromPage.
• app/lib/services/wals.dart
  • Modified Wal constructor to initialize data as an empty constant list.
  • Implemented FlashPageWalSync with new functionality for managing 'orphaned files' (locally saved files not yet uploaded) using SharedPreferences for persistence and recovery.
  • Added uploadOrphanedFiles, hasOrphanedFiles, and orphanedFilesCount to handle these recovery scenarios.
  • Completely refactored _syncWal to first save extracted frames to local files (batching them by time gaps or session markers), send incremental ACKs to the device, and then sequentially upload these local files to the backend.
  • Replaced _flushChunk with _saveBatchToFile for local file storage and _extractTimestampFromFilename for chronological sorting.
  • The start() method now automatically checks for and initiates the upload of any existing orphaned files.
  • Updated _acknowledgeProcessedData and _enableRealTimeMode to safely cast and interact with LimitlessDeviceConnection.
• backend/database/conversations.py
  • Modified update_conversation_segments to accept an optional finished_at parameter, allowing the conversation's end time to be updated alongside segments.
  • Updated get_closest_conversation_to_timestamps to use datetime.fromtimestamp with timezone.utc for more accurate and consistent timestamp handling.
• backend/routers/sync.py
  • Imported timezone from datetime and ConversationSource from models.conversation.
  • The process_segment function now accepts an optional source parameter, defaulting to ConversationSource.omi, to identify the origin of the conversation.
  • Ensured started_at and finished_at for new conversations are created with timezone.utc.
  • The sync_local_files endpoint now detects the ConversationSource (omi or limitless) from the uploaded filenames and passes it to process_segment.
  • When updating an existing conversation, process_segment now calculates and updates the finished_at timestamp based on the latest segment, ensuring the conversation's duration is accurately reflected.

Activity

• mdmohsin7 requested a Gemini summary.
• gemini-code-assist[bot] suggested refactoring the SyncBottomSheet to move business logic (orphaned files, timeAgo calculation) into SyncProvider for better separation of concerns and testability.
• gemini-code-assist[bot] recommended calculating totalPendingSeconds once in the build method of SyncBottomSheet to avoid redundant computation.