Skip to content

feat(health): optimize library sync for large libraries - #333

Merged
javi11 merged 1 commit into
kipsilabs:mainfrom
drondeseries:perf/library-sync-optimizations
Feb 25, 2026
Merged

feat(health): optimize library sync for large libraries#333
javi11 merged 1 commit into
kipsilabs:mainfrom
drondeseries:perf/library-sync-optimizations

Conversation

@drondeseries

Copy link
Copy Markdown
Contributor

Description

This PR optimizes the library sync process to handle large libraries (15k+ files) more efficiently, especially when using network mounts (rclone).

Key Changes

  • Streaming Batch Additions: Instead of collecting all new files in a large slice in RAM, records are now processed and inserted into the database in batches of 1,000 as they are discovered. This significantly reduces the memory footprint.
  • Improved Filesystem Performance: Switched from filepath.Walk to filepath.WalkDir for metadata scanning to avoid unnecessary os.Lstat calls.
  • Progress Visibility: Added periodic log messages every 1,000 files during scanning and processing phases to provide feedback during long syncs.
  • Minor Optimizations: Moved path cleaning outside of loops in filesystem walk functions.

- Implement streaming batch additions to reduce RAM usage\n- Use WalkDir instead of Walk for faster metadata scanning\n- Add periodic progress logging every 1000 files
@javi11
javi11 merged commit 2fc5682 into kipsilabs:main Feb 25, 2026
1 check passed
yoshitaka420 pushed a commit to yoshitaka420/altmount that referenced this pull request Jun 1, 2026
- Implement streaming batch additions to reduce RAM usage\n- Use WalkDir instead of Walk for faster metadata scanning\n- Add periodic progress logging every 1000 files

Co-authored-by: drondeseries <drondeseries@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants