medialibrary(index) : reducing the index file size and saving them in batches

[kmurugulla]

Optimize index storage and fix incremental indexing bugs

Summary

Reduces index file sizes by 37MB and fixes external media (YouTube/Vimeo) being incorrectly removed during
incremental builds.

Changes

Storage Optimization

• Remove usage sheet from storage - (usage now derived from media entries)
• Adaptive chunk sizing - 6k-8k entries/chunk (down from 20k) for ~3-5MB files
• Batched uploads - 3 chunks at a time with 500ms delays (prevents 503 errors)
• Deleted dead code: saveIndexChunks from admin-api.js, unused chunkMediaSheet

Bug Fixes

1. Fixed external media removal - YouTube/Vimeo videos now preserved across incremental runs

   • processPageMediaUpdates now only compares medialog-sourced entries (medialog.js)
   • Image truthing excludes external media from processing (incremental.js)
   • Path normalization fix in external media removal check (linked-content.js)

2. Code consolidation - Moved getAdaptiveChunkSize into worker/fetch.js

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):

Fix #

[aem-code-sync]

Hello, I'm the AEM Code Sync Bot and I will run some actions to deploy your branch.
In case there are problems, just click the checkbox below to rerun the respective action.

• Re-sync branch

Commits

• bd628f2 ✅ (latest)
• e8c3157 ✅
• 607f6c5 ✅
• f3e449a ✅