fix(migration): chunked 0x03 GCM format to fix export OOM - #1128
Conversation
Whole-file AES-GCM decryption buffers an entire ciphertext internally (SunJCE), so importing a large encrypted export could OOM the app; the export side held the full plaintext and ciphertext in memory. Introduce a version 0x03 chunked-GCM format: - one PBKDF2 key, 8 MiB plaintext chunks, per-chunk counter IVs (base IV XOR chunk index in bytes 4..11), per-chunk GCM tag - encryptFile/encryptToFile stream in both directions (bounded memory) - decryptFile/decryptStream decrypt chunk by chunk (~2x chunk peak) - 0x02 legacy files still import (whole-file path, unchanged behavior) Import now stages the bundle as a local plaintext ZIP in the app cache instead of holding zipBytes in memory: preview copies the file (no full read), the manifest parses via a streaming JSON decoder, and import re-streams the staged file for attachments. The staged file is owned by the caller and removed in finally/onCleared/init cleanup. Verified: 120 MiB export decrypts under a 90 MiB heap in the functional test; MigrationCryptoTest (19), MigrationImporterEncryptionTest (8), and MigrationViewModelTest (20) all pass; ktlint + detekt clean.
Greptile SummaryThe PR replaces whole-file migration encryption and import buffering with authenticated, chunked AES-GCM and disk-backed streaming.
Confidence Score: 5/5The PR appears safe to merge; the previous findings are resolved and the latest sequence guard closes the stale password-prompt race. No new actionable issue remains. Current code authenticates chunk framing, handles large-length arithmetic safely, rejects invalid zero-sized chunks, removes failed or superseded staged files, and prevents stale preview checks from updating current state. All previous root threads were manually resolved. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
ExportZip[Plaintext export ZIP] --> ChunkEncrypt[Chunked AES-GCM encryption]
ChunkEncrypt --> Encrypted03[Encrypted 0x03 bundle]
Encrypted03 --> Stage[Stage bundle in cache]
Legacy02[Legacy 0x02 bundle] --> Stage
Stage --> Decrypt[Version-aware decryption]
Decrypt --> StagedZip[Staged plaintext ZIP]
StagedZip --> Manifest[Stream manifest]
StagedZip --> Attachments[Stream attachments]
Manifest --> Import[Import application data]
Attachments --> Import
Import --> Cleanup[Delete owned staged file]
Reviews (3): Last reviewed commit: "fix(migration): sequence-guard stale enc..." | Re-trigger Greptile |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…cycle Address Greptile review feedback on the chunked export format: - Authenticate 0x03 framing so the stream cannot be reordered, replayed, or truncated. Each chunk IV is now derived from (base IV, index) on the decrypt side instead of being read from the file, and each chunk's GCM AAD binds the fixed header plus the chunk index into the tag. After the final chunk, the stream verifies the file is exhausted, so a lowered total length (authenticated-prefix attack) is rejected. - Apply the 8 MiB chunk cap while the length is still a Long so exports between 2 and 4 GiB no longer wrap to a zero chunk size and divide by zero. - Reject a zero chunk size for non-empty plaintext. - Delete the staged bundle file on every readMigrationBundle failure path (wrong password, password-required, decrypt/manifest decode) so repeated failed attempts do not leak full bundle copies or a plaintext ZIP. - Track the in-flight preview job and guard the install with a sequence counter so a slow, superseded preview cannot overwrite the newer preview's staged file or UI state; the replaced staged file is deleted. - Add regression tests for framing tamper, ciphertext tamper, trailing data, negative total length, zero-chunk header, and a multi-chunk round trip.
A stale preview job suspended in the isEncryptedExport check can resume after a newer preview has started (job cancellation is cooperative). Its success branch updated the password prompt without a sequence check, so it could redirect the UI to the wrong import URI. Apply the same sequence guard used by the failure and post-preview paths.
Summary
Fixes the OOM crash in the Android data-export/import path by replacing the
single-shot GCM format with a chunked 0x03 variant and by streaming the
importer off an on-disk staged file instead of an in-memory
zipBytes.(
baseIv + chunk index),streamEncrypt/decryptStreamread and writechunk-by-chunk so peak memory is bounded to ~2x chunk size regardless of
bundle size. Legacy 0x02 bundles still decrypt (importer is dual-version).
importDatanow stages the bundle to a local plaintext ZIPunder
cacheDir/migration_import(reusing the preview's staged file whenpresent), parses the manifest in a single streaming
decodeFromStreampass, and streams attachments from the staged file.
PreviewWithDatacarries a
File, not aByteArray. Staged-file ownership is explicit:the importer deletes files it created; the ViewModel deletes the preview's
file;
cleanupStagedImports()clears leftovers at ViewModel init.MigrationCrypto.encryptFile); now emits 0x03.The previously-OOMing case (120 MiB export decrypting under a 90 MiB heap)
now passes.
Scope: Option A only. The export path still materializes the full
MigrationBundlein RAM before encryption (streaming export / ratchets-as-ZIP-entriesis a tracked follow-up, Option C).
Rebase note
Rebased onto current
main, which landed call-history migration work(
760734d5) that touched the same importer/exporter/data regions this PRrewrites. Adapted to the new API:
MigrationImporter.importDataInternalversion-8 guard: plainreturninstead of the old
return@withContextlabel (nowithContextscope left).MigrationCallHistoryImportTest:cachedZipBytes = zip(...)call sitesmigrated to
cachedZipFile = tempZipFile(zip(...))via a new helper thatwrites the ZIP to a temp file (matching the file-based import contract).
Verification (exact head)
:app:compileSentryKotlinBackendDebug{Kotlin,UnitTestKotlin}- clean:app:ktlintCheck+:app:detekt- clean:app:testSentryKotlinBackendDebugUnitTestformigration.*+MigrationViewModelTest- all pass (crypto 19 incl. 3 legacy 0x02 tests,importer-encryption 8, viewmodel 20, call-history import suite green)
Files
migration/MigrationCrypto.kt- 0x03 chunked-GCM + 0x02 legacy, streamingmigration/MigrationData.kt-PreviewWithDataholdsFilemigration/MigrationImporter.kt- staging, streaming, cleanup, API renameviewmodel/MigrationViewModel.kt-cachedImportFile, init cleanuptest/.../MigrationCallHistoryImportTest.kt- adapted tocachedZipFiletest/.../MigrationCryptoTest.kt- +3 legacy 0x02 compat teststest/.../MigrationViewModelTest.kt-PreviewWithData(File)stubs