Skip to content

fix(migration): chunked 0x03 GCM format to fix export OOM - #1128

Merged
torlando-tech merged 3 commits into
mainfrom
fix/migration-export-oom-v2
Sep 8, 2026
Merged

fix(migration): chunked 0x03 GCM format to fix export OOM#1128
torlando-tech merged 3 commits into
mainfrom
fix/migration-export-oom-v2

Conversation

@torlando-tech

Copy link
Copy Markdown
Owner

Summary

Fixes the OOM crash in the Android data-export/import path by replacing the
single-shot GCM format with a chunked 0x03 variant and by streaming the
importer off an on-disk staged file instead of an in-memory zipBytes.

  • Crypto (new 0x03 format): 8 MiB max chunks, per-chunk counter IVs
    (baseIv + chunk index), streamEncrypt/decryptStream read and write
    chunk-by-chunk so peak memory is bounded to ~2x chunk size regardless of
    bundle size. Legacy 0x02 bundles still decrypt (importer is dual-version).
  • Importer: importData now stages the bundle to a local plaintext ZIP
    under cacheDir/migration_import (reusing the preview's staged file when
    present), parses the manifest in a single streaming decodeFromStream
    pass, and streams attachments from the staged file. PreviewWithData
    carries a File, not a ByteArray. Staged-file ownership is explicit:
    the importer deletes files it created; the ViewModel deletes the preview's
    file; cleanupStagedImports() clears leftovers at ViewModel init.
  • Exporter: unchanged API surface (already routed through
    MigrationCrypto.encryptFile); now emits 0x03.

The previously-OOMing case (120 MiB export decrypting under a 90 MiB heap)
now passes.

Scope: Option A only. The export path still materializes the full
MigrationBundle in RAM before encryption (streaming export / ratchets-as-ZIP-entries
is a tracked follow-up, Option C).

Rebase note

Rebased onto current main, which landed call-history migration work
(760734d5) that touched the same importer/exporter/data regions this PR
rewrites. Adapted to the new API:

  • MigrationImporter.importDataInternal version-8 guard: plain return
    instead of the old return@withContext label (no withContext scope left).
  • MigrationCallHistoryImportTest: cachedZipBytes = zip(...) call sites
    migrated to cachedZipFile = tempZipFile(zip(...)) via a new helper that
    writes the ZIP to a temp file (matching the file-based import contract).

Verification (exact head)

  • :app:compileSentryKotlinBackendDebug{Kotlin,UnitTestKotlin} - clean
  • :app:ktlintCheck + :app:detekt - clean
  • :app:testSentryKotlinBackendDebugUnitTest for migration.* +
    MigrationViewModelTest - all pass (crypto 19 incl. 3 legacy 0x02 tests,
    importer-encryption 8, viewmodel 20, call-history import suite green)
  • 120 MiB export decrypt under a 90 MiB heap: passes

Files

  • migration/MigrationCrypto.kt - 0x03 chunked-GCM + 0x02 legacy, streaming
  • migration/MigrationData.kt - PreviewWithData holds File
  • migration/MigrationImporter.kt - staging, streaming, cleanup, API rename
  • viewmodel/MigrationViewModel.kt - cachedImportFile, init cleanup
  • test/.../MigrationCallHistoryImportTest.kt - adapted to cachedZipFile
  • test/.../MigrationCryptoTest.kt - +3 legacy 0x02 compat tests
  • test/.../MigrationViewModelTest.kt - PreviewWithData(File) stubs

Whole-file AES-GCM decryption buffers an entire ciphertext internally
(SunJCE), so importing a large encrypted export could OOM the app; the
export side held the full plaintext and ciphertext in memory.

Introduce a version 0x03 chunked-GCM format:
- one PBKDF2 key, 8 MiB plaintext chunks, per-chunk counter IVs
  (base IV XOR chunk index in bytes 4..11), per-chunk GCM tag
- encryptFile/encryptToFile stream in both directions (bounded memory)
- decryptFile/decryptStream decrypt chunk by chunk (~2x chunk peak)
- 0x02 legacy files still import (whole-file path, unchanged behavior)

Import now stages the bundle as a local plaintext ZIP in the app cache
instead of holding zipBytes in memory: preview copies the file (no
full read), the manifest parses via a streaming JSON decoder, and
import re-streams the staged file for attachments. The staged file is
owned by the caller and removed in finally/onCleared/init cleanup.

Verified: 120 MiB export decrypts under a 90 MiB heap in the
functional test; MigrationCryptoTest (19), MigrationImporterEncryptionTest
(8), and MigrationViewModelTest (20) all pass; ktlint + detekt clean.
@greptile-apps

greptile-apps Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR replaces whole-file migration encryption and import buffering with authenticated, chunked AES-GCM and disk-backed streaming.

  • Adds the authenticated 0x03 chunk format while retaining 0x02 import compatibility.
  • Stages imports in cache, streams manifest and attachment processing, and defines staged-file ownership and cleanup.
  • Tracks preview jobs and sequence IDs to prevent stale preview results and password prompts.
  • Adds crypto compatibility, framing-integrity, large-payload, importer, and ViewModel regression coverage.

Confidence Score: 5/5

The PR appears safe to merge; the previous findings are resolved and the latest sequence guard closes the stale password-prompt race.

No new actionable issue remains. Current code authenticates chunk framing, handles large-length arithmetic safely, rejects invalid zero-sized chunks, removes failed or superseded staged files, and prevents stale preview checks from updating current state. All previous root threads were manually resolved.

Important Files Changed

Filename Overview
app/src/main/java/network/columba/app/migration/MigrationCrypto.kt Implements authenticated chunked 0x03 encryption and bounded-memory decryption while preserving legacy 0x02 support.
app/src/main/java/network/columba/app/migration/MigrationImporter.kt Replaces in-memory bundles with staged files, streaming manifest and attachment reads, and explicit failure cleanup.
app/src/main/java/network/columba/app/viewmodel/MigrationViewModel.kt Manages staged-file ownership and prevents superseded preview jobs or encryption checks from updating current state.
app/src/main/java/network/columba/app/migration/MigrationData.kt Changes preview payload ownership from a ByteArray to a staged File.
app/src/test/java/network/columba/app/migration/MigrationCryptoTest.kt Adds legacy compatibility and authenticated-framing regression tests.
app/src/test/java/network/columba/app/migration/MigrationCallHistoryImportTest.kt Adapts call-history import coverage to the staged-file API.
app/src/test/java/network/columba/app/viewmodel/MigrationViewModelTest.kt Updates preview fixtures and initialization stubs for file-backed imports and cleanup.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  ExportZip[Plaintext export ZIP] --> ChunkEncrypt[Chunked AES-GCM encryption]
  ChunkEncrypt --> Encrypted03[Encrypted 0x03 bundle]
  Encrypted03 --> Stage[Stage bundle in cache]
  Legacy02[Legacy 0x02 bundle] --> Stage
  Stage --> Decrypt[Version-aware decryption]
  Decrypt --> StagedZip[Staged plaintext ZIP]
  StagedZip --> Manifest[Stream manifest]
  StagedZip --> Attachments[Stream attachments]
  Manifest --> Import[Import application data]
  Attachments --> Import
  Import --> Cleanup[Delete owned staged file]
Loading

Reviews (3): Last reviewed commit: "fix(migration): sequence-guard stale enc..." | Re-trigger Greptile

Comment thread app/src/main/java/network/columba/app/migration/MigrationCrypto.kt Outdated
Comment thread app/src/main/java/network/columba/app/migration/MigrationCrypto.kt Outdated
Comment thread app/src/main/java/network/columba/app/migration/MigrationCrypto.kt Outdated
@codecov

codecov Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…cycle

Address Greptile review feedback on the chunked export format:

- Authenticate 0x03 framing so the stream cannot be reordered, replayed,
  or truncated. Each chunk IV is now derived from (base IV, index) on the
  decrypt side instead of being read from the file, and each chunk's GCM
  AAD binds the fixed header plus the chunk index into the tag. After the
  final chunk, the stream verifies the file is exhausted, so a lowered
  total length (authenticated-prefix attack) is rejected.
- Apply the 8 MiB chunk cap while the length is still a Long so exports
  between 2 and 4 GiB no longer wrap to a zero chunk size and divide by
  zero.
- Reject a zero chunk size for non-empty plaintext.
- Delete the staged bundle file on every readMigrationBundle failure path
  (wrong password, password-required, decrypt/manifest decode) so repeated
  failed attempts do not leak full bundle copies or a plaintext ZIP.
- Track the in-flight preview job and guard the install with a sequence
  counter so a slow, superseded preview cannot overwrite the newer
  preview's staged file or UI state; the replaced staged file is deleted.
- Add regression tests for framing tamper, ciphertext tamper, trailing
  data, negative total length, zero-chunk header, and a multi-chunk
  round trip.
A stale preview job suspended in the isEncryptedExport check can resume
after a newer preview has started (job cancellation is cooperative). Its
success branch updated the password prompt without a sequence check, so it
could redirect the UI to the wrong import URI. Apply the same sequence
guard used by the failure and post-preview paths.
@torlando-tech
torlando-tech merged commit cffa02a into main Sep 8, 2026
13 checks passed
@torlando-tech
torlando-tech deleted the fix/migration-export-oom-v2 branch September 8, 2026 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant