perf: bound import RAM with a per-import total-byte budget#70
Merged
Conversation
The Whoop and Wearable export importers each collect every recognised entry into a [String: Data] held entirely in memory, guarded only by a PER-ENTRY ceiling (maxEntryBytes). Nothing bounded the SUM of the retained set: the Wearable path can carry up to maxFiles (200k) entries, so a large or crafted export could accumulate unbounded Data before the parser ever runs. Add an injectable maxTotalBytes budget (default 1 GB) and enforce it in both importers' zip AND folder loaders: once adding the next retained entry would exceed the budget, collection stops. Peak RAM is now bounded regardless of entry count. Real exports (a few MB Whoop / tens of MB Wearable) never approach it. Why a byte budget and not temp-file streaming (as AppleHealthImporter does): AppleHealth streams because its XML parser reads incrementally from the file. These importers instead parse in-memory Data (CSV tables / JSON), and brand-detection needs the whole file set together — so the bytes must live in RAM to be parsed, and the correct RAM bound is a cap on their sum, not a spill to disk (which you'd only read back). Adds ImportByteBudgetTests: a cap admitting only the first entry drops the rest (Whoop end-to-end through parse; Wearable via collectFiles), and the retained bytes stay within budget. Full StrandImport suite green (170 tests).
ryanbr
added a commit
that referenced
this pull request
Jul 9, 2026
… parity) Android twin of the iOS StrandImport change in PR #70. WhoopCsvImporter and WearableExportImporter each collect every recognised entry into an in-memory Map<String, ByteArray> guarded only by a per-entry cap (256 MB) and, for Wearable, a per-COUNT cap (200k). Nothing bounded the SUM of the retained set -- a large or crafted export (the Wearable path can carry up to 200k entries) could accumulate unbounded ByteArray in the map before parsing ever runs. Add an aggregate MAX_TOTAL_BYTES budget (1 GB) enforced in both importers' zip loaders: once adding the next retained entry would push the running total past the budget, collection stops. Real exports (a few MB Whoop, tens of MB Wearable) never approach it, so this is behaviour-preserving. Matches the Swift maxTotalBytes. Improvement over #70: not silent. #70 (like the pre-existing MAX_FILES cap) stops collecting with a bare break. Here the loaders return a `truncated` flag and the ImportSummary message says "(partial -- export exceeded the 1 GB import memory budget)" when it trips, so a truncated import is diagnosable instead of reading as a clean import over incomplete data. Also: the Wearable zip loader now dedups first-wins (was last-wins overwrite), which keeps the running total exact and matches the Swift zip loader's guard. Testability: extracted WearableExportImporter.collectZipEntries as an InputStream seam (same pattern as AppleHealthImporter.parseStreamForTest) so a JVM unit test drives the budget from an in-memory zip -- no ContentResolver needed. Verified locally: :app:compileFullDebugKotlin + the new ImportByteBudgetTest + the full com.noop.ingest test package all pass.
ryanbr
added a commit
that referenced
this pull request
Jul 9, 2026
Android import: bound peak RAM with a per-import total-byte budget (#70 parity)
digitalerdude
pushed a commit
to digitalerdude/noop
that referenced
this pull request
Jul 9, 2026
New raw-SpO2 readout (ryanbr#93), unit/display fixes (skin temp degF ryanbr#101, Effort scale ryanbr#45), the workout-in-progress card restored on Home (ryanbr#105), Apple Health steps counted (ryanbr#89), steadier battery alerts (ryanbr#80/ryanbr#99/ryanbr#114), clearer overnight-HRV copy (ryanbr#92), a lighter/faster app (ryanbr#128/ryanbr#129/ryanbr#70/ryanbr#125), and honest diagnostics (Oura anchor ryanbr#91, Test Centre sleep capture ryanbr#127). - docs/releases/v8.5.0.md (release notes) - android build.gradle.kts: versionCode 267->268, versionName 8.4.0->8.5.0 - project.yml: MARKETING_VERSION 8.4.0->8.5.0, CURRENT_PROJECT_VERSION 178->179 - in-app What's New: AppChangelog.swift + AppChangelog.kt 8.5.0 entry (parallel) Android :app:compileFullDebugKotlin passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The Whoop and Wearable export importers each collect every recognised entry into a
[String: Data]held entirely in memory, guarded only by a per-entry ceiling (maxEntryBytes= 256 MB). Nothing bounded the sum of the retained set — and the Wearable path can carry up tomaxFiles= 200 000 entries. A large (or crafted) export could accumulate unboundedDatain the result dict before the parser ever runs.The change
Add an injectable
maxTotalBytesbudget (default 1 GB) and enforce it in both importers, in both the zip and folder loaders: once adding the next retained entry would push the running total past the budget, collection stops. Peak import RAM is now bounded regardless of how many entries the archive holds. Real exports — a few MB (Whoop) to tens of MB (Wearable) — never approach it.Why a byte budget, not temp-file streaming
AppleHealthImporterstreams its single bigexport.xmlto a temp file — but that works only because its XML parser reads incrementally from disk. These two importers instead parse in-memoryData(CSV tables / JSON), and brand-detection needs the whole file-set together. So the bytes have to be in RAM to be parsed at all; spilling them to disk would just mean reading them back. The correct RAM bound here is therefore a cap on their sum, which is what this adds.Verification
ImportByteBudgetTests: a cap admitting only the first entry drops the rest — Whoop end-to-end through parse, Wearable viacollectFiles— and the retained bytes stay within budget. (ZIPFoundation preserves entry order, so the outcome is deterministic.)WhoopExportImporterTests(20),WearableExportImporterTests(15),ImportCoordinatorTests(9) all green.StrandImportpackage suite green — 170 tests, 0 failures (swift test).Checklist
swift testgreen (StrandImport, 170)Perf/robustness pass, AI-assisted; budget behavior tested locally with tiny injected caps.