Skip to content

fix(desktop): bound parallel JSON reads during import validation#631

Merged
r0ssing merged 1 commit into
OpenDataEnsemble:devfrom
najuna-brian:fix/import-json-bounded-reads
May 13, 2026
Merged

fix(desktop): bound parallel JSON reads during import validation#631
r0ssing merged 1 commit into
OpenDataEnsemble:devfrom
najuna-brian:fix/import-json-bounded-reads

Conversation

@najuna-brian
Copy link
Copy Markdown
Contributor

Description

Large drops of observation JSON files could trigger NotReadableError (Invalid JSON) when the UI read every file’s contents at once via File.text(). This change caps how many reads run in parallel so validation stays reliable on big batches.

What changed

  • desktop/src/lib/importSummary.ts Added mapPool (bounded concurrency, default 12) and routed parseObservationJsonFiles through it. parseObservationJsonFile still parses JSON and surfaces read failures; NotReadableError keeps a short hint to retry validation after load settles.
  • desktop/src/pages/ImportPage.tsx Validation continues to use parseObservationJsonFiles for staged JSON; wiring stays aligned with the helper above.

How to verify

  1. Build/run ODE Desktop, open Import, stage hundreds of observation .json files (plus attachments if you like).
  2. Run Validate (or the flow that parses staged JSON).
  3. Confirm no mass NotReadableError / bogus “invalid JSON” from read contention; failures should only reflect real read/parse issues.

Use mapPool in parseObservationJsonFiles so File.text() is not run
unbounded in parallel, avoiding NotReadableError on large drops.
ImportPage continues to call parseObservationJsonFiles for validate.
@najuna-brian najuna-brian requested a review from r0ssing May 13, 2026 13:09
Copy link
Copy Markdown
Contributor

@r0ssing r0ssing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!!

@r0ssing r0ssing merged commit b14bfb4 into OpenDataEnsemble:dev May 13, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants