Skip to content

perf: add low-overhead row decode APIs for Node.js#48

Merged
Nebu1eto merged 3 commits into
mainfrom
perf/phase3-node-decode
Feb 11, 2026
Merged

perf: add low-overhead row decode APIs for Node.js#48
Nebu1eto merged 3 commits into
mainfrom
perf/phase3-node-decode

Conversation

@Nebu1eto

Copy link
Copy Markdown
Owner

Summary

  • Add column name cache to existing decodeRowsBuffer() eliminating redundant string computations
  • Add getRowsRaw() API returning typed arrays instead of JS objects (zero per-cell object allocation)
  • Add getRowsIterator() generator for lazy row-at-a-time processing without full materialization

Performance Impact

  • Column name cache: Reduces O(rows * cols) string computations to O(cols) unique lookups
  • getRowsRaw(): For 50k x 20 sheet, creates 7 typed arrays instead of 50,000 JsRowData + 1,000,000 JsRowCell objects
  • getRowsIterator(): Avoids holding all rows in memory simultaneously
  • Target: Node Read Multi-Sheet >= 10% improvement

New APIs

// Typed-array based (zero object allocation)
getRowsRaw(sheet: string): RawRowsResult

// Generator-based (lazy, streaming)
*getRowsIterator(sheet: string): Generator<JsRowData>

Changes

  • packages/sheetkit/buffer-codec.ts: Column cache, decodeRowsRawBuffer(), decodeRowsIterator(), shared parseBuffer()
  • packages/sheetkit/index.ts: getRowsRaw(), getRowsIterator() methods + RawRowsResult export
  • packages/sheetkit/__test__/index.spec.ts: 22 new tests across 4 describe blocks

Test plan

  • 22 new tests (getRowsRaw, decodeRowsRawBuffer, getRowsIterator, decodeRowsIterator)
  • All 486 existing tests pass (includes 115 Node.js tests + Rust tests)
  • Cross-validation: getRowsRaw results match getRows for all cell types
  • No Rust changes required - pure JS/TS optimization

🤖 Generated with Claude Code

Optimize the existing decodeRowsBuffer() with a column name cache that
avoids recalculating column letters on every cell of every row. Add two
new APIs for performance-sensitive workloads:

- getRowsRaw() / decodeRowsRawBuffer(): returns cell data in flat typed
  arrays (Uint32Array, Float64Array, etc.) instead of JsRowData objects,
  eliminating per-cell/per-row object allocation entirely.

- getRowsIterator() / decodeRowsIterator(): generator that yields one
  JsRowData at a time without materializing the entire array, reducing
  peak memory usage for sequential row processing.

Includes 22 new tests covering all cell types, sparse data, large
datasets, consistency with existing getRows(), and edge cases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude

claude Bot commented Feb 11, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@Nebu1eto

Copy link
Copy Markdown
Owner Author

Review finding (performance regression): column-name cache prewarm penalizes sparse/wide sheets.

decodeRowsBuffer() and decodeRowsIterator() now precompute all column names up to maxCol before decoding:

  • packages/sheetkit/buffer-codec.ts (cachedColumnName(maxCol) in decodeRowsBuffer)
  • packages/sheetkit/buffer-codec.ts (cachedColumnName(maxCol) in decodeRowsIterator)

For sparse buffers, this changes complexity from roughly O(non-empty-cells) column-name work to O(colCount). In this format, colCount is the full bounding range width (from minCol to maxCol), not the number of populated columns.

I reproduced this on a sparse wide sheet (A1, XFD1):

  • main decodeRowsBuffer first call: ~0.435 ms
  • this PR decodeRowsBuffer first call: ~1.599 ms
  • follow-up call is faster due to warmed global cache, but first-call latency regresses by ~3.7x.

This also weakens iterator behavior: decodeRowsIterator() now does non-trivial up-front work before first next(), which is opposite to expected lazy/streaming ergonomics.

Suggested fix:

  • Only prewarm for dense layout (or cap prewarm to a small threshold).
  • For sparse layout, resolve column names lazily per encountered cell using the cache-backed function.

Nebu1eto and others added 2 commits February 12, 2026 00:24
Only pre-warm the column name cache for dense layout where every column
will be visited. For sparse layout, column names are now resolved lazily
per encountered cell, restoring O(non-empty-cells) complexity instead of
O(colCount). This also fixes decodeRowsIterator() doing non-trivial
upfront work before first next(), preserving lazy/streaming semantics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Nebu1eto
Nebu1eto merged commit 3a4246f into main Feb 11, 2026
7 checks passed
@Nebu1eto
Nebu1eto deleted the perf/phase3-node-decode branch February 11, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant