Skip to content

feat(config): add arrayFunctionResultOverwritesData opt-in overwrite flag (HF-305) - #1714

Open
marcin-kordas-hoc wants to merge 6 commits into
developfrom
feat/hf-305-overwrite-flag
Open

feat(config): add arrayFunctionResultOverwritesData opt-in overwrite flag (HF-305)#1714
marcin-kordas-hoc wants to merge 6 commits into
developfrom
feat/hf-305-overwrite-flag

Conversation

@marcin-kordas-hoc

@marcin-kordas-hoc marcin-kordas-hoc commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Implements HF-305. Paired tests: hyperformula-tests#45 (branch feat/hf-305-overwrite-flag, same name — merge it first).

What & why

Adds an opt-in boolean config option arrayFunctionResultOverwritesData (default false).

  • false (default): unchanged behavior — an array function spilling onto an occupied cell yields #SPILL! and leaves the occupant intact.
  • true: the spill overwrites the occupied cells (clears occupants, spills the array, reroutes dependents) instead of emitting #SPILL!.

Requested by a customer; the feature + option name were pre-agreed (~8 SP).

How

Reuses the existing spill-placement exchange primitive rather than inventing a mechanism:

  • ConfigParams / Config: new option, mirrored on useArrayArithmetic.
  • DependencyGraph gains config + two guarded early-returns (exchangeOrAddFormulaVertex, setAddressMappingForArrayVertex).
  • Evaluator: recompute no longer emits #SPILL! when overwrite is allowed.

Array-vs-array safety: a spill colliding with another array always keeps #SPILL! and leaves that array intact, even in overwrite mode (canOverwriteArrayResult / overwriteWouldHitArray). Matches Excel; overwrite only clears static data. Default false guarantees no silent data loss.

Tests

They live in hyperformula-tests#45, unit/array-function-result-overwrites-data.spec.ts — 53 cases in nine groups: configuration parsing · disabled (default) parity · enabled overwrite incl. dependent reroute and 2-D blocks · array-vs-array collisions keeping #SPILL! · undo and redo · column-index staleness (incl. addColumns growth) · structural operations · cycles · unrepresentable spill ranges.

They arrived on this branch as test/hf-305-overwrite.spec.ts, in the public repository. Specs do not live here — #1612 took every test out of handsontable/hyperformula, which is why test/ on develop carries only the setup files, fetch-tests.sh and smoke.spec.ts. Jest's testMatch did pick the file up, so the suite was running; it was running from the wrong repository. Moved in f051ab3.

Verification of the move:

  • 53/53 green under Jest against this branch.
  • Patch coverage closed. Every executable line this PR adds to src/Operations.ts (29) and src/DependencyGraph/DependencyGraph.ts (17) is exercised by that spec alone — measured by intersecting this PR's added lines with a coverage run of the single file. That is what the red codecov/patch was reporting.
  • Not vacuous — four mutations, all killed: canOverwriteArrayResult forced to false → 22 failures; the snapshot guard flipped (<= 1>= 1) → 6; the EmptyCellVertex guard on the overwrite content-change inverted → 3; the array-vs-array guard removed → 3.

Known limitations

The opt-in overwrite behavior is intentionally scoped to the direct spill/recalc path. The following edges are not fully handled yet; each is a nuance to weigh, not a silent-corruption risk (the default false config is fully backward compatible and untouched by all of these):

  • buildFromArray initial-build path: declaring an array formula and a conflicting occupant in the same initial buildFromArray call still lets the occupant win — overwrite only applies on the recalc / setCellContents path (existing data, then an array formula lands on it), not the un-gated initial-build placement path.
  • Undo does not restore cells overwritten via direct setCellContentsno longer true, and this line was stale. setCellContent now returns { oldContent, overwrittenCells } from a pre-spill snapshot, and SetCellContentsUndoEntry restores them after the anchor formula is reverted. Pinned by eight undo/redo cases (value, formula, multiple occupants, rerouted dependent, redo, expand-over-static, blocked array-vs-array).
  • The undo leg of the structural-op path is still unpinned. When addColumns/addRows/removeColumns/removeRows shifts a dependency so an existing array grows over a previously-static cell, the overwrite itself is handled correctly for evaluation and for useColumnIndex (the stale value is dropped from the index at the moment of overwrite — see rewriteAffectedArrays), and both are pinned by tests. What no test covers either way is whether undoing that structural op restores the static cell. Measured for the direct-edit expand path — it does restore — but the structural path was not reproduced. This is the item for the reviewer to weigh, and it is narrower than the version this section previously described.
  • useColumnIndex + undo, on the structural path only: a workflow relying on undo() to roll back an overwrite that a structural op caused may see the array formula disappear while the previously-static data stays gone; the index and the data would then be consistent with each other but not with the pre-overwrite state. On the direct-edit path this no longer applies — undo restores both the cell and its index entry, and that is pinned.

None of these limitations can produce the crash or stale-lookup-index bugs originally flagged by Bugbot on this PR — both were reproduced with throwaway probes and fixed at the root cause (see review thread replies on commit b877ed4). What remains out of scope is exclusively the undo leg of overwrite bookkeeping.


Note

Medium Risk
Opt-in destructive sheet mutation in dependency graph and CRUD/undo paths; default false preserves Excel-compatible behavior, but enabled mode can clear live cell data and depends on correct column-index and undo bookkeeping.

Overview
Adds arrayFunctionResultOverwritesData (default false). When enabled, array formulas that would spill onto occupied non-array cells clear those cells and place the spill instead of returning #SPILL!. Collisions with another array or an unrepresentable spill range still yield #SPILL!; initial buildFromArray placement is unchanged (occupant wins).

Dependency graph: DependencyGraph now holds config and uses canOverwriteArrayResult / overwriteWouldHitArray so exchangeOrAddFormulaVertex can claim occupied spill cells, record content changes for the column index, and only then evaluate. The graph build path (setAddressMappingForArrayVertex) does not use overwrite mode.

CRUD & undo: setCellContent returns { oldContent, overwrittenCells } with a pre-spill snapshot of cleared occupants; SetCellContentsUndoEntry restores them after reverting the anchor formula. restoreCell / rewriteAffectedArrays apply column-index updates so stale lookup values are not left after overwrite or undo on the direct-edit path.

Reviewed by Cursor Bugbot for commit e410ed0. Bugbot is set up for automated code reviews on this repo. Configure here.

@qunabu

qunabu commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@netlify

netlify Bot commented Jul 22, 2026

Copy link
Copy Markdown

Deploy Preview for hyperformula-dev-docs ready!

Name Link
🔨 Latest commit 467c186
🔍 Latest deploy log https://app.netlify.com/projects/hyperformula-dev-docs/deploys/6a6602deeedd020008331aa9
😎 Deploy Preview https://deploy-preview-1714--hyperformula-dev-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch from 9009463 to b2da92e Compare July 22, 2026 16:18
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown

Performance comparison of head (e410ed0) vs base (114fd5d)

                                     testName |    base |    head | change
--------------------------------------------------------------------------
                                      Sheet A |   315.8 |  304.38 | -3.62%
                                      Sheet B |   99.44 |   98.39 | -1.06%
                                      Sheet T |   88.98 |   86.61 | -2.66%
                                Column ranges |   461.2 |  455.76 | -1.18%
                                Sorted lookup | 15566.6 | 16955.9 | +8.92%
Sheet A:  change value, add/remove row/column |    9.68 |    8.88 | -8.26%
 Sheet B: change value, add/remove row/column |   87.82 |    82.9 | -5.60%
                   Column ranges - add column |   108.3 |  109.81 | +1.39%
                Column ranges - without batch |  343.89 |  350.97 | +2.06%
                        Column ranges - batch |   89.55 |   91.17 | +1.81%

@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

bugbot run

Comment thread src/DependencyGraph/DependencyGraph.ts
@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch 2 times, most recently from 4c02791 to d2a2671 Compare July 23, 2026 03:53
@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

bugbot run

Comment thread src/Operations.ts
Comment thread src/DependencyGraph/DependencyGraph.ts Outdated
@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch from d2a2671 to 5e397d1 Compare July 24, 2026 11:14
@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 5e397d1. Configure here.

@marcin-kordas-hoc
marcin-kordas-hoc marked this pull request as ready for review July 25, 2026 10:58
@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch from 5e397d1 to 2f48fba Compare July 26, 2026 02:31
Comment thread src/UndoRedo.ts
@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch from 2f48fba to b877ed4 Compare July 26, 2026 02:43
@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

bugbot run

Comment thread src/DependencyGraph/DependencyGraph.ts
Comment thread src/Operations.ts
@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch from b877ed4 to 467c186 Compare July 26, 2026 12:51
@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 467c186. Configure here.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 28, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
hyperformula-docs e410ed0 Commit Preview URL

Branch Preview URL
Aug 31 2026, 07:37 AM

marcin-kordas-hoc and others added 3 commits August 28, 2026 06:23
…flag (HF-305)

Adds a public boolean Config option `arrayFunctionResultOverwritesData`
(default `false`). When `false`, behavior is unchanged: an array spill
onto an occupied cell yields `#SPILL!` and leaves the occupant intact.
When `true`, the spill overwrites the occupied cells (clears occupants,
spills the array, reroutes dependents) instead of emitting `#SPILL!`.

Implementation reuses the existing spill-placement exchange primitive:
- ConfigParams/Config: new option, mirrored on `useArrayArithmetic`.
- DependencyGraph gains `config` + two guarded early-returns in
  exchangeOrAddFormulaVertex / setAddressMappingForArrayVertex.
- Evaluator: recompute no longer emits `#SPILL!` when overwrite is allowed.

Array-vs-array safety: a spill colliding with ANOTHER array always keeps
`#SPILL!` and leaves that array intact, even in overwrite mode
(canOverwriteArrayResult / overwriteWouldHitArray). This matches Excel and
avoids corrupting a pre-existing array; overwrite only clears static data.

Default `false` guarantees no existing embedder loses data silently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 53 cases now live in hyperformula-tests on the same-name branch
(feat/hf-305-overwrite-flag), which is what test/fetch-tests.sh pairs on. They
arrived here as test/hf-305-overwrite.spec.ts, and #1612 took every spec out of
this repository: on develop, test/ carries only the setup files, fetch-tests.sh
and smoke.spec.ts. Jest's testMatch (test/**/*spec.(ts|js)) did pick the file up,
so the suite was running - it was running from the wrong repository, where it is
invisible to reviewers of the tests repo and ships in the public tree.

Coverage is not lost by the move: measured against this commit, every executable
line this PR adds to Operations.ts (29) and DependencyGraph.ts (17) is exercised
by the ported spec alone, which is what the red codecov/patch was about.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rebase onto develop surfaced this: the entry was written when the branch's
base still had 3.4.0 open, so replaying it landed the line inside the released
3.4.0 section. Moved to [Unreleased], and given the PR link every neighbouring
entry carries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@marcin-kordas-hoc
marcin-kordas-hoc force-pushed the feat/hf-305-overwrite-flag branch from f051ab3 to 63e2137 Compare August 28, 2026 06:27
Comment thread src/Operations.ts
The overwrite snapshot guarded on `width * height <= 1`, but placement
(FormulaVertex.fromAst) treats every ref-flavored size as a non-spilling
scalar via ArraySize.isScalar(). A bare range formula such as =C1:C3
therefore snapshotted neighbours it never overwrites, and undoing that
edit spuriously restored them (re-entering formulas, re-rolling
volatiles). Guard on size.isScalar() so the snapshot scope matches the
placement scope, as the JSDoc already described.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
marcin-kordas-hoc added a commit that referenced this pull request Aug 28, 2026
This morning's develop merge landed the TEXT/numberFormat entry inside
the already-published 3.4.0 Fixed list, putting CHANGELOG.md out of sync
with docs/guide/release-notes.md for that released version. The fix is
unreleased, so the entry belongs under [Unreleased]; the 3.4.0 lists in
the two files match again. Same correction as 63e2137 on #1714.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

Known-gap disclosure from writing the paired suite (tests #45), re-verified today on this head (3e2a099): with the flag ON, an array whose spill range cannot be represented silently loses #SPILL! and truncates to its anchor.

Repro: useArrayArithmetic: true, =2*(B:B) in A2 — flag OFF: #SPILL! (pinned by arrays.spec.ts); flag ON: A2 = 2, no error indicator anywhere. Same silent truncation on the buildFromArray / updateConfig path.

Root cause: overwriteWouldHitArray returns false for an undefined range, so canOverwriteArrayResult returns true and Evaluator skips setNoSpace(). Overwrite mode should bypass occupancy collisions only, not a range that cannot be spanned at all — suggested one-line fix: canOverwriteArrayResult returns false when arrayVertex.getRangeOrUndef() is undefined.

The paired suite deliberately pins only the stable half (occupants below are not clobbered), so it stays green whichever way this is resolved.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 08f5a06. Configure here.

Comment thread src/DependencyGraph/DependencyGraph.ts
…ill ranges

The overwrite feature resolves a spill collision when the array formula is
placed: exchangeOrAddFormulaVertex claims the whole spill range in the
address mapping before evaluation, so isThereSpaceForArray is already true
when the array is recomputed. The canOverwriteArrayResult bypass added to
the Evaluator (and to the build-path setAddressMappingForArrayVertex) could
therefore only ever fire for arrays whose overwrite never happened - an
array blocked by an occupant at buildFromArray/updateConfig time, or an
array whose spill range cannot be represented on the sheet (e.g. =2*(B:B)
anchored outside row 1, where getRangeOrUndef() is undefined and
overwriteWouldHitArray vacuously returns false). In both cases the array
silently truncated to its anchor value instead of yielding #SPILL!.

Remove the bypass from both sites so an array that does not own its spill
range keeps #SPILL! exactly like with the flag off, and make
canOverwriteArrayResult itself refuse unrepresentable ranges - overwrite
mode bypasses occupancy collisions only, never a range that cannot be
spanned at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@marcin-kordas-hoc

Copy link
Copy Markdown
Collaborator Author

Fixed in e410ed0f2 (tests 1116587, +4 cases → 60). The operative insight: every legitimate overwrite claims its spill range in the address mapping (exchangeOrAddFormulaVertex) before evaluation, so isThereSpaceForArray is already true when the array recomputes — the canOverwriteArrayResult bypass in Evaluator could only ever fire for arrays whose overwrite never happened, which is exactly the two silent-truncation paths. The bypass is removed; canOverwriteArrayResult additionally refuses an unrepresentable range as a defensive guard, and the build path keeps #SPILL! under the flag exactly as with it off. Fail-then-pass: the 3 new error assertions fail on the previous head, 60/60 after; changelog clause added. Verified independently on the pushed head: =2*(B:B) and the build-time collision both show #SPILL! with the flag on.

@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.72131% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.34%. Comparing base (114fd5d) to head (e410ed0).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
src/DependencyGraph/DependencyGraph.ts 90.00% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1714      +/-   ##
===========================================
+ Coverage    97.32%   97.34%   +0.01%     
===========================================
  Files          195      195              
  Lines        15739    15790      +51     
  Branches      3390     3402      +12     
===========================================
+ Hits         15318    15370      +52     
+ Misses         421      420       -1     
Files with missing lines Coverage Δ
src/Config.ts 94.17% <100.00%> (+0.05%) ⬆️
src/CrudOperations.ts 99.20% <100.00%> (+<0.01%) ⬆️
src/Evaluator.ts 100.00% <ø> (ø)
src/Operations.ts 98.98% <100.00%> (+0.04%) ⬆️
src/UndoRedo.ts 100.00% <100.00%> (ø)
src/DependencyGraph/DependencyGraph.ts 98.58% <90.00%> (-0.23%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants