Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(Google Sheets Node): Fix "Maximum call stack size exceeded" error on too many rows #7384

Merged
merged 1 commit into from Oct 9, 2023
Merged

fix(Google Sheets Node): Fix "Maximum call stack size exceeded" error on too many rows #7384

merged 1 commit into from Oct 9, 2023

Conversation

netroy
Copy link
Member

@netroy netroy commented Oct 9, 2023 •
edited

@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2023

Great PR! Please pay attention to the following items before merging:

Files matching packages/**:

  • If fixing bug, added test to cover scenario.
  • If addressing forum or Github issue, added link to description.

Files matching packages/**/*.ts:

  • Added unit tests to cover new or updated functionality.

Files matching packages/nodes-base/nodes/**:

  • Added workflow tests for nodes if possible.

Make sure to check off this list before asking for review.

@netroy netroy force-pushed the fix-max-call-stack-google-sheets branch from 064d348 to 7cdff07 Compare October 9, 2023 14:14
@n8n-assistant n8n-assistant bot added n8n team Authored by the n8n team node/improvement New feature or request labels Oct 9, 2023
netroy
netroy commented Oct 9, 2023
View reviewed changes
packages/nodes-base/nodes/Google/Sheet/v2/actions/router.ts
results = await spreadsheet[googleSheets.operation].execute.call(this);
}
if (results?.length) {
operationResult = operationResult.concat(results);
Copy link
Member Author

@netroy netroy Oct 9, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a small benchmark comparing .concat, pushing items from results into operationResult one at a time, pushing in batches of 1k, and pushing in batches of 100k.
.concat was consistently many times faster, while also produced the least amount of garbage for the GC to worry about.

benchmark

@codecov
Copy link

codecov bot commented Oct 9, 2023 •
edited

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (c5ee06c) 33.42% compared to head (7cdff07) 33.42%.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #7384   +/-   ##
=======================================
  Coverage   33.42%   33.42%           
=======================================
  Files        3389     3389           
  Lines      206610   206612    +2     
  Branches    22281    22286    +5     
=======================================
+ Hits        69050    69056    +6     
+ Misses     136443   136439    -4     
  Partials     1117     1117
Files Coverage Δ
...nodes-base/nodes/Google/Sheet/GoogleSheets.node.ts 0.00% <ø> (ø)
...nodes-base/nodes/Google/Sheet/v2/actions/router.ts 0.00% <0.00%> (ø)

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cypress
Copy link

cypress bot commented Oct 9, 2023

1 flaky test on run #2438 ↗︎

0 250 0 0 Flakiness 1

Details:

🌳 fix-max-call-stack-google-sheets 🖥️ browsers:node18.12.0-chrome107 🤖 netroy...
Project: n8n Commit: 7cdff0796a
Status: Passed Duration: 07:59 💡
Started: Oct 9, 2023 2:46 PM Ended: Oct 9, 2023 2:54 PM
Flakiness  cypress/e2e/2-credentials.cy.ts • 1 flaky test

View Output Video

Test Artifacts
Credentials > should create a new credential using Add Credential button Output Screenshots Video

Review all test suite changes for PR #7384 ↗︎

@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2023

✅ All Cypress E2E specs passed

@netroy netroy merged commit 732b15a into master Oct 9, 2023
57 of 58 checks passed
@netroy netroy deleted the fix-max-call-stack-google-sheets branch October 9, 2023 14:58
netroy added a commit that referenced this pull request Oct 9, 2023
@netroy
fix(Google Sheets Node): Fix "Maximum call stack size exceeded" error…
200fb54 
… on too many rows (#7384)

Fixes:

1. https://community.n8n.io/t/google-sheets-error-maximum-call-stack-size-exceeded/31006
2. https://community.n8n.io/t/error-maximum-call-stack-size-exceeded-in-google-sheet-read-rows/20307
@github-actions github-actions bot mentioned this pull request Oct 9, 2023
Merged
netroy added a commit that referenced this pull request Oct 9, 2023
@github-actions @netroy
🚀 Release 1.9.2 (#7387)
4777333 
## [1.9.2](https://github.com/n8n-io/n8n/compare/n8n@1.9.1...n8n@1.9.2)
(2023-10-09)


### Bug Fixes

* **core:** Add an option to enable postgres ssl with default certs
([#6889](#6889))
([0de6d7d](0de6d7d))
* **editor:** Fix disappearing NDV header in code nodes
([#7290](#7290))
([7d87392](7d87392))
* **editor:** Remove excess margin below run data editor
([#7372](#7372))
([8c38984](8c38984))
* **editor:** Use display option's @Version specifier
([#7351](#7351))
([73a98cb](73a98cb))
* **Google BigQuery Node:** Location default to jobReference
([#7354](#7354))
([489a12d](489a12d))
* **Google Drive Trigger Node:** Add Shared Drives support
([#7369](#7369))
([138d26c](138d26c))
* **Google Sheets Node:** Fix "Maximum call stack size exceeded" error
on too many rows ([#7384](#7384))
([200fb54](200fb54))
* **HTML Node:** Update property fields to not use expressions on drag
([#7379](#7379))
([4517001](4517001))

Co-authored-by: netroy <netroy@users.noreply.github.com>
@janober
Copy link
Member

janober commented Oct 10, 2023

Got released with n8n@1.9.3

MiloradFilipovic added a commit that referenced this pull request Oct 10, 2023
@MiloradFilipovic
Merge branch 'master' into ADO-1148-remote-options-fix
c5cc371 
* master: (42 commits)
  fix(editor): Make workflow history button available only for dev builds (#7392)
  refactor: Upgrade to pnpm 8.9 (no-changelog) (#7393)
  ci: Identify NPM releases as `stable` (no-changelog)
  fix(editor): Implement canvas zoom UX improvements (#7376)
  feat(core): Switch binary filesystem mode to nested path structure (#7307)
  fix(editor): Fix completions for `.json` on quoted node name in Code node (#7382)
  fix(Notion Node): Handle empty values correctly for Notion selects + multi selects (#7383)
  feat(editor): Make PDF and Audio binary-data viewable in the UI (#7367)
  fix(Google Sheets Node): Fix "Maximum call stack size exceeded" error on too many rows (#7384)
  refactor(core): Refactor nodes loading (no-changelog) (#7283)
  fix(core): Add an option to enable postgres ssl with default certs (#6889)
  feat(editor): Workflow history [WIP]- Add restore and clone into new workflow actions (no-changelog) (#7359)
  fix(HTML Node): Update property fields to not use expressions on drag (#7379)
  fix: Add role check for upgrade path (#7374)
  fix(editor): Remove excess margin below run data editor (#7372)
  fix(Google Drive Trigger Node): Add Shared Drives support (#7369)
  feat(core): Add Job Summary to Worker response (#7360)
  feat(Execute Workflow Node): Run once for each item mode (#7289)
  refactor(core): Move `copyInputItems` to node helpers (no-changelog) (#7299)
  refactor(core): Create controller for binary data (no-changelog) (#7363)
  ...

# Conflicts:
#	cypress/e2e/5-ndv.cy.ts
@github-actions github-actions bot mentioned this pull request Oct 11, 2023
Merged
netroy added a commit that referenced this pull request Oct 11, 2023
@github-actions @netroy
🚀 Release 1.11.0 (#7402)
e6ba841 
# [1.11.0](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.11.0)
(2023-10-11)


### Bug Fixes

* **core:** Add an option to enable postgres ssl with default certs
([#6889](#6889))
([789e1e7](789e1e7))
* **core:** Fix error on missing paired item data
([#7399](#7399))
([47e8953](47e8953))
* **core:** Missing pairing info
([#7326](#7326))
([e2c3c7a](e2c3c7a))
* **core:** Prevent object deletion request on no prefix match
([#7366](#7366))
([63e11e4](63e11e4))
* **editor:** Fix completions for `.json` on quoted node name in Code
node ([#7382](#7382))
([86e7ec7](86e7ec7))
* **editor:** Implement canvas zoom UX improvements
([#7376](#7376))
([7e06b31](7e06b31))
* **editor:** Make workflow history button available only for dev builds
([#7392](#7392))
([7ed466d](7ed466d))
* **editor:** Remove excess margin below run data editor
([#7372](#7372))
([3fa2764](3fa2764))
* **editor:** Sanitize HTML binary-data before rendering in the UI
([#7400](#7400))
([2b075bf](2b075bf))
* **editor:** Use display option's @Version specifier
([#7351](#7351))
([afbf0c3](afbf0c3))
* **Google BigQuery Node:** Location default to jobReference
([#7354](#7354))
([97bb703](97bb703))
* **Google Drive Trigger Node:** Add Shared Drives support
([#7369](#7369))
([3e7a4d3](3e7a4d3))
* **Google Sheets Node:** Fix "Maximum call stack size exceeded" error
on too many rows ([#7384](#7384))
([732b15a](732b15a))
* **HTML Node:** Update property fields to not use expressions on drag
([#7379](#7379))
([77643e5](77643e5))
* **Notion Node:** Handle empty values correctly for Notion selects +
multi selects ([#7383](#7383))
([fbcd1d4](fbcd1d4))
* **Set Node:** Increase search priority
([#7358](#7358))
([e5ad1e7](e5ad1e7))
* **Webhook Node:** Backward compatible form-data parsing for non-array
files ([#7385](#7385))
([6479eb1](6479eb1))


### Features

* **core:** Add Job Summary to Worker response
([#7360](#7360))
([b8608ce](b8608ce))
* **core:** Integrate object store as binary data manager
([#7253](#7253))
([1a661e6](1a661e6))
* **core:** Switch binary filesystem mode to nested path structure
([#7307](#7307))
([0847623](0847623))
* **editor:** Make PDF and Audio binary-data viewable in the UI
([#7367](#7367))
([8187be1](8187be1))
* **editor:** Support autologin for upgrade path
([#7316](#7316))
([1dfa052](1dfa052))
* **Execute Workflow Node:** Run once for each item mode
([#7289](#7289))
([c8c14ca](c8c14ca))
* **Item Lists Node:** Split merge binary data
([#7297](#7297))
([965db8f](965db8f))
* **Loop Over Items (Split in Batches) Node:** Automatically add a loop
+ rename ([#7228](#7228))
([7b773cc](7b773cc))
* **Notion Node:** Fetch child blocks recursively
([#7304](#7304))
([193181a](193181a))

Co-authored-by: netroy <netroy@users.noreply.github.com>
elsmr pushed a commit that referenced this pull request Oct 19, 2023
@github-actions @netroy @elsmr
🚀 Release 1.11.0 (#7402)
f10b5f0 
# [1.11.0](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.11.0)
(2023-10-11)


### Bug Fixes

* **core:** Add an option to enable postgres ssl with default certs
([#6889](#6889))
([789e1e7](789e1e7))
* **core:** Fix error on missing paired item data
([#7399](#7399))
([47e8953](47e8953))
* **core:** Missing pairing info
([#7326](#7326))
([e2c3c7a](e2c3c7a))
* **core:** Prevent object deletion request on no prefix match
([#7366](#7366))
([63e11e4](63e11e4))
* **editor:** Fix completions for `.json` on quoted node name in Code
node ([#7382](#7382))
([86e7ec7](86e7ec7))
* **editor:** Implement canvas zoom UX improvements
([#7376](#7376))
([7e06b31](7e06b31))
* **editor:** Make workflow history button available only for dev builds
([#7392](#7392))
([7ed466d](7ed466d))
* **editor:** Remove excess margin below run data editor
([#7372](#7372))
([3fa2764](3fa2764))
* **editor:** Sanitize HTML binary-data before rendering in the UI
([#7400](#7400))
([2b075bf](2b075bf))
* **editor:** Use display option's @Version specifier
([#7351](#7351))
([afbf0c3](afbf0c3))
* **Google BigQuery Node:** Location default to jobReference
([#7354](#7354))
([97bb703](97bb703))
* **Google Drive Trigger Node:** Add Shared Drives support
([#7369](#7369))
([3e7a4d3](3e7a4d3))
* **Google Sheets Node:** Fix "Maximum call stack size exceeded" error
on too many rows ([#7384](#7384))
([732b15a](732b15a))
* **HTML Node:** Update property fields to not use expressions on drag
([#7379](#7379))
([77643e5](77643e5))
* **Notion Node:** Handle empty values correctly for Notion selects +
multi selects ([#7383](#7383))
([fbcd1d4](fbcd1d4))
* **Set Node:** Increase search priority
([#7358](#7358))
([e5ad1e7](e5ad1e7))
* **Webhook Node:** Backward compatible form-data parsing for non-array
files ([#7385](#7385))
([6479eb1](6479eb1))


### Features

* **core:** Add Job Summary to Worker response
([#7360](#7360))
([b8608ce](b8608ce))
* **core:** Integrate object store as binary data manager
([#7253](#7253))
([1a661e6](1a661e6))
* **core:** Switch binary filesystem mode to nested path structure
([#7307](#7307))
([0847623](0847623))
* **editor:** Make PDF and Audio binary-data viewable in the UI
([#7367](#7367))
([8187be1](8187be1))
* **editor:** Support autologin for upgrade path
([#7316](#7316))
([1dfa052](1dfa052))
* **Execute Workflow Node:** Run once for each item mode
([#7289](#7289))
([c8c14ca](c8c14ca))
* **Item Lists Node:** Split merge binary data
([#7297](#7297))
([965db8f](965db8f))
* **Loop Over Items (Split in Batches) Node:** Automatically add a loop
+ rename ([#7228](#7228))
([7b773cc](7b773cc))
* **Notion Node:** Fetch child blocks recursively
([#7304](#7304))
([193181a](193181a))

Co-authored-by: netroy <netroy@users.noreply.github.com>
@netroy netroy mentioned this pull request Jan 15, 2024
Merged
1 task
netroy added a commit that referenced this pull request Feb 1, 2024
@netroy
fix(MongoDB Node): Fix "Maximum call stack size exceeded" error on to…
29b7c71 
…o many rows

Previous similar fixes:

* Google Sheets Node #7384
* Microsoft SQL Node #8334
@netroy netroy mentioned this pull request Feb 1, 2024
Merged
1 task
@netroy netroy mentioned this pull request Feb 16, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michael-radency michael-radency michael-radency approved these changes

Assignees
No one assigned
Labels
n8n team Authored by the n8n team node/improvement New feature or request Released
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@netroy @janober @michael-radency