Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(gatsby-transformer-json): Fix high memory consumption #34084

Merged
merged 1 commit into from
Nov 29, 2021
Merged

fix(gatsby-transformer-json): Fix high memory consumption #34084

merged 1 commit into from
Nov 29, 2021

Conversation

joernroeder
Copy link
Contributor

@joernroeder joernroeder commented Nov 26, 2021 •
edited by LekoArts
Loading

What changed with v4 is we now write data to LMDB which is more costly than writing to memory. The sync forEach loop doesn't allow LMDB to flush data to disk meaning in progress changes accumulate. Switching to async solves as does using e.g. process.nextTick to let the event loop run through again.

Closes #34081

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Nov 26, 2021
@LekoArts LekoArts added topic: plugins Related to plugin system, themes & catch-all for plugins that don't have a label and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Nov 29, 2021
LekoArts
LekoArts approved these changes Nov 29, 2021
View reviewed changes
Copy link
Contributor

@LekoArts LekoArts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@LekoArts LekoArts merged commit 2a94a48 into gatsbyjs:master Nov 29, 2021
@joernroeder joernroeder deleted the jr/fix/gatsby-transform-json-memory branch November 29, 2021 09:14
@KyleAMathews KyleAMathews mentioned this pull request Apr 27, 2022
Open
@ascott97 ascott97 mentioned this pull request Sep 14, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LekoArts LekoArts LekoArts approved these changes

Assignees
No one assigned
Labels
topic: plugins Related to plugin system, themes & catch-all for plugins that don't have a label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

gatsby-transform-json out of memory
2 participants
@joernroeder @LekoArts