Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(gatsby): do not cause stack overflow over circular refs #19802

Merged
merged 3 commits into from
Nov 26, 2019

Conversation

pvdz
Copy link
Contributor

@pvdz pvdz commented Nov 26, 2019

This allows nodes to have a circular dependency without causing stack overflow exceptions when traversing the graph.

Fixes #11364 (hopefully). Will have to confirm this after publishing an update.

Causes somewhere between 0 and 1% perf regression for our 25k page benchmark site. But it's within variance of benchmarking, not even sure if there's a noticable regression at all. See this table:

The first two columns are on master (at the time of writing), the third and fourth column are after this change. (The (govbook) lines are printed from within the benchmark site). Shrug.

    master:                |      branch:                  |
------------------------------------------------------------------------------------------------------
     0.025s        0.026s  |       0.026s         0.030s   |   success open and validate gatsby-configs 
     0.365s        0.361s  |       0.333s         0.362s   |   success load plugins 
     0.004s        0.004s  |       0.005s         0.005s   |   success onPreInit 
     0.012s        0.011s  |       0.011s         0.011s   |   success delete html and css files from previous builds 
     0.008s        0.008s  |       0.008s         0.008s   |   success initialize cache 
     0.025s        0.021s  |       0.018s         0.027s   |   success copy gatsby files 
     1.420s        1.530s  |       1.432s         1.655s   |   success onPreBootstrap 
     0.003s        0.004s  |       0.007s         0.006s   |   success createSchemaCustomization 
    11.587s       12.116s  |      12.048s        13.105s   |   success source and transform nodes 
     0.431s        0.491s  |       0.304s         0.332s   |   success building schema 
  560.858ms     591.782ms  |    617.005ms      668.053ms   |   (govbook) initial graphql query
    25.176s       27.344s  |      24.406s        27.665s   |   success govbook/gatsby-node.js 
25198.194ms   27369.563ms  |  24429.250ms    27689.333ms   |   (govbook) created pages
    5.316ms       5.221ms  |      5.142ms        5.059ms   |   (govbook) _wrapGraphql
   35.774ms      32.683ms  |     31.356ms       34.587ms   |   (govbook) _graphql
    2.906ms       3.225ms  |      2.761ms        3.011ms   |   (govbook) _createMarkdownPages
25819.979ms   28018.942ms  |  25102.183ms    28422.466ms   |   (govbook) total exports.createPages
   104.950s      110.702s  |     105.509s       112.224s   |   success createPages 
     0.399s        0.175s  |       0.383s         0.387s   |   success createPagesStatefully 
     0.002s        0.002s  |       0.002s         0.002s   |   success onPreExtractQueries 
     0.316s        0.181s  |       0.356s         0.306s   |   success update schema 
     0.340s        0.146s  |       0.359s         0.319s   |   success extract queries from components 
     0.831s        0.545s  |       0.951s         0.825s   |   success write out requires 
     0.002s        0.002s  |       0.002s         0.002s   |   success write out redirect data 
     0.124s        0.095s  |       0.125s         0.119s   |   success Build manifest and related icons 
     2.233s        1.516s  |       2.303s         2.193s   |   success onPostBootstrap 
   125.512s      130.371s  |     126.555s       134.379s   |   info bootstrap finished 
    13.429s       12.381s  |      14.701s        12.956s   |   success Building production JavaScript and CSS bundles 
     0.005s        0.005s  |       0.008s         0.008s   |   success Rewriting compilation hashes 
    85.845s       95.184s  |         30/s        91.392s   |   success run queries 
    24.421s       27.864s  |      25.243s        25.466s   |   success Building static HTML for pages 
   237.088s      254.554s  |      245.77s       252.497s   |   info Done building 

@pvdz pvdz requested a review from a team as a code owner November 26, 2019 12:54
@pvdz pvdz changed the title Circular nodes fix(gatsby): do not cause stack overflow over circular refs Nov 26, 2019
@pvdz
Copy link
Contributor Author

pvdz commented Nov 26, 2019

Thanks @cameron-martin for the inspiration!

}) => {
const updateValueDescriptor = (
{ nodeId, key, value, operation = `add` /*: add | del*/, descriptor = {} },
path = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other places use Set for keeping track if already visited things, was there special reason to use array here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I benchmarked it, this seemed more efficient (although it could be variation). Perhaps because this is an explicit push/pop stack, where the other is arbitrary push.

This allows nodes to have a circular dependency without causing stack overflow exceptions when traversing the graph.

Fixes #11364 (hopefully). Will have to confirm this after publishing an update.

Causes somewhere between 0 and 1% perf regression for our 25k page benchmark site.
@pvdz
Copy link
Contributor Author

pvdz commented Nov 26, 2019

Rebased

Copy link
Contributor

@sidharthachatterjee sidharthachatterjee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done! Thanks @pvdz 🥇

@sidharthachatterjee sidharthachatterjee added the bot: merge on green Gatsbot will merge these PRs automatically when all tests passes label Nov 26, 2019
@gatsbybot gatsbybot merged commit 89c6b89 into master Nov 26, 2019
@delete-merged-branch delete-merged-branch bot deleted the circular-nodes branch November 26, 2019 22:42
pragmaticpat pushed a commit to pragmaticpat/gatsby that referenced this pull request Apr 28, 2022
…#19802)

* fix(gatsby): do not cause stack overflow over circular refs

This allows nodes to have a circular dependency without causing stack overflow exceptions when traversing the graph.

Fixes gatsbyjs#11364 (hopefully). Will have to confirm this after publishing an update.

Causes somewhere between 0 and 1% perf regression for our 25k page benchmark site.

* Use actual `if` for conditionals

* Only add data to the path when actually an object
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bot: merge on green Gatsbot will merge these PRs automatically when all tests passes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

gatsby-source-contentful: Maximum call stack size exceeded
4 participants