Upgrade from gatsby@1.9.123 to gatsby@1.9.127 seemingly breaks JSON parsing when there's a significant amount of JSON

[ghost]

Preamble

Gatsby version: 1.9.127
node.js version: v8.9.1
OS version: Linux version 4.13.16-35.current (Solus OS)

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `someTitle`,
    buildID: '22166834-86c7-4c26-9770-afd9b2889e09'
  },
  plugins: [
    `gatsby-plugin-react-helmet`,
    `gatsby-plugin-styled-components`,
    `gatsby-plugin-react-next`,
  ],
  pathPrefix: '/e2fd2e20-0e66-4c28-b503-e841e8f04769'
}

package.json:

{
  "name": "someName"
  "description": "some landing pages"
  "version": "1.0.0",
  "author": ""
  "dependencies": {
    "gatsby": "^1.9.73",
    "gatsby-link": "^1.6.22",
    "gatsby-plugin-react-helmet": "^1.0.8",
    "gatsby-plugin-react-next": "^1.0.4",
    "moment": "^2.19.1",
    "query-string": "^5.0.1",
    "react-redux": "^5.0.6",
    "rebass": "^1.0.4",
    "redux": "^3.7.2",
    "uuid": "^3.1.0"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "main": "n/a",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --trailing-comma es5 --no-semi --single-quote --write 'src/**/*.js'",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "devDependencies": {
    "gatsby-plugin-styled-components": "^1.0.5",
    "prettier": "^1.7.4"
  }
}

The issue

In development mode, some pages with long JSON strings as variables throw this error before successfully recompiling (after which the pages can be successfully rendered in-browser):

$ gatsby develop
success delete html files from previous builds — 0.027 s
success open and validate gatsby-config.js — 0.004 s
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.005 s
success source and transform nodes — 0.011 s
success building schema — 0.060 s
success createLayouts — 0.015 s
success createPages — 0.004 s
success createPagesStatefully — 0.146 s
success onPreExtractQueries — 0.001 s
success update schema — 0.054 s
success extract queries from components — 0.083 s
success run graphql queries — 0.034 s
success write out page data — 0.008 s
success write out redirect data — 0.000 s
success onPostBootstrap — 0.000 s
info bootstrap finished - 1.401 s
>  ERROR  Failed to compile with 2 errors                                                                                              18:40:46
 error  in ./.cache/json/district-8055.json
Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
 @ ./.cache/sync-requires.js 145:22837-22947
 error  in ./.cache/json/district-8056.json
Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
 @ ./.cache/sync-requires.js 145:23135-23245
 WAIT  Compiling...                                                                                                                  18:40:46
 DONE  Compiled successfully in 389ms

When trying to create the production build, this error is thrown, causing the whole build to fail:

$ gatsby build
success delete html files from previous builds — 0.030 s
success open and validate gatsby-config.js — 0.004 s
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.007 s
success source and transform nodes — 0.013 s
success building schema — 0.090 s
success createLayouts — 0.018 s
success createPages — 0.001 s
success createPagesStatefully — 0.150 s
success onPreExtractQueries — 0.001 s
success update schema — 0.059 s
success extract queries from components — 0.041 s
success run graphql queries — 0.021 s
success write out page data — 0.006 s
success write out redirect data — 0.000 s
success onPostBootstrap — 0.000 s
info bootstrap finished - 1.342 s
success Building CSS — 4.001 s
success Building production JavaScript bundles — 11.812 s
error Building static HTML for pages failed
See our docs page on debugging HTML builds for help https://goo.gl/yL9lND
  Error: Module build failed: SyntaxError: Unexpected end of JSON input
  - JSON.parse
  - index.js:4 Object.module.exports
    [mobytrip-landing-pages]/[json-loader]/index.js:4:49
  - @ ./.cache/sync-requires.js 144:15387-15497
error Command failed with exit code 1.

Do let me know if you need more information or an example page itself to work with. Thanks in advance.

[c2pig]

i have the same error

Gatsby version: 1.9.126
node.js version: v8.7.0
OS version: Darwin Kernel Version 17.0.0

success createPages — 9.367 s
success createPagesStatefully — 11.824 s
success onPreExtractQueries — 0.005 s
success update schema — 2.286 s
success extract queries from components — 0.243 s
success run graphql queries — 0.611 s
success write out page data — 0.013 s
success write out redirect data — 0.001 s
Generating image thumbnails [==============================] 720/720 26.1 secs 100%

info bootstrap finished - 107.035 s

success onPostBootstrap — 0.001 s
success Building CSS — 30.489 s
success Building production JavaScript bundles — 52.952 s

error Building static HTML for pages failed

See our docs page on debugging HTML builds for help https://goo.gl/yL9lND


  Error: Module build failed: SyntaxError: Unexpected end of JSON input
  
  - JSON.parse
  
  - index.js:4 Object.module.exports
    [xxxxxxxxx]/[json-loader]/index.js:4:49
  
  - @ ./.cache/sync-requires.js 26:13635-13717
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! Exit status 1

[KyleAMathews]

And downgrading reliably fixes this?

This actually makes sense — we saw this before where some queries hadn't finished being written out before moving to next step which meant the JSON was sometimes only partially written out. It's possible this recent PR introduced a similar bug #2969

[c2pig]

Error message below does not show the root cause(deeper error stack) :-

  Error: Module build failed: SyntaxError: Unexpected end of JSON input  
  - JSON.parse  
  - index.js:4 Object.module.exports
    [xxxxxxxxx]/[json-loader]/index.js:4:49

so i have to console.log(webpackErrors) manually. :-

Building static HTML for pages[ './.cache/json/quantity-surveying.json\nModule build failed: SyntaxError: Unexpected end of JSON input\n    at JSON.parse (<anonymous>)\n    at Object.module.exports (/Users/c2pig/projects/postschool-fe/node_modules/json-loader/index.js:4:49)\n @ ./.cache/sync-requires.js 26:13635-13717',
  './.cache/json/veterinary-science.json\nModule build failed: SyntaxError: Unexpected end of JSON input\n    at JSON.parse (<anonymous>)\n    at Object.module.exports (/Users/c2pig/projects/postschool-fe/node_modules/json-loader/index.js:4:49)\n @ ./.cache/sync-requires.js 26:13877-13959',
  './.cache/json/real-estate-management.json\nModule build failed: SyntaxError: Unexpected end of JSON input\n    at JSON.parse (<anonymous>)\n    at Object.module.exports (/Users/c2pig/projects/postschool-fe/node_modules/json-loader/index.js:4:49)\n @ ./.cache/sync-requires.js 26:14123-14209',

I found that image links in markdown files transformed to base64 string and larger json files will be produced.

Once gatsby-remarks-images remove from gatsby-config.js, project built successfully. ie:

      resolve: `gatsby-transformer-remark`,
      options: {
        plugins: [
          /* REMOVED
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 1000,
            },
          },
          */
        ...
        ],
      },

Hope this helps.

[ghost]

@KyleAMathews Yep, downgrading reliably fixes this. I've run builds both on 1.9.123 and on 1.9.127 for various sites containing a few hundred pages each. 1.9.123 builds succeed every time whereas 1.9.127 cannot be relied upon to do the same.

Example of a long line in our case:

const hotels = JSON.parse(`[{"hotel_id": 240617, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Oslo Apartments - Sven Bruns Gate, Part of Forenom", "stars": 0.0, "cnt_points_in_1km": 75, "cnt_restaurants_in_1km": 85, "deeplink": "http://www.booking.com/hotel/no/oslo-apartments-as.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/376/37662599.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 6}, "review_nr": 592, "review_score": "7.8", "review_score_word": "Good", "price": 87, "price_formatted": "$87", "max_price": 124, "max_price_formatted": "$124", "p_off": 30}, {"hotel_id": 2040900, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Norwegian Hotelapartments - Bygd\u00f8y All\u00e9", "stars": 0.0, "cnt_points_in_1km": null, "cnt_restaurants_in_1km": null, "deeplink": "http://www.booking.com/hotel/no/apartment-bygdoy-alle.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/827/82744477.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 1}, "review_nr": 8, "review_score": "6.5", "review_score_word": "Pleasant", "price": 141, "price_formatted": "$141", "max_price": 195, "max_price_formatted": "$195", "p_off": 28}, {"hotel_id": 411653, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Oslo Apartments - Rosenborggate, Part of Forenom", "stars": 0.0, "cnt_points_in_1km": 20, "cnt_restaurants_in_1km": 24, "deeplink": "http://www.booking.com/hotel/no/oslo-apartments-rosenborggate-24.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/786/78664437.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 2}, "review_nr": 208, "review_score": "6.9", "review_score_word": "Pleasant", "price": 87, "price_formatted": "$87", "max_price": 113, "max_price_formatted": "$113", "p_off": 23}, {"hotel_id": 1986972, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Frogner House Apartments - Huitfeldtsgate 19", "stars": 0.0, "cnt_points_in_1km": 61, "cnt_restaurants_in_1km": 66, "deeplink": "http://www.booking.com/hotel/no/frogner-house-apartments-huitfeldtsgate-19.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/800/80046450.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 4}, "review_nr": 120, "review_score": "8.9", "review_score_word": "Fabulous", "price": 354, "price_formatted": "$354", "max_price": 406, "max_price_formatted": "$406", "p_off": 13}, {"hotel_id": 1093856, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Saga Apartments Oslo", "stars": 0.0, "cnt_points_in_1km": 28, "cnt_restaurants_in_1km": 39, "deeplink": "http://www.booking.com/hotel/no/majorstuen-apartments.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/463/46341344.jpg", "room_min_price": {"roomtype_id": 1, "refundable": true, "num_rooms_available_at_this_price": 1}, "review_nr": 457, "review_score": "8.7", "review_score_word": "Fabulous", "price": 168, "price_formatted": "$168", "max_price": 229, "max_price_formatted": "$229", "p_off": 26}, {"hotel_id": 235675, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Saga Hotel Oslo", "stars": 4.0, "cnt_points_in_1km": 35, "cnt_restaurants_in_1km": 47, "deeplink": "http://www.booking.com/hotel/no/saga-oslo-as.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/167/16765034.jpg", "room_min_price": {"roomtype_id": 10, "refundable": false, "num_rooms_available_at_this_price": 2}, "review_nr": 825, "review_score": "8.7", "review_score_word": "Fabulous", "price": 181, "price_formatted": "$181"}, {"hotel_id": 47736, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Frogner House Apartments - Bygd\u00f8y All\u00e9 53", "stars": 0.0, "cnt_points_in_1km": 16, "cnt_restaurants_in_1km": 14, "deeplink": "http://www.booking.com/hotel/no/frogner-house-hotel-norum.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/407/4075639.jpg", "room_min_price": {"roomtype_id": 12, "refundable": false, "num_rooms_available_at_this_price": 9}, "review_nr": 599, "review_score": "8.3", "review_score_word": "Very good", "price": 156, "price_formatted": "$156", "max_price": 197, "max_price_formatted": "$197", "p_off": 21}, {"hotel_id": 280862, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Cochs Pensjonat", "stars": 2.0, "cnt_points_in_1km": 44, "cnt_restaurants_in_1km": 51, "deeplink": "http://www.booking.com/hotel/no/cochs-pensjonat.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/629/62949726.jpg", "room_min_price": {"roomtype_id": 24, "refundable": true, "num_rooms_available_at_this_price": 2}, "review_nr": 2403, "review_score": "8.1", "review_score_word": "Very good", "price": 113, "price_formatted": "$113"}, {"hotel_id": 2776957, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Frogner House Apartments- Helgesens gate 1", "stars": 0.0, "cnt_points_in_1km": null, "cnt_restaurants_in_1km": null, "deeplink": "http://www.booking.com/hotel/no/frogner-house-apartments-helgesens-gate-1.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/116/116707425.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 1}, "review_nr": 0, "review_score": null, "review_score_word": null, "price": 187, "price_formatted": "$187"}, {"hotel_id": 23636, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Thon Hotel Gyldenl\u00f8ve", "stars": 3.0, "cnt_points_in_1km": 26, "cnt_restaurants_in_1km": 33, "deeplink": "http://www.booking.com/hotel/no/thon-gyldenlove.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/204/20438979.jpg", "room_min_price": {"roomtype_id": 10, "refundable": true, "num_rooms_available_at_this_price": 6}, "review_nr": 781, "review_score": "7.9", "review_score_word": "Good", "price": 168, "price_formatted": "$168"}]`)

[KyleAMathews]

Can one of you create a site which reproduces the problem and push it to GitHub? Or would you like to take a crack at figuring out how the changes in #2969 broke things for you?

[ghost]

I can do the first for sure, just gotta clean up the data for public access. I'll take a crack at the second too if I get some time later.

[ghost]

Here's the site that reproduces the problem: https://github.com/TripElephant/gatsby-issue-3094

[kennedyrose]

I've got the same issue. I don't have any long JSON strings, I just have a lot of small JSON files. I get the error on about 10 of them in dev. Same deal, seems to work fine in 1.9.123.

[KyleAMathews]

Should be fixed in #3218 lemme know if you're still seeing troubles!

[gaearon]

I am still seeing this (or maybe other?) issue when editing a newly added long blog post, even after to upgrading to 1.9.135.

Here is my log after touching a file and immediately editing it back:

info changed file at /Users/dan/p/reactjs.org/content/blog/2017-12-14-improving-the-repository-infrastructure.md
 WAIT  Compiling...                                                                              01:34:15

info changed file at /Users/dan/p/reactjs.org/content/blog/2017-12-14-improving-the-repository-infrastructure.md
 DONE  Compiled successfully in 4336ms                                                           01:34:21

 WAIT  Compiling...                                                                              01:34:21

info changed file at /Users/dan/p/reactjs.org/content/blog/2017-12-14-improving-the-repository-infrastructure.md
 ERROR  Failed to compile with 4 errors                                                          01:34:23

 error  in ./.cache/json/blog-2014-01-06-community-roundup-14-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)


 @ ./.cache/sync-requires.js 28:18085-18179

 error  in ./.cache/json/blog-2014-02-05-community-roundup-15-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)


 @ ./.cache/sync-requires.js 28:18351-18445

 error  in ./.cache/json/blog-2014-02-15-community-roundup-16-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)


 @ ./.cache/sync-requires.js 28:18617-18711

 error  in ./.cache/json/blog-2014-02-16-react-v-0-9-rc-1-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)


 @ ./.cache/sync-requires.js 28:18879-18969

 WAIT  Compiling...                                                                              01:34:23

 DONE  Compiled successfully in 1712ms                                                           01:34:26

 WAIT  Compiling...                                                                              01:34:26

 DONE  Compiled successfully in 3748ms                                                           01:34:31

 WAIT  Compiling...                                                                              01:34:32

 DONE  Compiled successfully in 984ms                                                            01:34:35

 WAIT  Compiling...                                                                              01:34:35

 DONE  Compiled successfully in 447ms                                                            01:34:36

 WAIT  Compiling...                                                                              01:34:36

 DONE  Compiled successfully in 784ms                                                            01:34:37

 WAIT  Compiling...                                                                              01:34:37

 DONE  Compiled successfully in 485ms                                                            01:34:38

 WAIT  Compiling...                                                                              01:34:38

 ERROR  Failed to compile with 1 errors                                                          01:34:40

 error  in ./.cache/json/blog-2017-06-13-react-v-15-6-0-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)


 @ ./.cache/sync-requires.js 28:40543-40631

 WAIT  Compiling...                                                                              01:34:40

 DONE  Compiled successfully in 1113ms                                                           01:34:42

Note how, in addition to these errors, it also recompiles the same thing five times.

[ghost]

@KyleAMathews Will try it out soon! Thanks so much for looking into this.

[ghost]

@KyleAMathews Just tested #3218 on 660 different sites spanning 56,275 pages. Works like a charm! Thanks again for the fix--we really appreciate it.

[KyleAMathews]

@healingfromjava lol what in the world are you doing? 660 sites? Glad it works though!

[attfarhan]

@gaearon were you able to resolve your issue? I'm seeing a similar error to the one you posted, which results in hot reloading not working for pages generated from markdown files, even after updating to the latest Gatsby version.

[gaearon]

My understanding is it was supposed to be fixed with #3237. I haven't actually checked.

[stevensurgnier]

I'm still seeing this behavior on 1.9.157 with our project. I'll report back with more info.