Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors when processing large(ish) numbers of images with gatsby-plugin-sharp #6291

Open
jp887 opened this issue Jul 4, 2018 · 58 comments

Comments

@jp887
Copy link

@jp887 jp887 commented Jul 4, 2018

Description

I'm processing around 1500 images with gatsby-plugin-sharp/gatsby-transformer-sharp and GraphQL, to display with gatsby-image as per this page in the docs.

For smaller numbers of images I have had no issues, but now that I'm processing 1500 images the build randomly fails with the following error:

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: gtype.c:4265: type id '0' is invalid

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:104): GLib-GObject-WARNING **: 09:35:11.293: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

(sharp:104): GLib-GObject-CRITICAL **: 09:35:11.293: g_value_transform: assertion 'G_IS_VALUE (src_value)' failed
Received 'segmentation fault' signal

This happens during the "Generating image thumbnails" step in the build process and it occurs apparently randomly (sometimes 10% of the way through the images, sometimes at 80%, sometimes not at all). Therefore I do not believe it is caused by a "faulty image".

@m-allanson

This comment has been minimized.

Copy link
Member

@m-allanson m-allanson commented Jul 6, 2018

Hmm that's not a lot of info to go on.. can you post an example repo that creates the error you're seeing?

@rensje

This comment has been minimized.

Copy link

@rensje rensje commented Sep 2, 2018

Had the same issue. Doing a manual npm install git+https://github.com/lovell/sharp.git to install the latest unreleased Sharp package seems to have fixed it for me. This might not be a good idea, however I'm not sure what else to do until a fixed version of Sharp is released.

@HriBB

This comment has been minimized.

Copy link

@HriBB HriBB commented Sep 14, 2018

I can confirm this bug. In my case it started happening, when I added some large images, 5mb or more.

@btk

This comment has been minimized.

Copy link
Member

@btk btk commented Oct 31, 2018

I had the same problem;

Using an older version for gatsby-transformer-sharp (2.1.1) and gatsby-plugin-sharp (2.0.5) fixed it for me. Maybe if there is any debuggers, this might be an additional information.

Note that I also have a lot of images on my build.

@jp887

This comment has been minimized.

Copy link
Author

@jp887 jp887 commented Nov 2, 2018

@m-allanson I've started working on a repro here: https://github.com/jp887/gatsby-issue6291-repro
Basically generate a large number of random images with a script and then query for them in a basic gatsby 2 site.

I'm building the project on CircleCI so build outputs can be inspected without having to run the project locally.

The most recent build reproduced the issue: https://circleci.com/gh/jp887/gatsby-issue6291-repro/3

@btk

This comment has been minimized.

Copy link
Member

@btk btk commented Nov 2, 2018

@jp887 would you mind trying to take a build on your repro with these gatsby-transformer-sharp (2.1.1) gatsby-plugin-sharp (2.0.5) exact versions, on a new branch maybe? I have downgraded to these versions, and I have taken more than 10 successful builds since.

@jp887

This comment has been minimized.

Copy link
Author

@jp887 jp887 commented Nov 2, 2018

@btk OK done. Builds on the 'downgraded' branch can be found here: https://circleci.com/gh/jp887/workflows/gatsby-issue6291-repro/tree/downgraded

@jp887

This comment has been minimized.

Copy link
Author

@jp887 jp887 commented Nov 2, 2018

Seems it's still reproducing the issue on the new downgraded branch: https://circleci.com/gh/jp887/gatsby-issue6291-repro/4

@niklasravnsborg

This comment has been minimized.

Copy link
Member

@niklasravnsborg niklasravnsborg commented Nov 22, 2018

I also have the problem in a recent project.
screenshot 2018-11-22 at 16 15 11

I also get a lot of vips warning: exif: unknown EXIF resolution unit errors.

@TylerBarnes

This comment has been minimized.

Copy link
Contributor

@TylerBarnes TylerBarnes commented Nov 29, 2018

I'm running into this as well but with a lot less images

screen shot 2018-11-29 at 2 30 27 pm

For me it happens when I'm changing maxWidth and quality on a fluid graphql query

                        fluid(maxWidth: 400, quality: 100) {
                          ...GatsbyImageSharpFluid_withWebp_tracedSVG
                        }
@jonniebigodes

This comment has been minimized.

Copy link
Contributor

@jonniebigodes jonniebigodes commented Nov 30, 2018

@TylerBarnes ohhh the dreaded segmentation fault error, it brings me back to when i was using c language as a development tool. That looks like that somewhere deep down in the execution of that plugin, it's trying to allocate memory where's not allowed and it was not deallocated and trying to retrieve something that is not allowed. That is not a gatsby plugin issue per se, more like the underlying vips/sharp, the one written in c.

@gatsbot

This comment has been minimized.

Copy link

@gatsbot gatsbot bot commented Feb 3, 2019

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

@gatsbot gatsbot bot added the stale? label Feb 3, 2019
@kamilgrzegorczyk

This comment has been minimized.

Copy link

@kamilgrzegorczyk kamilgrzegorczyk commented Feb 4, 2019

I am having the same issue at the same build step.

My plugin versions are:

    "bourbon-neat": "3.0.1",
    "dotenv": "^6.2.0",
    "gatsby": "^2.0.112",
    "gatsby-image": "^2.0.29",
    "gatsby-plugin-manifest": "^2.0.17",
    "gatsby-plugin-offline": "^2.0.22",
    "gatsby-plugin-react-helmet": "^3.0.6",
    "gatsby-plugin-sass": "^2.0.10",
    "gatsby-plugin-sharp": "^2.0.20",
    "gatsby-source-wordpress": "^3.0.32",
    "gatsby-transformer-sharp": "^2.1.13",
    "node-sass": "^4.11.0",
    "normalize-scss": "^7.0.1",
    "react": "^16.7.0",
    "react-dom": "^16.7.0",
    "react-helmet": "^5.2.0"
@briandamaged

This comment has been minimized.

Copy link

@briandamaged briandamaged commented Feb 9, 2019

I seem to have encountered this issue as well. I'll add more to this thread if I discover any additional clues.

@briandamaged

This comment has been minimized.

Copy link

@briandamaged briandamaged commented Feb 9, 2019

Also... I haven't been able to figure out how to apply the "not stale" label to this issue. I'm guessing that I need to be granted a permission?

In either case: the issue does appears to be somewhat random. I just reran the same build without encountering the problem this time. (In both cases, I cleared all cached files before starting the build)

@btk btk added not stale and removed stale? labels Feb 10, 2019
@btk

This comment has been minimized.

Copy link
Member

@btk btk commented Feb 10, 2019

@briandamaged added it for you.

@briandamaged

This comment has been minimized.

Copy link

@briandamaged briandamaged commented Feb 16, 2019

@btk : Thx! So far, the issue has not reoccurred.

@mdornseif

This comment has been minimized.

Copy link

@mdornseif mdornseif commented Feb 18, 2019

I still see this when build by nelify:

10:08:33 AM: success extract queries from components — 0.324 s
10:08:51 AM: (sharp:1324): GLib-GObject-WARNING **: 09:08:51.843: gtype.c:4265: type id '0' is invalid
10:08:51 AM: (sharp:1324): GLib-GObject-WARNING **: 09:08:51.843: can't peek value table for type '<invalid>' which is not currently referenced
10:08:51 AM: (sharp:1324): GLib-GObject-WARNING **: 09:08:51.843: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
10:08:51 AM: (sharp:1324): GLib-GObject-CRITICAL **: 09:08:51.843: g_value_transform: assertion 'G_IS_VALUE (dest_value)' failed
10:08:54 AM: /usr/local/bin/build: line 32:  1324 Segmentation fault      (core dumped) gatsby build
10:08:54 AM: Caching artifacts
…
10:08:55 AM: failed during stage 'building site': Build script returned non-zero exit code: 139

It randomly happens using exactly the same git revision:

See http://filez.foxel.org/4dec3e172673 for version Information.

@mdornseif

This comment has been minimized.

Copy link

@mdornseif mdornseif commented Feb 18, 2019

This seems to be unrelated and dos not lead to segfaults / failed builds (same repository):

12:04:08 AM: success Building production JavaScript and CSS bundles — 41.675 s
12:04:10 AM: success Building static HTML for pages — 2.570 s — 109/109 87.88 pages/second
12:04:16 AM: info Done building in 661.124 sec
12:04:16 AM: (sharp:1506): GLib-CRITICAL **: 23:04:16.327: g_hash_table_lookup: assertion 'hash_table != NULL' failed
12:04:16 AM: (sharp:1506): GLib-CRITICAL **: 23:04:16.750: g_hash_table_lookup: assertion 'hash_table != NULL' failed
12:04:17 AM: (sharp:1506): GLib-CRITICAL **: 23:04:17.106: g_hash_table_lookup: assertion 'hash_table != NULL' failed
12:04:17 AM: Caching artifacts
12:04:17 AM: Started saving node modules
12:04:17 AM: Finished saving node modules
@mdornseif

This comment has been minimized.

Copy link

@mdornseif mdornseif commented Feb 19, 2019

Regarding the segfaults I have found this in the gatsby-plugin-sharp:

// Try to enable the use of SIMD instructions. Seems to provide a smallish
// speedup on resizing heavy loads (~10%). Sharp disables this feature by
// default as there's been problems with segfaulting in the past but we'll be
// adventurous and see what happens with it on.
sharp.simd(true);

So far locally I haven't seen any segfaults after commenting out sharp.simd(true);.

@TylerBarnes

This comment has been minimized.

Copy link
Contributor

@TylerBarnes TylerBarnes commented Feb 19, 2019

That would make sense. I've only seen this on sites with a lot of images and then it's still rare on those sites. Maybe once every 20 - 30 builds.

@niklasravnsborg

This comment has been minimized.

Copy link
Member

@niklasravnsborg niklasravnsborg commented Feb 20, 2019

@mdornseif @TylerBarnes Thanks for the discovery! I went ahead and opened a PR, removing the lines: #11925 😌

@tinder-kyleboss

This comment has been minimized.

Copy link
Contributor

@tinder-kyleboss tinder-kyleboss commented Feb 21, 2019

Thank you so much for the awesome finds @mdornseif & the PR @niklasravnsborg. 😄

I am also running into these gatsby-plugin-sharp-inspired segmentation faults. Unfortunately, deactivating the SIMD functionality did not seem to improve the frequency of these segmentation faults. 😢

@KyleAMathews

This comment has been minimized.

Copy link
Contributor

@KyleAMathews KyleAMathews commented Feb 21, 2019

Something else we could do to improve stability is to move Sharp to a child process as if it crashes, we could just restart it.

@tinder-kyleboss

This comment has been minimized.

Copy link
Contributor

@tinder-kyleboss tinder-kyleboss commented Feb 21, 2019

That would be a cool approach, I like it! 👍

Despite this probably slowing down a build or two, I think that this would definitely satisfy a majority of the folks who run into this issue. 😄

@tombunn

This comment has been minimized.

Copy link

@tombunn tombunn commented Jun 3, 2019

I ran into this issue running the latest versions of gatsby-plugin-sharp and gatsby-transformer-sharp. Could not fix it at all until i downgraded to gatsby-transformer-sharp 2.1.1 and gatsby-plugin-sharp 2.0.5 as @btk suggested. Built locally first try after that.

@jp887

This comment has been minimized.

Copy link
Author

@jp887 jp887 commented Jun 3, 2019

@KyleAMathews I've updated the dependencies in the original repro and I'm still seeing the same error. After upgrading the first build succeeded but the next one reproduced the error.

For whatever reason, my experience when building locally is that each time I run the build it gets a bit further before failing. I don't see this pattern when building in CircleCI (which spins up a new container each time), so I assume some kind of caching is causing this behaviour locally.

@browniebroke

This comment has been minimized.

Copy link
Member

@browniebroke browniebroke commented Jun 11, 2019

In case it helps, here is a pull request on a public repo with the issue: browniebroke/chez-nicole-web#351

The Netlify logs: https://app.netlify.com/sites/chez-nicole/deploys/5cff6ca75c28e2000917dfc4

Update: the above pull request was later retried by Renovate and ended up working fine. The issue appear intermittently still.

@PatrickLambert1101

This comment has been minimized.

Copy link

@PatrickLambert1101 PatrickLambert1101 commented Jun 12, 2019

For me, my local build is working and may Netlify build has the same 1393 Segmentation fault (core dumped) gatsby build issue

@wardpeet

This comment has been minimized.

Copy link
Member

@wardpeet wardpeet commented Jun 12, 2019

It's a hard nut to crack... You could try setting the env variable GATSBY_CPU_COUNT=logical_cores. I'm unsure if this even helps but might be worth the try

@lvidal1

This comment has been minimized.

Copy link

@lvidal1 lvidal1 commented Jun 22, 2019

I have the same issue. Running good on my local machine but getting the 1393 Segmentation fault (core dumped) gatsby build message on Netlify.
Any update on this?

@portenez

This comment has been minimized.

Copy link

@portenez portenez commented Jun 24, 2019

This might be helpful...

The segfault happens to me when running the gatsby build in an alpine based container, but not in a debian based container.

To reproduce, try use this docker file based on alpine, and then this other dockerfile using debian. The error never happens on the debian based. Notice that those two, are both base on official node images.

This probably means netflify is using an alpine version. Or another container with a similar problem.

@wardpeet

This comment has been minimized.

Copy link
Member

@wardpeet wardpeet commented Jun 24, 2019

Let me see if I can get this reproduced on alpine docker container.

@stephenh

This comment has been minimized.

Copy link

@stephenh stephenh commented Jun 24, 2019

FWIW we worked with @KyleAMathews & the author of sharp to debug this on our internal ~large/~5,000 image gatsby build and the fixes were:

  • The core issue was our build running in CircleCI and CircleCI reporting that our build had ~32 some cores to use (i.e. all of the cores of the entire host machine, instead of our build's docker container). This confused/made the sharp glibc concurrency buggy (i.e. running 32 threads against an image when there was only 1 physical cpu anyway). This confusing resource detection is a known issue in CircleCI: https://ideas.circleci.com/ideas/CCI-I-578
  • Setting GATSBY_CPU_COUNT=1 in our build environment to ignore the bad cpu count +
  • A fix for gatsby-plugin-sharp to respect GATSBY_CPU_COUNT instead of ignoring it ( #14624)
  • Adding sharp.cache(false) and sharp.simd(false) to our project's ./gatsby-node.js to further turn off other "too much concurrency" behavior in sharp's implementation, again due to the misreported cpu/etc. stats from the CircleCI environment.

With these changes, our build went from failing ~80% of the time to stable for a ~week. Props to @lovell for all of this work, I'm just the messenger. Also defer to @KyleAMathews and the other gatsby maintainers for how these lessons learned/etc. can get worked into the maintain gatsby code.

@MWalid

This comment has been minimized.

Copy link

@MWalid MWalid commented Jun 29, 2019

Hi @stephenh,

How did you add sharp.simd(false) and sharp.cache(false) to gatsby-node.js?

We have a website with 9K images. For one month we cannot get the build to work, it always fails with segmentation fault!

@wardpeet wardpeet self-assigned this Jul 4, 2019
@stephenh

This comment has been minimized.

Copy link

@stephenh stephenh commented Jul 5, 2019

@MWalid I assume that a file named gatsby-node.js in the root of your project is just auto-included by gatsby, i.e. see https://www.gatsbyjs.org/tutorial/part-seven/ and search for gatsby-node.js.

@MWalid

This comment has been minimized.

Copy link

@MWalid MWalid commented Jul 7, 2019

Thank you so much @stephenh.

Actually I figured it out the same day I asked :)

Your comment resolved the issue for me, builds now take 4.5 hours to complete, but that's totally fine with a daily build.

earthday47 added a commit to thirdandgrove/thirdandgrove-com-gatsby that referenced this issue Jul 9, 2019
@grantglidewell

This comment has been minimized.

Copy link
Contributor

@grantglidewell grantglidewell commented Jul 9, 2019

We had run into a similar issue, building on Netlify started to fail consistently after updating something related to gatsby-image.
our errors look like:

11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed
11:15:13 AM: (sharp:1397): GLib-CRITICAL **: 18:15:13.593: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:15:21 AM: /usr/local/bin/build: line 34:  1397 Segmentation fault      (core dumped) gatsby build

we implemented the approaches above, adding

const sharp = require('sharp')

sharp.simd(false)
sharp.cache(false)

to our gatsby-node file. we also added a NODE_VERSION environment variable set to 12 in Netlify. Node 12 has an inherently higher memory cap. also added NODE_ENV as production.

Now in Netlify our builds don't immediately succeed, we have to clear the cache and rebuild

@jlin95

This comment has been minimized.

Copy link

@jlin95 jlin95 commented Jul 11, 2019

@grantglidewell Did you put
const sharp = require('sharp') sharp.simd(false) sharp.cache(false)

this in any of the API hooks in gatsby-node.js or did you just place this block at the top of the file?

@grantglidewell

This comment has been minimized.

Copy link
Contributor

@grantglidewell grantglidewell commented Jul 11, 2019

just in the file, not in any exports. this was per @DSchau

@jlin95

This comment has been minimized.

Copy link

@jlin95 jlin95 commented Jul 12, 2019

just in the file, not in any exports. this was per @DSchau

I see. I tried that in my project. So while the build passed, some image thumbnails were still not getting generated. Wondering if there were other things you had to implement in the code in order for this to work. We don't use Netlify for our builds :)

Thanks!

@jamessimone

This comment has been minimized.

Copy link
Contributor

@jamessimone jamessimone commented Aug 2, 2019

@TylerBarnes your comment is very interesting - while I am not experiencing the issue at hand, I am experiencing massive slowdowns in builds caused by gatsby-remark-image / filesystem constantly rebuilding all of my image files on deploy. I made an example repo and filed an issue with the creator of the Netlify cache plugin, but he hasn’t responded as of yet and I’m going half mad trying to figure out where I’ve gone wrong with my caching strategy. Any insight / tips on correctly caching images downloaded using createRemoteFileNode (TL;DR from my linked example repo, I try to cache the created gatsby node containing the images), that would be EXTREMELY appreciated

@MaximeHeckel

This comment has been minimized.

Copy link

@MaximeHeckel MaximeHeckel commented Aug 8, 2019

Hey, I had the exact same issue recently with gatsby-images and gatsby-plugin-sharp. I'd get the same error messages as @grantglidewell on Netlify, and not all my images would be generated at build time (which resulted in only 2 out of 5 images being able to render on my website, for the rest I'd only get the blurred version)
After 2 hours of trial and error I tried to update to the latest "gatsby-plugin-sharp": "^2.2.10", (as of the day this comment is posted) and my builds are now 100% successful and the images are properly processed. Hope that helps
Note I'm on "gatsby": "^2.13.52",

@dmaynard

This comment has been minimized.

Copy link

@dmaynard dmaynard commented Aug 10, 2019

Thank you @MaximeHeckel I was having the same problem with local builds. gatsby build seemed to exit before all of the images were packaged (but gatsby develop worked fine). Upgrading to "gatsby-plugin-sharp": "^2.2.10" seems to have fixed this problem for me.

@antoinerousseau

This comment has been minimized.

Copy link
Contributor

@antoinerousseau antoinerousseau commented Aug 21, 2019

I'm having the same problem with a local gatsby build if I don't disable shard's simd and cache.

[===========                 ]   5.893 s 344/850 40% Generating image thumbnails
/bin/sh: line 1:  5438 Segmentation fault: 11  gatsby build
error Command failed with exit code 139.

gatsby@2.13.73
gatsby-plugin-sharp@2.2.13
gatsby-transformer-sharp@2.2.7

node v10.15.3 on macOS

@kpennell

This comment has been minimized.

Copy link
Contributor

@kpennell kpennell commented Oct 28, 2019

Downgrading to gatsby-plugin-sharp@2.2.14 fixes the issue.

per: #16957 (comment)

@kpennell kpennell closed this Nov 4, 2019
@kpennell kpennell reopened this Nov 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.