Parallel Query Running #32389
Replies: 18 comments 64 replies
-
|
Hi Just updated to 3.10.0 and enabled There was an error in your GraphQL query: Unknown type "HATCH_ContentMenuItemType".
A few details:
Hopefully, this is helpful... |
Beta Was this translation helpful? Give feedback.
-
|
@vladar Hope you had a nice time off. Just wondered if you had a chance to review this? |
Beta Was this translation helpful? Give feedback.
-
|
Had to switch to another urgent thing. Plan to get back to it the next week. Sorry for the miscommunication. |
Beta Was this translation helpful? Give feedback.
-
|
I tried to run the project but I don't have env vars to test it and see the error. Do you have a sandbox environment I can try it with? E-mail: vladimir@gatsbyjs.com |
Beta Was this translation helpful? Give feedback.
-
|
Email sent 👍 |
Beta Was this translation helpful? Give feedback.
-
|
@vladar with your above advice and some refactoring, I have got PQR working with the toolkit (I can see workers ... its merged into the master branch if you wanted to take a look. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, reporting an error with PQR and gatsby-source-wordpress it looks like: Logs from gatsbyjs.com on Gatsby Cloud @TylerBarnes this seems to be an issue with the wordpress plugin |
Beta Was this translation helpful? Give feedback.
-
|
We were able to get past this in the latest |
Beta Was this translation helpful? Give feedback.
-
|
Hey there, I just wanted to report that I got the following error when trying PQR also |
Beta Was this translation helpful? Give feedback.
-
|
Hi, thanks for trying PQR! Can you share the project with us so that we can investigate it? |
Beta Was this translation helpful? Give feedback.
-
|
Do you use schema customization API in your project? Maybe you use |
Beta Was this translation helpful? Give feedback.
-
|
Im using the file system route api, I believe the error is related to it. The project is private but I could try to create a minimal reproduction. Also the error doesn't happen during |
Beta Was this translation helpful? Give feedback.
-
|
PQR is only happening during |
Beta Was this translation helpful? Give feedback.
-
I hope this is documented somewhere, I missed this entirely |
Beta Was this translation helpful? Give feedback.
-
|
Getting these errors after enabling PQR on Netlify builds. |
Beta Was this translation helpful? Give feedback.
-
|
@TylerBarnes we tried |
Beta Was this translation helpful? Give feedback.
-
|
getting the same as @rburgst when building w/o cache, triggering PQR |
Beta Was this translation helpful? Give feedback.
-
|
@LekoArts @TylerBarnes we still see this error happening in our CI and with gatsby 4 there is no way to disable this anymore. Therefore, we are stuck with gatsby 3. Is there some way we can get more information which process is exactly failing and why? |
Beta Was this translation helpful? Give feedback.
-
|
So we have been able to get around these |
Beta Was this translation helpful? Give feedback.
-
|
@rburgst - have tried the latest version of Gatsby? In Gatsby 4.8 we introduced changes that seem relevant, given that you claimed your site has many large images. I'd be curious if this resolves the memory spikes for you. |
Beta Was this translation helpful? Give feedback.
-
|
Seeing the following message logged multiple times (once for each worker);
The offending code is here: https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/services/initialize.ts#L35-L54 it looks like this is inadvertently loaded for each child worker. |
Beta Was this translation helpful? Give feedback.
-
|
heh, yeah, I think I've also seen this locally. Thanks for the report! Something we should fix at some point :) |
Beta Was this translation helpful? Give feedback.
-
|
I'm using and crashes. When I remove the |
Beta Was this translation helpful? Give feedback.
-
|
I've created this for you: https://github.com/laurenskling/PARALLEL_QUERY_RUNNING Build on Gatsby Cloud: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for creating this! The problem is this line: This is one of the constraints mentioned in the starting post. We'll need to fix this |
Beta Was this translation helpful? Give feedback.
-
|
Ah, it was under the hood. I guess lots of people are using |
Beta Was this translation helpful? Give feedback.
-
|
@LekoArts will this be fixed for |
Beta Was this translation helpful? Give feedback.
-
|
#33310 just found this :) |
Beta Was this translation helpful? Give feedback.
-
|
The error only occurs when PARALLEL_QUERY_RUNNING=true and mdx files in nested dirs > 255 characters. gatsby info |
Beta Was this translation helpful? Give feedback.
-
|
Got this error when doing a cold build: Gatsby info: |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am creating a Gatsby project with Contentful and deploying it in Netlify. 9:03:36 PM: error ENOENT: no such file or directory, open '/opt/build/repo/.cache/caches/gatsby-source-contentful/e16675795aa57c817fbc2ebb9efecfdb-3/chuecking-pc-2.jpg' |
Beta Was this translation helpful? Give feedback.
-
|
I'm having the exact same consistent error, a missing image in the cache. Weirdly enough, only happens when the build is triggered via the contentful webhook, if I hit the 'deploy' button in netlify UI it works fine, even without clearing the cache. |
Beta Was this translation helpful? Give feedback.
-
|
For this issue, please
If you still encounter the issue, please submit an issue via our standard bug reporting process here |
Beta Was this translation helpful? Give feedback.
-
|
I'm getting this error after upgrading to gatsby v4. Builds on vercel and local develop works but local build doesn't work. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I resolved this error by decreasing the width of one of the images. |
Beta Was this translation helpful? Give feedback.
-
|
@jibin2706 I may face up with the same issue. Do you have any tips for debugging? How did you know that you have to decrease the width of the image? |
Beta Was this translation helpful? Give feedback.
-
|
@HuyAms First I removed TRACED_SVG placeholder option from all the images then the build started working. After that it was just trial and error adding TRACED_SVG back, one by one to check which image was causing the issue |
Beta Was this translation helpful? Give feedback.
-
|
@jibin2706 - can you share the faulty image back with us via our standard bug reporting process here so we can create a minimal reproduction? It would certainly be helpful if we were able to be clearer about the specific image causing the problem. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Can you turn off parallel query running for Gatsby 4? @LekoArts |
Beta Was this translation helpful? Give feedback.
-
|
I would like to know that. It seem to break my builds by not having all the images ready on time. |
Beta Was this translation helpful? Give feedback.
-
|
@MiKatre can you try with gatsby@~4.5.0 (basically, a version of Gatsby on 4.5.x). I was hitting this error above, as well, in updating my personal site to Gatsby@4.6.0, and downgrading to 4.5.0 did the trick. If using npm: If that also works for you, let me know, and we'll add an issue as it seems like it may be a regression in 4.6.0 around images or something else related to parallel query running. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @DSchau ! I will try as soon as I can. Not knowing this, I rolled back all the way to ~3.14.0 |
Beta Was this translation helpful? Give feedback.
-
|
@MiKatre - were you able to successfully build via Gatsby 4.5? I'd also ask if you're able to build successfully with the latest version of Gatsby ( 4.9 as of this post ). If you're unable to build with the latest version of Gatsby as well, please submit an issue via our standard bug reporting process here |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I managed to migrate my site from gatsby 2 to version 4. The Build is successful in my local. However, when I deploy it to Netlify, I get the following error. I also got this error during the build time Not sure why the building time increases dramatically after I migrating my site to gatsby 4 (with same codebase) Could you suggest to me what this error means and possibly how to fix the issue? Many thanks |
Beta Was this translation helpful? Give feedback.
-
|
After the build is successful, I have cache data. If I then build the site again without cleaning the cache, I got this error |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps the MaxListenersExceededWarning error is related to #34795? |
Beta Was this translation helpful? Give feedback.
-
|
@HuyAms - if you're still seeing issues please submit a bug through our standard process here. |
Beta Was this translation helpful? Give feedback.
-
|
Just upgraded from Gatsby 4.4 to 4.6 and started getting this build error, both locally and on Netlify. it's a different image every time, but it seems to consistently happen while querying my contentful images. I tried downgrading to v4.5.5 and I got build success locally and in Netlify. I was content with that solution, but then I noticed that I had left the flag I know DETECT_NODE_MUTATIONS aren't supposed to be left on (my bad) but I'm guessing something in 4.6 made it take so long it was causing build errors. |
Beta Was this translation helpful? Give feedback.
-
|
As a note, I've run ~ 6 builds since posting this an hour ago (between deploys, branch deploys, and deploy previews) and 1 failed at the same step. Clearing the cache and building worked.. but there was no cache for this build to start with. Seems to be an intermittent issue with the query workers. Considering rolling back to 4.5.5 again to see if it's more stable. |
Beta Was this translation helpful? Give feedback.
-
|
@panzacoder - was the issue resolved when you rolled back to 4.5.5? We're trying to determine whether this is an issue with Gatsby itself, or some other factor. If the issue wasn't resolved by rolling back, please submit a bug following our standard process here. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Stuck with this error when trying to deploy my site on netlify. I have recently update from gatsby v2 to v4. After that I am facing the issue, please help. This is the output of my |
Beta Was this translation helpful? Give feedback.
-
|
@mohammed786 It might caused by memory limitations. Can you try specifying |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am trying to migrate my site from gatsby 2 to version 4. Stuck with this error while fetching the hero image. Would you know what the error means and how to fix it? Many thanks My Gatsby info |
Beta Was this translation helpful? Give feedback.
-
|
@HuyAms - thank you for providing that information. Please submit a bug for this issue following our standard process here. |
Beta Was this translation helpful? Give feedback.
-
|
I have been getting a ton of build failures that send me here. Sometimes the output is unhelpful, just saying that there was an error, other times I get something more descriptive, but it is almost always about an image that is having trouble being downloaded from contentful. Just now the issue was: It seems that PQR doesn't allow time for processing images using Is there any stop-gap solutions being worked on? This one in particular is odd because that image clearly exists (you can put that url in your browser and see the image) but it's reporting a 500 error. |
Beta Was this translation helpful? Give feedback.
-
Did you encounter a problem with Query Running?
March 1, 2022 Update: This discussion is now locked so we can better track issue reports via our standard issue reporting process here
🙏 Thank you to everyone that's shared information so far.
After introducing LMDB Data Storage in Gatsby v3.7 we're now offering a new experiment that directly builds on top of LMDB: Parallel Query Running
What is Parallel Query Running?
Gatsby's build process is composed of multiple steps (see our documentation for more details) and one step that will increase in time with more and more pages/nodes is query running. You're seeing this step as
run static queriesandrun page queriesin your build log.This step currently only runs in a singular process and the goal of Parallel Query Running is to spread out the work to multiple processes to better utilize available cores & memory. We're using
gatsby-worker(highly inspired byjest-worker) andlmdb-storeto accomplish this. In Gatsby v3.7 we've added support for LMDB in Gatsby and are now leveraging this new data storage option to enable communication between the main process and the workers.How to Use
Make sure that you're on
gatsby@latest,gatsby-plugin-mdx@latest,gatsby-plugin-sharp@latest,gatsby-transformer-sharp@latestand that you're using Node >= 14.15.Install
lmdb-storeas a dependency:npm install lmdb-store.Add the
PARALLEL_QUERY_RUNNINGflag to yourgatsby-config.js:Run
gatsby build(as this only applies to thebuildstep).By default, Gatsby will use
cpuCount() - 1workers to run the queries (physical cores). So if you have 4 cores/8 threads it'll use 3 workers. You can override this number with theGATSBY_CPU_COUNTenvironment variable.Plugins
gatsby-source-contentfulplease install this version:gatsby-source-contentful@alpha-pqr-compatgatsby-source-wordpressplease installgatsby-source-wordpress@5.12.0-alpha-wordpress.27+67978b34c0Your feedback
Please link to a minimal reproduction or your publicly available project on GitHub.
Feedback directly related to LMDB should still go into the LMDB Umbrella Discussion.
We're very much looking for feedback from you using this new data storage! We'll expand this section with more asks and details in the future, but for now our specific asks would be:
gatsby developandgatsby build)?--verboseflag =>gatsby build --verbosegatsby cleanbeforehand; Warm: With a cache)run page queriesvsrun queries in workersgatsby buildfor before/after PQR to see which steps increased and decreased in timeConstraints
The same notes and constraints from LMDB apply, in addition to these constraints:
ts-nodeyou might get problems with memory since every worker will load the runtime into the process.createTypes,createFieldExtensionandaddThirdPartySchemainsourceNodesis not allowed. Only allowed increateSchemaCustomizationandcreateResolversAPIBeta Was this translation helpful? Give feedback.
All reactions