Update on Gatsby Page Limit - Sep 2019 #17321

crock · 2019-09-02T18:50:08Z

I have this Laravel/Vue site that has over 160k rows and counting (it increases by a couple hundred rows every 5 minutes) in the database. I'm thinking about porting it to a static, Gatsby site, but based on this issue reply, Gatsby hasn't reached the point that it can handle 1 million pages yet as of this time last year. Has anything changed in that regard since then?

If it helps anyone, I would be using the gatsby-source-mysql plugin. My current attempts to source the entire database in one go result in a timeout. When I put a small LIMIT on the select query for testing, Gatsby handles it like a champ.

Any implementation suggestions or is Gatsby really not ideal for a site with this many pages?

matbrgz · 2019-09-23T04:29:58Z

I'm running a benchmarking, gatsby definitely can't handle with broad amount of files.

40M	src/pages/posts
Number of File: 10000 with Content Size: 0.500 bytes and Min Runs: 10
Benchmark #1: yarn run build
yarn run v1.17.3
$ gatsby build
╔════════════════════════════════════════════════════════════════════════╗
║                                                                        ║
║   Gatsby has started collecting anonymous usage analytics              ║
║   to help improve Gatsby for all users.                                ║
║                                                                        ║
║   If you'd like to opt-out, you can use `gatsby telemetry --disable`   ║
║   To learn more, checkout https://gatsby.dev/telemetry                 ║
║                                                                        ║
╚════════════════════════════════════════════════════════════════════════╝
�[2K�[1A�[2K�[G⠋ open and validate gatsby-configs
�[2K�[1A�[2K�[Gsuccess open and validate gatsby-configs - 0.037 s
⠋ open and validate gatsby-configs
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ load plugins
�[2K�[1A�[2K�[Gsuccess load plugins - 0.257 s
⠋ load plugins
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ onPreInit
�[2K�[1A�[2K�[Gsuccess onPreInit - 0.014 s
⠋ onPreInit
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ delete html and css files from previous builds
�[2K�[1A�[2K�[Gsuccess delete html and css files from previous builds - 0.015 s
⠋ delete html and css files from previous builds
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ initialize cache
�[2K�[1A�[2K�[Gsuccess initialize cache - 0.015 s
⠋ initialize cache
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ copy gatsby files
�[2K�[1A�[2K�[Gsuccess copy gatsby files - 0.057 s
⠋ copy gatsby files
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ onPreBootstrap
�[2K�[1A�[2K�[Gsuccess onPreBootstrap - 0.010 s
⠋ onPreBootstrap
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ source and transform nodes
�[2K�[1A�[2K�[G⠙ source and transform nodes
�[2K�[1A�[2K�[G⠹ source and transform nodes
�[2K�[1A�[2K�[G
 ERROR
UNHANDLED REJECTION ENOSPC: System limit for number of file watchers reached, watch '/opt/gatsby/src/src/pages/posts/2019-09-23-08173.md'
⠹ source and transform nodes
�[2K�[1A�[2K�[G

  Error: ENOSPC: System limit for number of file watchers reached, watch '/opt/g  atsby/src/src/pages/posts/2019-09-23-08173.md'
  
  - watchers.js:165 FSWatcher.start
    internal/fs/watchers.js:165:26
  
  - nodefs-handler.js:64 createFsWatchInstance
    [src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:64:15
  
  - nodefs-handler.js:113 setFsWatchListener
    [src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:113:15
  
  - nodefs-handler.js:279 NodeFsHandler._watchWithNodeFs
    [src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:279:14
  
  - nodefs-handler.js:309 NodeFsHandler._handleFile
    [src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:309:23
  
  - nodefs-handler.js:557 NodeFsHandler._addToNodeFs
    [src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:557:21
  

⠹ source and transform nodes
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Command terminated with non-zero exit code. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.

gatsbot · 2019-10-14T17:27:34Z

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

gatsbot · 2019-10-25T17:27:36Z

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

skhaz · 2019-12-29T19:03:11Z

not stale

skhaz · 2019-12-29T19:04:05Z

I have the same problem

UNHANDLED REJECTION ENOSPC: System limit for number of file watchers reached, watch '/home/runner/work/...'



  Error: ENOSPC: System limit for number of file watchers reached, watch '/home/  runner/work/viralizedfy.gatsby/viralizedfy.gatsby/src/posts/images/a-fe-nao-e-  a-estrada.md'
  
  - watchers.js:165 FSWatcher.start
    internal/fs/watchers.js:165:26
  
  - nodefs-handler.js:118 createFsWatchInstance
    [viralizedfy.gatsby]/[chokidar]/lib/nodefs-handler.js:118:15
  
  - nodefs-handler.js:165 setFsWatchListener
    [viralizedfy.gatsby]/[chokidar]/lib/nodefs-handler.js:165:15
  
  - nodefs-handler.js:330 NodeFsHandler._watchWithNodeFs
    [viralizedfy.gatsby]/[chokidar]/lib/nodefs-handler.js:330:14
  
  - nodefs-handler.js:360 NodeFsHandler._handleFile
    [viralizedfy.gatsby]/[chokidar]/lib/nodefs-handler.js:360:23
  
  - nodefs-handler.js:620 NodeFsHandler._addToNodeFs
    [viralizedfy.gatsby]/[chokidar]/lib/nodefs-handler.js:620:21

sidharthachatterjee · 2019-12-30T21:43:11Z

@MatheusRV #11406 (comment)

sumitsg10 · 2019-12-31T17:52:13Z

#shubham gaur
I'm also running a benchmarking, gatsby definitely can't handle with the broad amount of files.

40M src/pages/posts
Number of File: 10000 with Content Size: 0.500 bytes and Min Runs: 10
Benchmark #1: yarn run build
yarn run v1.17.3
$ gatsby build
╔════════════════════════════════════════════════════════════════════════╗
║ ║
║ Gatsby has started collecting anonymous usage analytics ║
║ to help improve Gatsby for all users. ║
║ ║
║ If you'd like to opt-out, you can use gatsby telemetry --disable ║
║ To learn more, check out https://gatsby.dev/telemetry ║
║ ║
╚════════════════════════════════════════════════════════════════════════╝
�[2K�[1A�[2K�[G⠋ open and validate gatsby-configs
�[2K�[1A�[2K�[Gsuccess open and validate gatsby-configs - 0.037 s
⠋ open and validate gatsby-configs
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ load plugins
�[2K�[1A�[2K�[Gsuccess load plugins - 0.257 s
⠋ load plugins
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ onPreInit
�[2K�[1A�[2K�[Gsuccess onPreInit - 0.014 s
⠋ onPreInit
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ delete html and css files from previous builds
�[2K�[1A�[2K�[Gsuccess delete html and css files from previous builds - 0.015 s
⠋ delete html and css files from previous builds
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ initialize the cache
�[2K�[1A�[2K�[Gsuccess initialize cache - 0.015 s
⠋ initialize cache
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ copy gatsby files
�[2K�[1A�[2K�[Gsuccess copy gatsby files - 0.057 s
⠋ copy gatsby files
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ onPreBootstrap
�[2K�[1A�[2K�[Gsuccess onPreBootstrap - 0.010 s
⠋ onPreBootstrap
�[2K�[1A�[2K�[G
�[2K�[1A�[2K�[G⠋ source and transform nodes
�[2K�[1A�[2K�[G⠙ source and transform nodes
�[2K�[1A�[2K�[G⠹ source and transform nodes
�[2K�[1A�[2K�[G
ERROR
UNHANDLED REJECTION ENOSPC: System limit for number of file watchers reached, watch '/opt/gatsby/src/src/pages/posts/2019-09-23-08173.md'
⠹ source and transform nodes
�[2K�[1A�[2K�[G

Error: ENOSPC: System limit for number of file watchers reached, watch '/opt/g atsby/src/src/pages/posts/2019-09-23-08173.md'

watchers.js:165 FSWatcher.start
internal/fs/watchers.js:165:26
nodefs-handler.js:64 createFsWatchInstance
[src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:64:15
nodefs-handler.js:113 setFsWatchListener
[src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:113:15
nodefs-handler.js:279 NodeFsHandler._watchWithNodeFs
[src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:279:14
nodefs-handler.js:309 NodeFsHandler._handleFile
[src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:309:23
nodefs-handler.js:557 NodeFsHandler._addToNodeFs
[src]/[gatsby-source-filesystem]/[chokidar]/lib/nodefs-handler.js:557:21

⠹ source and transform nodes
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Command terminated with non-zero exit code. Use the '-I '/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.
so please help me out of the issue

muescha · 2020-01-01T16:24:04Z

FYI: @pvdz is looking for large sites in #19512

pvdz · 2020-01-06T09:19:17Z

(fyi; I'm a perf engineer at gatsby)

@crock Things have definitely improved, but that may not have been enough just yet. I will reach out to you at some point later when I feel comfortable about this many pages. I don't suppose I could have a look at the build to see what we might improve? It depends on a few factors so no promises, but 160k should be doable, just takes time. If it's a lot of images or sourcing it may not be right now. There are some low hanging fruits that might help perf at scale. Would need to take a look first :)

@MatheusRV @skhaz @sumitsg10 your issue has little to do with Gatsby scaling, although we may want to consider not watching files for very big sites. Regardless, your issue is different from the one reported by @crock. If you can't resolve your issue and you think the problem is with Gatsby either;

see instructions at Error: ENOSPC: System limit for number of file watchers reached, watch '/home/foldername/abcrypto/static' #11406 (or that thread in general) and see if you can unblock the file watcher problem that way, and
see if there's a different issue that is specifically about your problem, or
post a new issue explaining your problem.

I am closing this for now because OP had closed it already. Thank you :)

marcysutton added the type: question or discussion Issue discussing or asking a question about Gatsby label Sep 4, 2019

crock closed this as completed Sep 9, 2019

matbrgz reopened this Sep 23, 2019

gatsbot bot added the stale? Issue that may be closed soon due to the original author not responding any more. label Oct 14, 2019

gatsbot bot closed this as completed Oct 25, 2019

matbrgz reopened this Dec 29, 2019

matbrgz added help wanted Issue with a clear description that the community can help with. and removed stale? Issue that may be closed soon due to the original author not responding any more. labels Dec 29, 2019

pvdz closed this as completed Jan 6, 2020

pvdz self-assigned this Jan 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update on Gatsby Page Limit - Sep 2019 #17321

Update on Gatsby Page Limit - Sep 2019 #17321

crock commented Sep 2, 2019

matbrgz commented Sep 23, 2019

gatsbot bot commented Oct 14, 2019

gatsbot bot commented Oct 25, 2019

skhaz commented Dec 29, 2019

skhaz commented Dec 29, 2019 •

edited

sidharthachatterjee commented Dec 30, 2019

sumitsg10 commented Dec 31, 2019

muescha commented Jan 1, 2020 •

edited

pvdz commented Jan 6, 2020 •

edited

Update on Gatsby Page Limit - Sep 2019 #17321

Update on Gatsby Page Limit - Sep 2019 #17321

Comments

crock commented Sep 2, 2019

matbrgz commented Sep 23, 2019

gatsbot bot commented Oct 14, 2019

gatsbot bot commented Oct 25, 2019

skhaz commented Dec 29, 2019

skhaz commented Dec 29, 2019 • edited

sidharthachatterjee commented Dec 30, 2019

sumitsg10 commented Dec 31, 2019

muescha commented Jan 1, 2020 • edited

pvdz commented Jan 6, 2020 • edited

skhaz commented Dec 29, 2019 •

edited

muescha commented Jan 1, 2020 •

edited

pvdz commented Jan 6, 2020 •

edited