Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maybe warn if we're inadvertently watching loads of files #1317

Labels

Comments

@stuartlangridge
Copy link

  • nodemon -v: 1.17.3
  • node -v: v6.9.5
  • Operating system/terminal environment: Ubuntu 16.04
  • Command you ran: nodemon --exec "node -e 'console.log(2)'"

I had a large folder with (recursively) some tens of thousands of files in it in the current folder, and I wasn't ignoring it (because I'd forgotten that it was there). Nodemon was refusing to restart when I changed a file, and I thought it was broken. After adding a bunch of debug prints to nodemon's lib/monitor/watch.js, it became apparent that it was taking many tens of seconds to add all these files to chokidar's watch, and before that's complete, chokidar doesn't fire change events (and therefore nodemon doesn't get told that a file had changed, and therefore doesn't restart). This is of course not actually nodemon's fault. But perhaps if after a couple of seconds or so, chokidar still isn't finished adding files, maybe nodemon could print a warning ("Warning: not yet monitoring for changes; still scanning files in ./largefolder"), and then print "Now monitoring files" once scanning is complete, or similar?

Since this isn't something that someone would explicitly turn on (if you know you need to turn it on, then you don't need it!), it would need to be something on by default, and that may not be something you want, of course.

@remy
Copy link
Owner

remy commented Apr 19, 2018

This used to be in place, but I had a quick scan through the code and I couldn't find it anymore…

@stuartlangridge
Copy link
Author

stuartlangridge commented Apr 19, 2018

I would have thought that stuartlangridge@0671614 would have worked, but inexplicably it doesn't; the setTimeout never fires, and I don't understand why.

(To replicate, make a folder with loads of files to be monitored in it; I just did for i in $(seq 1 100); do for j in $(seq 1 100); do cp whatever.js hugefolder/file-$i-$j.js; done; done to get 10,000 JS files which nodemon thinks it needs to care about.)

@remy
Copy link
Owner

remy commented Apr 20, 2018

I wonder if the CPU blocks during the add, and thus the timeout never gets the chance, then the ready event is fired. Seems odd that it would be synchronous though…

@stale
Copy link

stale bot commented May 4, 2018

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

@stale stale bot added the stale no activity for 2 weeks label May 4, 2018
remy added a commit that referenced this issue May 5, 2018
Fixes #1317

Originally a new Set was generated upon every individual file (or dir)
watched. This typically doesn't take a long time, but if there's 10,000
files to watch, it's very, very slow. Moving this logic out to the ready
event instead, cuts all the processing time of the watch right down.
@remy remy added has PR and removed stale no activity for 2 weeks labels May 5, 2018
@remy
Copy link
Owner

remy commented May 5, 2018

This turned out to be a perf bug. Simple to replicate, simple to fix. nodemon should have no problem watching 10,000 files - this fix changes it so it can again.

@remy remy closed this as completed in #1335 May 5, 2018
remy added a commit that referenced this issue May 5, 2018
Fixes #1317

Originally a new Set was generated upon every individual file (or dir)
watched. This typically doesn't take a long time, but if there's 10,000
files to watch, it's very, very slow. Moving this logic out to the ready
event instead, cuts all the processing time of the watch right down.
jimthedev pushed a commit to commitizen/cz-cli that referenced this issue May 24, 2018
This Pull Request updates dependency [nodemon](https://github.com/remy/nodemon) from `v1.11.0` to `v1.17.5`



<details>
<summary>Release Notes</summary>

### [`v1.17.5`](https://github.com/remy/nodemon/releases/v1.17.5)
[Compare Source](remy/nodemon@ff79835...v1.17.5)
##### Bug Fixes

* in watch, use fully filtered ignore rules ([b3fc3a9](remy/nodemon@b3fc3a9)), closes [#&#8203;1348](`remy/nodemon#1348)

---

### [`v1.17.4`](https://github.com/remy/nodemon/releases/v1.17.4)
[Compare Source](remy/nodemon@6925494...v1.17.4)
##### Bug Fixes

* {{filename}} is only replaced once ([f616258](remy/nodemon@f616258))
* increase perf of watching large file count ([58b82f2](remy/nodemon@58b82f2)), closes [#&#8203;1317](`remy/nodemon#1317)
* only hook SIGINT on boot - not at require ([9bbc219](remy/nodemon@9bbc219)), closes [#&#8203;1326](`remy/nodemon#1326)
* support env.SUPPRESS_SUPPORT ([#&#8203;1334](`remy/nodemon#1334)) ([ee2aac1](remy/nodemon@ee2aac1))

---

### [`v1.17.3`](https://github.com/remy/nodemon/releases/v1.17.3)
[Compare Source](remy/nodemon@v1.17.2...v1.17.3)
##### Bug Fixes

* don't throw when required in the repl ([aa18c80](remy/nodemon@aa18c80)), closes [#&#8203;1292](`remy/nodemon#1292)

---

### [`v1.17.2`](https://github.com/remy/nodemon/releases/v1.17.2)
[Compare Source](remy/nodemon@v1.17.1...v1.17.2)
##### Bug Fixes

* prevent throw when args missing ([#&#8203;1288](`remy/nodemon#1288)) ([89d6062](remy/nodemon@89d6062)), closes [#&#8203;1286](`remy/nodemon#1286)
* watch count regression ([#&#8203;1287](`remy/nodemon#1287)) ([372e6b2](remy/nodemon@372e6b2)), closes [#&#8203;1283](`remy/nodemon#1283)

---

### [`v1.17.1`](https://github.com/remy/nodemon/releases/v1.17.1)
[Compare Source](remy/nodemon@v1.17.0...v1.17.1)
##### Bug Fixes

* throwing exeception on run ([85ed19d](remy/nodemon@85ed19d)), closes [#&#8203;1276](`remy/nodemon#1276)

---

### [`v1.17.0`](https://github.com/remy/nodemon/releases/v1.17.0)
[Compare Source](remy/nodemon@v1.16.1...v1.17.0)
##### Bug Fixes

* make it possible for windows to checkout ([#&#8203;1270](`remy/nodemon#1270)) ([0f39b2e](remy/nodemon@0f39b2e)), closes [#&#8203;1255](`remy/nodemon#1255)
* windows exec resolution ([#&#8203;1274](`remy/nodemon#1274)) ([7ffd545](remy/nodemon@7ffd545)), closes [#&#8203;1251](`remy/nodemon#1251)
* make clear where more opts are in help ([#&#8203;1271](`remy/nodemon#1271)) ([f4391d4](remy/nodemon@f4391d4))
* put windows drive letter tweak in right place ([#&#8203;1272](`remy/nodemon#1272)) ([94b526f](remy/nodemon@94b526f)), closes [#&#8203;1263](`remy/nodemon#1263)
##### Features

* try to resolve exec in node_modules/.bin ([#&#8203;1275](`remy/nodemon#1275)) ([7fb365d](remy/nodemon@7fb365d)), closes [#&#8203;1268](`remy/nodemon#1268)
* feed args to exec when detecting script ([#&#8203;1273](`remy/nodemon#1273)) ([e41f3c3](remy/nodemon@e41f3c3)), closes [#&#8203;1263](`remy/nodemon#1263)

_Note: due to an oddity in the automated build process, nodemon was bumped two minor versions instead of one. Nothing to worry about though._

---

### [`v1.16.1`](remy/nodemon@v1.16.0...v1.16.1)
[Compare Source](remy/nodemon@v1.16.0...v1.16.1)


---

### [`v1.16.0`](remy/nodemon@1cda8fa...v1.16.0)
[Compare Source](remy/nodemon@1cda8fa...v1.16.0)


---

### [`v1.15.1`](https://github.com/remy/nodemon/releases/v1.15.1)
[Compare Source](remy/nodemon@v1.15.0...v1.15.1)
##### Bug Fixes

* ensure directories are watched, not files ([#&#8203;1260](`remy/nodemon#1260)) ([1cda8fa](remy/nodemon@1cda8fa)), closes [#&#8203;1259](`remy/nodemon#1259)

---

### [`v1.15.0`](https://github.com/remy/nodemon/releases/v1.15.0)
[Compare Source](remy/nodemon@v1.14.12...v1.15.0)
##### Features

* add support for `--spawn` ([#&#8203;1249](`remy/nodemon#1249)) ([5e88b04](remy/nodemon@5e88b04)), closes [#&#8203;1245](`remy/nodemon#1245)

---

### [`v1.14.12`](https://github.com/remy/nodemon/releases/v1.14.12)
[Compare Source](remy/nodemon@v1.14.11...v1.14.12)
##### Bug Fixes

* sniff for child.stdout before using it ([79e61f0](remy/nodemon@79e61f0))
* wrongly normalizing slashes in windows ([33fa6f4](remy/nodemon@33fa6f4)), closes [#&#8203;1236](`remy/nodemon#1236)

---

### [`v1.14.11`](https://github.com/remy/nodemon/releases/v1.14.11)
[Compare Source](remy/nodemon@v1.14.10...v1.14.11)
##### Bug Fixes

* don't ignore dot-directories ([4be493c](remy/nodemon@4be493c)), closes [#&#8203;1223](`remy/nodemon#1223)
* update deps - chokidar@&#8203;2.0.0 in particular ([9d49852](remy/nodemon@9d49852))

---

### [`v1.14.10`](https://github.com/remy/nodemon/releases/v1.14.10)
[Compare Source](remy/nodemon@v1.14.9...v1.14.10)
##### Bug Fixes

* node < 6.4.0 causing crash on 'rs' trigger ([e90f15a](remy/nodemon@e90f15a)), closes [#&#8203;1218](`remy/nodemon#1218)

---

### [`v1.14.9`](https://github.com/remy/nodemon/releases/v1.14.9)
[Compare Source](remy/nodemon@64a82ff...v1.14.9)
##### Bug Fixes

* correctly pass ignored rules to chokidar ([718a9ad](remy/nodemon@718a9ad)), closes [#&#8203;1202](`remy/nodemon#1202)
* ignorePermissionErrors in chokidar ([e95ea6f](remy/nodemon@e95ea6f))

---

### [`v1.14.8`](https://github.com/remy/nodemon/releases/v1.14.8)
[Compare Source](remy/nodemon@6e7ce4b...v1.14.8)
##### Bug Fixes

* clarify which config files are actually used ([2582d96](remy/nodemon@2582d96)), closes [#&#8203;1204](`remy/nodemon#1204)
* fail gracefully if postinstall fails ([64a82ff](remy/nodemon@64a82ff))

---

### [`v1.14.7`](https://github.com/remy/nodemon/releases/v1.14.7)
[Compare Source](remy/nodemon@v1.14.6...v1.14.7)
##### Bug Fixes

* swallow child.stdin err ([6e7ce4b](remy/nodemon@6e7ce4b)), closes [#&#8203;1195](`remy/nodemon#1195)

---

### [`v1.14.6`](https://github.com/remy/nodemon/releases/v1.14.6)
[Compare Source](remy/nodemon@v1.14.5...v1.14.6)
##### Bug Fixes

* watch both js and mjs files if main file is JavaScript ([d78bf3d](remy/nodemon@d78bf3d))

---

### [`v1.14.5`](https://github.com/remy/nodemon/releases/v1.14.5)
[Compare Source](remy/nodemon@v1.14.4...v1.14.5)
##### Bug Fixes

* don't use fork with `node inspect` ([0d9a892](remy/nodemon@0d9a892))

---

### [`v1.14.4`](https://github.com/remy/nodemon/releases/v1.14.4)
[Compare Source](remy/nodemon@v1.14.3...v1.14.4)
##### Bug Fixes

* expose no update flag ([c637717](remy/nodemon@c637717))
* handle exit(2) better ([5a914cb](remy/nodemon@5a914cb))
* incorrect count of watch files being reported ([7a04e2c](remy/nodemon@7a04e2c)), closes [#&#8203;1194](`remy/nodemon#1194)
* properly handle --no-update-notifier ([48048aa](remy/nodemon@48048aa))
* support implicit filename extension ([6e839d4](remy/nodemon@6e839d4)), closes [#&#8203;1193](`remy/nodemon#1193)

---

### [`v1.14.3`](https://github.com/remy/nodemon/releases/v1.14.3)
[Compare Source](remy/nodemon@v1.14.2...v1.14.3)
##### Bug Fixes

* ensure non-slurp is passed to script ([ad226af](remy/nodemon@ad226af)), closes [#&#8203;750](`remy/nodemon#750)

---

### [`v1.14.2`](https://github.com/remy/nodemon/releases/v1.14.2)
[Compare Source](remy/nodemon@v1.14.1...v1.14.2)
##### Bug Fixes

* stop saying there's another update required! ([dcd8911](remy/nodemon@dcd8911))

---

### [`v1.14.1`](https://github.com/remy/nodemon/releases/v1.14.1)
[Compare Source](remy/nodemon@v1.14.0...v1.14.1)
##### Bug Fixes

* defined diretory watching ([1e2516d](remy/nodemon@1e2516d))
* pass stdin to child process on -I ([1d88943](remy/nodemon@1d88943)), closes [#&#8203;1036](`remy/nodemon#1036)
* properly ignore defaults, don't match partial ([4589bc8](remy/nodemon@4589bc8)), closes [#&#8203;916](`remy/nodemon#916)

---

### [`v1.14.0`](https://github.com/remy/nodemon/releases/v1.14.0)
[Compare Source](remy/nodemon@v1.13.3...v1.14.0)
##### Bug Fixes

* remove scoped dep ([d15cf68](remy/nodemon@d15cf68)), closes [#&#8203;1180](`remy/nodemon#1180)
##### Features

* support wildcard extension matching ([009d868](remy/nodemon@009d868))

---

### [`v1.13.3`](https://github.com/remy/nodemon/releases/v1.13.3)
[Compare Source](remy/nodemon@v1.13.2...v1.13.3)
##### Bug Fixes

* nodemon reporting crash ([1171fc1](remy/nodemon@1171fc1)), closes [#&#8203;1173](`remy/nodemon#1173)

---

### [`v1.13.2`](https://github.com/remy/nodemon/releases/v1.13.2)
[Compare Source](remy/nodemon@v1.13.1...v1.13.2)
##### Bug Fixes

* switch to killing using numericals ([e9129c0](remy/nodemon@e9129c0)), closes [#&#8203;956](`remy/nodemon#956) [#&#8203;813](`remy/nodemon#813)

---

### [`v1.13.1`](https://github.com/remy/nodemon/releases/v1.13.1)
[Compare Source](remy/nodemon@v1.13.0...v1.13.1)
##### Bug Fixes

* fixed an issue where we tried to resolve null as path ([10ded94](remy/nodemon@10ded94))
* if no ps, walk /proc to kill child fully ([bf9b7a6](remy/nodemon@bf9b7a6))

---

### [`v1.13.0`](https://github.com/remy/nodemon/releases/v1.13.0)
[Compare Source](remy/nodemon@v1.12.7...v1.13.0)
##### Bug Fixes

* support nodemon index to expand to index.js ([a282afb](remy/nodemon@a282afb)), closes [#&#8203;1165](`remy/nodemon#1165)
##### Features

* support SIGHUP to restart nodemon ([30f999a](remy/nodemon@30f999a)), closes [#&#8203;393](`remy/nodemon#393)
* watch script regardless of extension ([f8a5abe](remy/nodemon@f8a5abe)), closes [#&#8203;461](`remy/nodemon#461)

---

### [`v1.12.7`](https://github.com/remy/nodemon/releases/v1.12.7)
[Compare Source](remy/nodemon@v1.12.6...v1.12.7)
##### Bug Fixes

* defensive check on accessing cmd.args ([fd961d6](remy/nodemon@fd961d6))
* support node arguments again ([d9e93ad](remy/nodemon@d9e93ad)), closes [#&#8203;1161](`remy/nodemon#1161)

---

### [`v1.12.6`](https://github.com/remy/nodemon/releases/v1.12.6)
[Compare Source](remy/nodemon@v1.12.5...v1.12.6)
##### Bug Fixes

* get tests to pass ([cf923a8](remy/nodemon@cf923a8))
* only use fork if node>4 ([f52615c](remy/nodemon@f52615c))
* restore node 4 min support ([6d760f4](remy/nodemon@6d760f4))
* travis CI build before-install ([3a6b8df](remy/nodemon@3a6b8df))

---

### [`v1.12.5`](https://github.com/remy/nodemon/releases/v1.12.5)
[Compare Source](remy/nodemon@v1.12.4...v1.12.5)
##### Bug Fixes

* pass through execArgs from config ([#&#8203;1142](`remy/nodemon#1142)) ([6a4fb22](remy/nodemon@6a4fb22))

---

### [`v1.12.4`](https://github.com/remy/nodemon/releases/v1.12.4)
[Compare Source](remy/nodemon@v1.12.3...v1.12.4)
##### Bug Fixes

* reduce help output (a teeny bit) ([2e1b496](remy/nodemon@2e1b496))

---

### [`v1.12.3`](https://github.com/remy/nodemon/releases/v1.12.3)
[Compare Source](remy/nodemon@24a4b84...v1.12.3)
##### Bug Fixes

* add commit lint ([2af6391](remy/nodemon@2af6391))
* Add support for multi-level filenames ([29a9a44](remy/nodemon@29a9a44))
* bump deps and fix tests for local dev ([#&#8203;1139](`remy/nodemon#1139)) ([6d57dac](remy/nodemon@6d57dac))
* catch module.parent.filename === undefined ([#&#8203;1053](`remy/nodemon#1053)) ([4c81e9a](remy/nodemon@4c81e9a))
* crash when passing argument to node (windows) ([469fa80](remy/nodemon@469fa80)), closes [#&#8203;1095](`remy/nodemon#1095)
* executable path handling under windows ([#&#8203;962](`remy/nodemon#962)) ([481dc8f](remy/nodemon@481dc8f))
* exit when fail to parse the config file ([#&#8203;921](`remy/nodemon#921)) ([63e8606](remy/nodemon@63e8606)), closes [#&#8203;651](`remy/nodemon#651)
* exit with code 1 on crash if --exitcrash ([#&#8203;946](`remy/nodemon#946)) ([facc8cb](remy/nodemon@facc8cb))
* help truncation on node 6.2 ([#&#8203;842](`remy/nodemon#842)) ([#&#8203;843](`remy/nodemon#843)) ([abc138f](remy/nodemon@abc138f))
* read config file before defaulting script parameter ([#&#8203;1110](`remy/nodemon#1110)) ([f3e0c29](remy/nodemon@f3e0c29))
* require node 6 or above in engines ([71eab1a](remy/nodemon@71eab1a))
* Support for .mjs es6 modules ([#&#8203;1138](`remy/nodemon#1138)) ([5a89df6](remy/nodemon@5a89df6))
* support signal on CLI ([#&#8203;1061](`remy/nodemon#1061)) ([3c352f2](remy/nodemon@3c352f2))
* sync help.txt to available options ([9b6c786](remy/nodemon@9b6c786))
* update readme ([3426224](remy/nodemon@3426224))
##### Features

* nodemonConfig support in package.json ([fb5da38](remy/nodemon@fb5da38)), closes [#&#8203;873](`remy/nodemon#873)

---

### [`v1.12.2`](remy/nodemon@469fa80...24a4b84)
[Compare Source](remy/nodemon@469fa80...24a4b84)


---

### [`v1.12.1`](remy/nodemon@6a4803d...469fa80)
[Compare Source](remy/nodemon@6a4803d...469fa80)


---

### [`v1.12.0`](remy/nodemon@8204b69...6a4803d)
[Compare Source](remy/nodemon@8204b69...6a4803d)


---

### [`v1.11.1`](remy/nodemon@v1.11.0...8204b69)
[Compare Source](remy/nodemon@v1.11.0...8204b69)


---

</details>




---

This PR has been generated by [Renovate Bot](https://renovatebot.com).
lowsky pushed a commit to lowsky/spotify-graphql-server that referenced this issue Jun 19, 2018
This Pull Request updates dependency [nodemon](https://github.com/remy/nodemon) from `v1.17.3` to `v1.17.5`



<details>
<summary>Release Notes</summary>

### [`v1.17.5`](https://github.com/remy/nodemon/releases/v1.17.5)
[Compare Source](remy/nodemon@v1.17.4...v1.17.5)
##### Bug Fixes

* in watch, use fully filtered ignore rules ([b3fc3a9](remy/nodemon@b3fc3a9)), closes [#&#8203;1348](`remy/nodemon#1348)

---

### [`v1.17.4`](https://github.com/remy/nodemon/releases/v1.17.4)
[Compare Source](remy/nodemon@v1.17.3...v1.17.4)
##### Bug Fixes

* {{filename}} is only replaced once ([f616258](remy/nodemon@f616258))
* increase perf of watching large file count ([58b82f2](remy/nodemon@58b82f2)), closes [#&#8203;1317](`remy/nodemon#1317)
* only hook SIGINT on boot - not at require ([9bbc219](remy/nodemon@9bbc219)), closes [#&#8203;1326](`remy/nodemon#1326)
* support env.SUPPRESS_SUPPORT ([#&#8203;1334](`remy/nodemon#1334)) ([ee2aac1](remy/nodemon@ee2aac1))

---

</details>




---

This PR has been generated by [Renovate Bot](https://renovatebot.com).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment