Files inside nested node_modules are cached until restart

[arggh]

Rich Harris shared a "JS lifehack": put common utilities in /src/node_modules or /client/node_modulesetc. and import them easily without the ../../../../../utils/module.js ritual.

It works in Meteor, but changing the code in those files located in this extra node_modules directory won't trigger a refresh in Meteor. The code that was loaded on startup is cached forever until a full restart happens.

Symlinking to /imports doesn't help.

Meteor version used: 1.8.2-beta.16

[arggh]

This might be relevant to what's going on in #10522

[benjamn]

There are two main reasons we don't watch (most) files in node_modules:

1. There are just too many files in most node_modules directories (vastly more than the application itself, typically), so even stat-polling-based watching becomes prohibitively expensive, and fs-event-based watching quickly exhausts the available file descriptors.
2. Compared to application code, we don't expect node_modules files to change very often except when doing an npm install, so picking up the changes on the next restart seemed (at the time this decision was made) to be a good compromise.

I'm not attempting to shoot down this idea, because I agree there are cases where you might be actively editing some files inside node_modules, but the solution cannot be to watch everything in node_modules. Whenever we reach this sort of dilemma, we need to empower the developer to choose which bits within node_modules they want to watch, which is a design problem that requires some creativity and taste.

In this particular case, I suppose we could watch node_modules directories other than / outside of the root node_modules directory, like /client/node_modules/.... That would be a relatively easy change to the shouldWatch function, I think.

As a historical note, we do watch node_modules/linked-package-name if that package directory is a symbolic link (see this line), since that's how npm link installs packages, and presumably npm link-ed packages are under active development (compared to normally-installed packages): #7978

[CaptainN]

I usually just import from root, and am perplexed by recent linter rules showing up in my configuration which prohibit that. A statement like import { thing } from '/imports/utils/thing' is infinitely more readable/understandable/useful than import { thing } from '../../../../../utils/thing.js'.

For watching - I noticed that even node_modules listed to in meteor -> nodeModules -> recompile are not watched. If that were changed, maybe that can be a way to add a watch exception?

[benjamn]

For watching - I noticed that even node_modules listed to in meteor -> nodeModules -> recompile are not watched. If that were changed, maybe that can be a way to add a watch exception?

That's definitely doable! We could also introduce a meteor.nodeModules.watch list, which could simply be a list of package names, without the additional architecture stuff that comes with meteor.nodeModules.recompile. I'm inclined to say recompile should imply watch, but not the other way around.

This may just be me rationalizing a new configuration option, but it's nice when a parent section has at least two possible sub-sections. Right now meteor.nodeModules.recompile is the only reason you'd need a meteor.nodeModules section, so adding meteor.nodeModules.watch would help flesh it out a bit?

[arggh]

In this particular case, I suppose we could watch node_modules directories other than / outside of the root node_modules directory, like /client/node_modules/.... That would be a relatively easy change to the shouldWatch function, I think.

Not exactly my area of expertise to consider all the weird and exciting patterns developers use in the wild, but I'd assume that for most Meteor apps, watching node_modules other than the root node_modules would be a zero cost operation, since those folders just don't exist?

I would also be happy to just configure the paths to the watched modules.

We could also introduce a meteor.nodeModules.watch list, which could simply be a list of package names, without the additional architecture stuff that comes with meteor.nodeModules.recompile

In that case, what do you think about allowing globs or patterns to be defined for meteor.nodeModules.watch ?

"meteor": {
  "nodeModules": {
    "watch": [
       "client/node_modules/**/*"
    ]
  }
}

Would be nicer, than keeping an up-to-date list of folders (that is actually application code). At least I would, from time to time, with 100% certainty, end up wondering why changes are not picked up when I forget to add an entry to package.json when new modules are introduced.

---

→ From the two options, I'm in favor of just treating node_modules outside of the root folder as application code.

[CaptainN]

@benjamn If we are looking for rationalizations for the "nodeModules" json node, how about adding package aliases! It'd be great in some cases to replace react (and react-dom) with preact (and preact-compat) client side.

[benjamn]

This should be fixed if you run meteor update --release 1.8.2-rc.0. I went with the strategy of watching node_modules directories that are not directly in the root application directory, so no additional configuration should be necessary.

Please feel free to reopen this issue if you continue to encounter similar problems, and thanks for all of your suggestions above.