Optimistic caching increases CPU load on idle a lot in huge codebases

[maxnowack]

At TeamGrid, we have a huge codebase with a lot of npm dependencies. The build process takes long, but I think that is pretty normal for an application of that size.
When the build is done and the application is idling, the CPU load is still very high (~100%).
This only occurs in a dev environment and not in production. So it's a minor issue for us.
The downsides only were a hot macbook and a short battery life 😉
We have this strange behavior for a long time and I didn't had the time for debugging what exactly is going on after the build.

I've analyzed the issue a few days ago and posted my result in another issue.

Today I've dived a little bit deeper and figured out that Meteor watches a lot of files with strange paths which aren't exist. For example:
/ … /client/views/analytics/node_modules/babel-plugin-styled-components.html
It seems that meteor is watching files for every npm module, with every registered file extensions in every folder and sub folder of our application. Regardless wether the file exists or not.
In huge codebases like ours, this results in a high cpu utilization, since every file is polled with a specific interval.

If I set METEOR_DISABLE_OPTIMISTIC_CACHING=1, Meteor doesn't watch these files and the cpu load stays normal. But I'm not sure what exactly the environment variable disables and what the downside are.
What exactly is the intent of the optimistic caching?

I've seen this behavior on macOS . I haven't tested, if the issue also exists in other environments.

[benjamn]

What version of Meteor are you using? In Meteor 1.5.1, we made a change to avoid using native file watchers unless a file had been changed at least once, and instead use only fs.watchFile polling (with a 5000ms interval) to detect the first change.

If you're not on Meteor 1.5.1 (at least) yet, then this issue is a good reason to consider running meteor update sooner rather than later.

If you're using a more recent version of Meteor, my guess is that the polling is happening too frequently given the number of files being watched, even though it's only happening once every five seconds for each file. That lines up with your observation of high CPU usage, since polling is CPU-intensive, whereas native file watching is only file-descriptor-intensive.

To help diagnose the problem, you can set

METEOR_WATCH_POLLING_INTERVAL_MS=10000

(or even higher) to poll less frequently, while leaving METEOR_DISABLE_OPTIMISTIC_CACHING unset (so that you still get the rebuild performance benefits of the caching system). Of course, this means you'll have to wait longer for changes to be detected the first time you modify a given file after starting Meteor, but at least it will help diagnose the source of the problem. I think you may be right that Meteor is polling too many files, so there's plenty of reason to believe this can be improved.

[maxnowack]

I'm using v1.6.1 but the issue exists since about v1.4
I'll play around with the polling interval and see if I still can optimize the cpu load.
One other strange behavior is, that the initial build time halves if METEOR_DISABLE_OPTIMISTIC_CACHING is set.
The rebuild time decreases from 47 seconds to 8 (!) seconds.

[benjamn]

Ah yeah, we were doing the polling in 1.4.x as well, so that behavior would be the same before and after Meteor 1.5.1. Thanks for the update.

What were the env variables you used for the 8sec rebuild times?

[maxnowack]

METEOR_DISABLE_OPTIMISTIC_CACHING=1, TOOL_NODE_FLAGS="--max-old-space-size=4096" and of course METEOR_PROFILE=1

[maxnowack]

I've created a reproduction in this repository: https://github.com/maxnowack/meteor-file-watching-performance
It contains the default meteor example app plus a bunch of npm dependencies and randomly generated folders with an empty test.js file.

If you run meteor within the directory of the app, you'll see that the cpu load is very high after the build has finished.
You can run dtruss on the node process with the high load to easily see the name of the files meteor tries to watch.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed it has not had recent activity.