Windows-only: npm global cache location violaties Windows SW design guide, problematic in a domain environment with roaming

[kkm000]

• Version: 10.22.00
• Platform: Windows 10 Pro in a domain environment, v10.0.18363, x64
• Subsystem: npm

What steps will reproduce the bug?

1. Install a large global package, e.g. npm i --global ngrok
2. Open cmd and list files as dir %APPDATA%\npm-cache (or open explorer with Win+E, press Alt+D, paste same path, press Enter).
3. Observe that the folder contains in excess of ten of megabytes of cached data.

This behavior is as documented: https://docs.npmjs.com/using-npm/config#cache

Explanation

But it is detrimental to domain functionality, and goes against Windows software design guidelines.

I can provide all necessary links, but in short, the parent directory of %APPDATA% contains two important directories: Roaming/ (this is where the env var points to), and Local/ (the env variable for it is %LOCALAPPDATA%), for example,

APPDATA=C:\Users\kkm\AppData\Roaming
LOCALAPPDATA=C:\Users\kkm\AppData\Local

This has been this way since Windows 2000 if not NT4. Despite their similarity, the semantic difference between the two is crucial. The names of the directories are telling.

• The local directory is sedentary on the machine it started it life on. This directory is used for various caches, log files, and other data not essential to program function. In fact, programs should not expect that this directory is consistent with their saved roaming state. The local directory is the one housing the per-user temporary directory: `TEMP=C:\Users\kkm\AppData\Local\Temp'.
• The roaming directory is very special. It houses data that are hosted and synchronized with the domain infrastructure. This is similar to NFS home directories in *nix, except the files are actually stored on the local drive, and are synchronized (akin to rsync). This ensures that the users set to "roam" in the domain, i.e. receiving their normal working environment when logging on into a domain machine they never logged before. I must note that a program need to be present on the new machine; it's only the data and settings that roam. A program is expected to restore (re-download, re-generate etc) it's local data counterpart if it's not present.

Examples of programs following the guidelines

Let me give an example of Mozilla Firefox following the guidelines correctly. If you examine %APPDATA%\Mozilla\Firefox\Profiles\<profile-name>, you'll find files such as user's preferences, bookmarks, browsing history, extensions etc: anything that allows the same experience "out of the box" on a different computer (provided Firefox is installed there). On the other hand, in the %LOCALAPPDATA%\Mozilla\Firefox\Profiles\<profile-name> there are files that are expendable: various caches, that will be rebuild transparently during browsing. When Firefox lights up that "Restart to update" icon, the update has been already uploaded to the Local directory.

There are programs, like Google Chrome, that self-update to their latest version by storing the whole kaboodle under the Local directory and run directly from there; what is actually installed in %PROGRAMFILES% is only a stub, possibly downloading the complete program and then launching it. This way the programs avoid asking administrator permission to update, and allow individual users update at their preferred time, not necessarily update the program on the whole machine for all users.

Obviously, storing things under %APPDATA% that do not belong there is worsening user experience (e.g., first login time), as synchronization of roaming profiles takes significantly longer (just for the ballpark, by current size of the Roaming directory is ~2GB, and Local ~4GB. If there were no distinction between the two, the amount of data to be provisioned on a fresh login would triple).

Misuse of %APPDATA% by software suppliers, including node and npm

I won't name name names, but there are major companies ignoring these guidelines. I can compare that to writing your log files to /etc or the root filesystem: making trouble and asking for trouble at the same time, with no reward obtained. Even for those, I see a trend going in the right direction: If there are standards, chances are someone would come across them in 20 years by pure chance, facepalm and try to fix the damage (I believe members of our guild are acting in good faith, until proved otherwise).

Having said that,

1. npm is a major misuser of %APPDATA%, storing often hundreds of megabytes of data in the roaming directory. The caches must be relocated to %LOCALAPPDATA%, as in today's Internet they are clearly expendable.
2. node stores its debug log under %APPDATA%. While this is also not a correct behavior, the amount of stored information is much smaller, and the older logs are deleted by node executable itself. Would be nice to have this fixed too.

Expected resolution

Please store the temporary or expendable files in %LOCALAPPDATA, and the user-configured, persistent files in %APPDATA%. The criterion that can be used is: "Will the user experience major loss of functionality, data or configuration if these files are deleted?" (minor inconvenience of waiting for a cache to rebuild does not count as a major functionality loss). If the answer is yes, the files must go to %APPDATA%. If no, %LOCALAPPDATA% is the right place for them.

Workaround

npm cache storage is configurable per user: https://docs.npmjs.com/using-npm/config#cache. node impact is not significant to work around.

How often does it reproduce? Is there a required condition?

Always.

• Default npm configuration is a required condition (no above workaround applied).
• node seems to use a hardcoded location, so the behavior is not preconditioned.

Impact Analysis

Minor. Only specific accounts roam in s domain environment, and this is not an often used feature (IT is reluctant of supporting it because many programs violate the design guidelines, or may not even be aware they exist. Storing, transmitting and maintaining 2GB per user may require upgrade to the infrastructure.)

However, I believe that fixing this unintentional mishap is a good start. Fixing one program at a time (especially starting with the major violators such as npm) is already a 10% reduction in the roaming profile size. It's a low-hanging fruit, and quite a few people, me included, would be quite grateful for this fix, which is likely not a very extensive change.

No-blame and good faith

I want to deeply thank you for the good work. Having been in the Windows world in an out since Windows 2.0, I know that there are a whole library of documents about using Windows API and conventions, and I know firsthand how few people even know about their existence. It's not a fault of any node developer; Microsoft could have done a better job promoting the awareness of these standards to multiplatform developers. In any case, there is no, and cannot be a blame for good faith mistakes. When I say "npm abuses X", this means only what it means: that it contains a design error that leads to a severe misuse of the OS feature X and needs to be fixed, with no implications of blame on anyone or other unsound inferences beyond the simple statement.

[addaleax]

While this is a thorough and helpful write-up, there may be some things to clarify:

• We’re not in control over npm’s code. We ship it, but any changes to the npm code will have to be brought up with that project itself. From the sounds of it, this also seems like the kind of problem where a pull request might be helpful.
• Node.js does not perform any I/O on its own by default, and has no concept of file-based debug logs. You may also be referring to npm’s debug logs here?

[DerekNonGeneric]

I've known about this for quite some time. It has never bothered me.

If it's something you're interested in seeing changed, you'll need to reach out to the appropriate teams.

Here's another reference, which is where I first learned about it myself:

• https://blog.codinghorror.com/dont-pollute-user-space/

[kkm000]

Thank you guys for a clear and detailed response. I was under impression that both Node.js and npm were two parts of one big thing. Is npm/cli the place, if you happen to know? I should re-open the issue there.

@addaleax, yep, my bad, dead-on, I was referring to npm debug logs. I still have hard time mentally separating the two. And I certainly hear you about the pull request! :) Oh, BTW, TIL about the Wagner-Fischer algorithm, thanks! The thing is, I mostly do HPC computing in the cloud, and use Node only for automation and integration.

@DerekNonGeneric, Windows, unfortunately, carries a lot of refactoring baggage, so what goes where is a huge mess; probably much worse than what already qualifies as a coding horror by that blog . They did away with directories like "My Documents", and changed most of them to space-less "Documents". But the horrible "Program Files" and "Program Files(x86)" are still alive and kicking; how did I only survive without parentheses in directory names to the day they arrived. Of course, there are hidden links with old names, like this. Programs for Window 95 are still supposed to run... more or less. This is from dir /a:

19-06-25  06:32    <JUNCTION>     Application Data [C:\Users\kkm\AppData\Roaming]

It had always bemused me before I happened to pick up a copy of The Old New Thing by Raymond Chen. It cleared up a lot of misconceptions of why and how Windows came to be the way it is. I wish the Linux kernel team had a 30-year vet and raconteur like him. :)

Ah, and here's my trophy from the recent Windows 10 (HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders):

[mwisnicki]

The npm part was fixed: npm/cli#899

[DerekNonGeneric]

Thanks, looks like this can be closed now.