Yarn cache will fill your PC when used with local (file:) packages

[hoegge]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Whenever a local package is rebuild, yarn creates a new hash version in the cache, which is good, since in prior versions, the old version was just used irrespective of making a new version of the package.
But, this eventually fills your hard-drive. Today I found out my yarn cache was around 156 GB and it took me 1,5 hour to delete it! About 300-600 copies of each package (4500 cached folder in total). That is not very smart

If the current behavior is a bug, please provide the steps to reproduce.

1. Create a local package
2. Install it in another project using yarn
3. Make a new version of the package
4. Install it in another project using yarn
5. Check your yarn cache - you now have two versions of the package including all its dependencies in your cache.
   Eventually your hard drive will be full, and you'll find out when there is no room left and it takes you hours to delete the cache folder.

If the same local packages is used in, e.g., 10 different places in your solution, then 10 different hash versions will be created in the cache.

What is the expected behavior?
When a new version of a local package is created and installed/updated/used by any project by yarn, the new version / hash should replace the old version.

Please mention your node.js, yarn and operating system version.
Node: 8.11.3
Yarn: 1.7.0
Win 10 Pro 1709

[hoegge]

It really doesn't make sense at all to cache local packages, unless I've overlooked something. You put a copy on the disk and then afterwards you copy from that cache to the disk. Actually yarn should instead support an option do symlink to local packages, so we wouldn't have to copy first and run linklocal afterwards.

[hoegge]

Actually, it seems like yarn makes a new hash of a local package not only when it has changed (which COULD make sense in some scenarios), but it also calculates a new hash and cache it again just when it is included by another application / solution with a package.json. Even when the contents of the folder is the EXACT same - checked with sha1 and md5 it caches a new version with another hash. Does it use the file access date in the sha1 hash also? (which is the only thing I can think of changes)

[line0]

Still present in yarn 1.9.5.

Take note this happens even when the local packages in question have been linked using yarn link, which doesn't make any sense whatsoever and we inevitably amass millions of cached files that are never even used once.

Worse yet, this also ends up slowing down package installation to a crawl even if installing a totally different package because yarn always creates fresh cached copies for all local packages referenced in dependencies/devdependencies. Since every copy includes a full copy of the packages' node_modules folder, things go south even faster.

[hasparus]

This is of course my local module cached gazillion times.
Yarn: 1.9.4

[deyan-a]

Mine is currently 57GB. Was starting to lose my mind where all my storage space was going

[fungilation]

Please PLEASE just stop caching local packages.

[Globegitter]

I am just running into an issue where I want to install a local package that itself is symlinked and the symlinks are fine in the cache but then when they get copied over to the real node_module they are incorrect as they get copied directly from the cache rather then new symlinks created.

[nojvek]

I just realized our Travis CI build was always finding some changed files in yarn cache and re-uploading.

Overtime things were just getting slower and slower downloading and uploading the cache.

Please please please fix it. In the name of the environment at least, the amount of energy wasted because of this bug 🐛 is probably worth a couple of trees.

[hoegge]

Had forgotten about this and suddenly had a full hard drive. Almost half of my drive was yarn's cache of local modules:

And it takes about 30-60 minutes for yarn to clear the cache - yawn!

[Haemp]

Please put some effort into fixing this issue guys - it had me completely stumped.

[isaachinman]

Still an issue - just discovered my yarn cache was close to 100gb.

[JZBo]

Is there any update for the resolution this issue? This is a completely blocking and breaking problem for anyone using local packages, and it unfortunately leaves us unable to leverage all the other superior features of yarn.

[Haemp]

@JZBo @isaachinman I ended up using yarn workspaces for my local modules instead. Basically I'm using a monorepo setup and have a yarn workspace setup that allows me to refer to the packages by their name instead of the file: syntax.

I know not everyone may have that option but it worked for me.

[isaachinman]

@Haemp Hadn't heard of yarn workspaces before. After reading the docs:

Note that the private: true is required! Workspaces are not meant to be published, so we’ve added this safety measure to make sure that nothing can accidentally expose them.

So it appears this is just for private/local testing?

[Haemp]

@isaachinman Yea that seems about right, it's still pretty new.

[isaachinman]

I don't think it helps me much then. My use case is over at next-i18next where we have an example subdir that consumes the main package.

[mrseanryan]

workaround - use yarn cache clean

example from package.json:

  "devDependencies": {
    "basic-parser": "file:../parsers/basic-parser",
   }

where ../parsers/basic-parser points to another folder that contains another package.json

Then regularly (after each yarn) run yarn cache clean for each local package:

yarn cache clean "basic-parser"

This seems to free up the disk space.

[rohit-gohri]

@mrseanryan Can add that to the "postinstall" script to run it automatically

[mrseanryan]

@rohit-smpx yes - that works too

in the consuming package.json:

  "scripts": {
    "postinstall": "yarn cache clean basic-parser && yarn cache clean my-other-package"
  },

cleaner than a bash script :)

[JZBo]

workaround - use yarn cache clean
.....

Then regularly (after each yarn) run yarn cache clean for each local package:

yarn cache clean "basic-parser"

I'm sorry, but as far as my understanding goes this is by no means a workaround, it's just a way to clean up the yarn cache.

In my opinion, this is not a workaround because it does not address the issue. The local package is still cached, and thus it's both copied to the cache and then written in the package directory. And then you have to spend time to clean it. There's a huge amount of waste.

The point of this issue is that local packages should not be cached at all by yarn.

[mrseanryan]

Alright - no ones suggesting the issue should not be fixed.

I shared the above as it mitigates some of the effects (disk filling) and thought it might be useful, if only as mitigation.

[jackflyer]

It seems that yarn creates different cache of local packages, whenever package.json updated.

[bluelovers]

pls allow set limit size of total cache

[stefan-toubia]

Hey everyone, 448GB checking in here... 🤯
How's that fix coming along?

[velipso]

checking in with 96GB rmed last year, and 70GB rmed this year

"fixed" via:

cd ~/Library/Caches/Yarn
rm -rf *

[manurampandit]

This happened with me today. After wasting about 1h of mine trying to run du on each sub-directory as to what is causing my new mac to run out of memory.
du -sh /Users//Library/Caches/Yarn/v6/
100G /Users//Library/Caches/Yarn/v6/

Took about 40 min to clear the cache.
yarn cache clean
yarn cache v1.22.10
success Cleared cache.
✨ Done in 2300.71s.

Hopefully someone should prioritise this.

[TomzBench]

I was pissed when I saw 21G cache. Boy was I let off easy

[kefniark]

It's impressive to see people complaining about this problem for 5 years, multiples issues over multiple repo (closed or not), few PR which never made it through.
And still people complaining with 50GB+ cache ...

Let's be honest, this issue is probably never gonna be fixed in Yarn1. And this workaround is probably the best solution available
#6037 (comment)

[thantos]

just wasted a few hours on this after my WSL disk filled and nothing would run... will know in the future the issue is yarn...

156G of cache for me.

[d-mooers]

Checking in here with 634GB yarn cache that I just discovered while trying to figure out where all of my disk space went - any updates regarding a fix?

[stefan-toubia]

pnpm is pretty great 👍

[andrico1234]

We came across a similar issue.

The use case we had was a local eslint plugin package that we wanted to use in our top-level project.

our top-level dependencies look like this:

"dependencies": {
  ... other deps
  "eslint-plugin-test": "link:./eslint-test"
}

Based on the documentation here, Yarn copies the contents to the cache. We noticed that due to OS discrepancies, the checksum generated on Mac and Windows machines were different.

Because we don't care about caching, since it's a local folder, we changed the protocol to portal instead.

This means Yarn creates a link to the folder instead of copying it. As a result, Yarn doesn't cache the folder and doesn't generate a checksum for it, so the yarn.lock is consistent across OSes.

I hope this works for others too :)

[prestondevelopment]

It's crazy that yarn hasn't fixed this considering this issue was opened in 2018. Fix it.

[Joeb3219]

Just ran into this myself... 250GB of cache.

[notjulian]

finally I've found this issue ... just cleaned 460 Gb of cache
thanks for this @mrseanryan

"scripts": {
    "postinstall": "yarn cache clean basic-parser && yarn cache clean my-other-package"
  },

[felixroos]

The top 3 so far:

place	size	name	date
1	634GB	d-mooers	Aug 2022
2	448GB	stefan-toubia	Jun 2020
3	460GB	notjulian	May 2023

who has more? ;-)

btw this is the old yarn repo, maybe v3 has the fix..

[Ronsku]

Just cleaned up my CI from 742Gb of cached files in AppData/Local/Yarn/Cache/v6.
...and I thought I liked Yarn, but it seems this has been an issue since 2018... it makes me wonder?

[felixroos]

Just cleaned up my CI from 742Gb of cached files in AppData/Local/Yarn/Cache/v6.

...and I thought I liked Yarn, but it seems this has been an issue since 2018... it makes me wonder?

You've beaten no 1 of the highscore! (See my last comment for more useful info)

[DominickVale]

144gb just cleared... I have no idea how some javascript code can take up more space than 3 AAA games...

[dsosborn]

My work macbook only has a 250GB hard drive, and I've been struggling get get enough free for an OS update. Digging through system files I wondered how big the yarn cache could be... and now half my hard drive is free.

[ispringer]

My work macbook only has a 250GB hard drive, and I've been struggling get get enough free for an OS update. Digging through system files I wondered how big the yarn cache could be... and now half my hard drive is free.

Tell your employer you need a new laptop with a bigger hard drive. You need 250 gigs just for the yarn cache man!

[tombohub]

this feels like I have found dead bodies (I didn't) in the basement of very nice house I bought

[VladimirMikulic]

84GB over here. Absolutely crazy. I'm definitely switching back to NPM.

[Ronsku]

84GB over here. Absolutely crazy. I'm definitely switching back to NPM.

Bun just released 1.0. Give Bun a try https://bun.sh/ 😊