Yarn needs to use .npmignore file when installing from repo

[danielkcz]

Do you want to request a feature or report a bug?
Feature (I would like think of it as a bug too)

What is the current behavior?
Trying to install three.js module from repository instead of registry yields 303MB folder in size. Installing same thing with NPM and resulting folder is just 11MB

If the current behavior is a bug, please provide the steps to reproduce.
yarn init
yarn add github:mrdoob/three.js

What is the expected behavior?
The .npmignore file should be used and actually remove files that conforms to globs there. Not sure if it's possible to actually avoid unpacking those files in first place. That would certainly improve performance much more.

Reason why to install package from repository is because of custom forks.

Please mention your node.js, yarn and operating system version.
Node 6.9.1, Yarn 0.17.7, Windows 10 x64

[Daniel15]

module from repository instead of registry

What's the use case for this? Is it just to install a development version?

[danielkcz]

@Daniel15 Already said that in original post...

Reason why to install package from repository is because of custom forks.

Or actually any private modules as well.

[Daniel15]

Ahhh, so sorry @FredyC, I totally missed that part of your post! For custom forks of open source libraries, one option is to publish them to npm as a namespaced/scoped package (like @fredyc/foo), which should behave better than installing it directly from the repo (in theory).

In any case, this definitely sounds like an improvement we should make :)

[danielkcz]

I personally dislike polluting npm registry more than is needed, even with something like scoped packages. Doesn't feel right, but I guess it's possible workaround for now.

Either way it wont work for private modules anyway.

[domakas]

This is a huge deal-breaker for us, would be nice to solve it as soon as possible.

Any ideas what strategy could be used for this? I might be interested in solving this issue myself, however, I don't have any other ideas as just checking out full repo and deleting files which match .npmignore/.yarnignore

[bbrzoska]

Same problem occurs when you try to install from a local directory #685.

[rmm5t]

Just ran into this problem myself. I was really surprised by the behavior, and it's causing the associated directory in our node_modules to sit at 12MB when using yarn install, but npm install correctly ignores the files from the repo leaving the directory at just 32KB.

We have a private module and share it via a git repository.

It's important for us to keep our node_modules small, because they are packaged up as part of our deployments to AWS Lambda. The larger the package deployment, the slower our cold-starts on Lambda.

$ node --version
v6.10.0

$ yarn --version
0.27.5

[mblackritter]

Hmmm, any news on that? I was just surprised when adding a dependency to a local modules via "[package-name]": "file:../[path]" and found ~300 MB in the imported ~10kb module after yarn install.

[stantoncbradley]

this hurts. importing a lib from git repo to react native. yarn not ignoring .babelrc so getting transform errors when compiling react native app because I don't have the babel transform plugins installed for my app

[HaykoKoryun]

Any update on this? It might not be a big deal when developing projects, but it becomes a sticking point when trying to use yarn in production.

[andreialecu]

Also got hit by this while trying to use webpack with the angular2+ AOT compiler to compile a project that references a private library which has .ts files along the transpiled .js files.

Due to configuration requirements, webpack will default to finding and using the .ts files, but the typescript compiler will refuse to compile anything from node_modules resulting in a failed build.

Using .npmignore to ignore .ts files would've solved this.

(Workaround was to reorganize the entire project to store the compiled .js files in a separate directory, so that the .ts files weren't siblings to them.)

[chevdor]

Also seeing this issue which prevents me from using yarn atm 😢
Any news about this issue?

[vileppanen]

Same issue here.
Not an absolute show stopper for us though, but before this is fixed we stick to use purely npm

[henninghall]

As @HaykoKoryun said, this is is a huge issue for using yarn in production if you have any private repos relying on .npmignore.

• Is there any workaround for this issue?

• Does anyone has an idea of how to make a branch of yarn that checks for .npmignore files and deleting these after yarn command?

[sedrubal]

A possible workaround is to add a .yarnclean file and use yarn autoclean to remove waste files.

[Nantris]

Is there a reason that Yarn still doesn't do this?

[waynebloss]

Likewise, yarn doesn't respect a "files" entry in package.json when installing from a repo.

[danielkcz]

I would like to say here that I don't think this is that much necessary anymore. Personally, I've opted to use Yalc for handling forks and I am much happier with it.

[rmm5t]

@FredyC I don't think yalc solves all the problems that this issue is addressing. It's not just about local development. It's legitimate to have a package.json point at a git repository. npm | yarn link and (it appears) yalc solve a slightly different problem. Granted, if you're willing to change your workflow for certain scenarios, yes, it does look like you could rely on yalc to workaround the issue. In fact, I'm grateful that you pointed me to Yalc here (it's good to know about new tools like that), but changing standard workflows somewhat defeats the purpose, IMO.

[danielkcz]

@rmm5t It may not be apparent from docs, but Yalc maintains .yalc directory within a project directory with a published version of the packages and references that in package.json. So whenever you do yarn/npm install it gets copied over to node_modules and dependencies are correctly installed. No magic with symlinks and friends.

They key point here is that you can commit .yalc folder so anyone on the team who runs then yarn/npm install gets correct version of that published module completely bypassing NPM registry. No need to have Yalc installed for everyone, only publishers.

The workflow quite the same. Just clone the fork/private repo and instead of npm publish you do yalc publish. In the target project, you do yalc add <module> and that's it. If you later want to do some further changes to the fork, you can publish again and yalc update in target folder to get these changes. Or yalc push in the source will update it in all targets.

And Yalc respects .npmignore and files, it behaves just like NPM. I think it's a clear win and NPM registry will be less messy with all those forks.

[waynebloss]

@FredyC Are you sure you're in the right repo? This may not be an issue for the thousands of users of yalc, but it's also not an issue for people who play Minecraft (given that they're completely different software).

This is still an issue for the millions of developers that use yarn though.

Thanks for your suggestion to switch to yalc, but if you don't mind we're going to keep pushing for this issue to get fixed in yarn.

[danielkcz]

@waynebloss I am not talking about replacing yarn with yalc. Both are different tools for a different job. It's your choice, I just wanted to show a painless way how to overcome this problem instead of just waiting to get it fixed.

Frankly, after 2 years without team response, it looks like one of those stale issues that are not that important to be worried about.

[waynebloss]

@FredyC I absolutely agree that it's not a huge issue. For front-end projects, the extra files will never make it into the production code. On the back-end things will end up on the server making my Docker images bigger, but anyone serious about building a lot of private NPM packages is probably going to run a private/cache-forwarding NPM registry.

But...it's still an issue. Mostly, I think, because people expect parity with npm in many areas. It also doesn't seem like an extremely difficult problem to solve either. However, in a repo with over 1,500 open issues I can see why it would be a pretty low priority!

[cpojer]

We are running into the same issue at Facebook. Is there anyone who is willing to send a Pull Request for this? I'm happy to review it, merge it and make a new release of Yarn for this bugfix.

[wethil]

I will look into it and try to create a PR

[wethil]

I opened a pr about that 8010