yarn install ignores files from local dependencies

[phtrivier]

What is the current behavior?

It seems that yarn install ignores files from package.json of projects that are locally depended on.

If the current behavior is a bug, please provide the steps to reproduce.

Using two simple projects that depends on each other locally:

○ → tree
.
|____project1
| |____bad.js
| |____good.js
| |____package.json
|____project2
| |____package.json

project1 specifies which files should be copied by dependent projects:

// project1/package.json
{
    "name" : "project1",
    "version" : "0.0.1",
    "files" : ["good.js"]
}

project2 depends on project1 locally:

// project2/package.json
{
    "name" : "project2",
    "version" : "0.0.1",
    "dependencies" : {
        "project1" : "file:../project1"
    }
}

npm install in project2 only copies the good file:

 2017-03-02 12:43:41 ⌚  turus in ~/tmp/yarn-issue/project2
○ → npm install
project2@0.0.1 /home/phtrivier/tmp/yarn-issue/project2
└── project1@0.0.1 

npm WARN project2@0.0.1 No description
npm WARN project2@0.0.1 No repository field.
npm WARN project2@0.0.1 No license field.

 2017-03-02 12:43:45 ⌚  turus in ~/tmp/yarn-issue/project2
○ → ll node_modules/project1/
total 32
drwxrwxr-x 2 phtrivier phtrivier 4096 mars   2 12:43 .
drwxrwxr-x 3 phtrivier phtrivier 4096 mars   2 12:43 ..
-rw-rw-r-- 1 phtrivier phtrivier   37 mars   2 12:37 good.js
-rw-rw-r-- 1 phtrivier phtrivier 1171 mars   2 12:43 package.json

However, yarn install copies everything, ignoring files:

  2017-03-02 12:43:48 ⌚  turus in ~/tmp/yarn-issue/project2
○ → rm -rf node_modules

 2017-03-02 12:44:55 ⌚  turus in ~/tmp/yarn-issue/project2
○ → yarn install
yarn install v0.21.3
warning project2@0.0.1: No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.26s.

 2017-03-02 12:44:58 ⌚  turus in ~/tmp/yarn-issue/project2
○ → ll node_modules/project1/
total 44
drwxrwxr-x 2 phtrivier phtrivier 4096 mars   2 12:44 .
drwxrwxr-x 3 phtrivier phtrivier 4096 mars   2 12:44 ..
-rw-rw-r-- 1 phtrivier phtrivier   57 mars   2 12:37 bad.js
-rw-rw-r-- 1 phtrivier phtrivier   37 mars   2 12:37 good.js
-rw-rw-r-- 1 phtrivier phtrivier   80 mars   2 12:38 package.json

 2017-03-02 12:45:03 ⌚  turus in ~/tmp/yarn-issue/project2
○ → 

What is the expected behavior?

Same behavior as npm.

Please mention your node.js, yarn and operating system version.

○ → npm --version
3.10.10
○ → yarn --version
0.21.3
○ → uname -a
Linux turus 4.8.0-39-generic #42-Ubuntu SMP Mon Feb 20 11:47:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[rally25rs]

I suspect this may be a common error across other resolvers as well.

Normally when you npm publish a package, only the items listed in files are included in the .tgz, so for normal NPM packages, the cli tool (npm or yarn) doesn't have to really worry about it.

(you can verify this by setting up an offline cache directory and yarn add bootstrap, then compare the downloaded .tgz to their GitHub repo. Only the items from files are in the downloaded/cached .tgz, not every file in the Github repo)

For other resolvers, it's going to have to filter on its own. I did a quick search through the yarn source and didn't see anything that would be doing that check (though I could have missed it).

[SimenB]

@rally25rs yarn pack does this filtering to avoid bundling them before publish.

[kohlmannj]

Confirming @rally25rs's suspicion: this issue appears to also affect installation from GitHub using the git+https:// protocol. I've seen Yarn install a private GitHub dependency with its src and build folders, despite its package.json:files array containing only the build folder.

I suppose publishing with yarn pack is the workaround, but that's not feasible for all projects. Ultimately it'd be better to honor the files property.

[foxymoron]

For private git repositories, if npm is able to filter packages by reading the files:[] property, shouldn't yarn be doing the same ?

[MonsieurBon]

We face exactly the same problem with a dependency we're pulling directly out of our private corporate git repository. Any idea when this will be fixed?

[chead2344]

Has there been any progress made on this?

[ZaLiTHkA]

just out of curiosity, is there any plan to change this behaviour?

I've just run into a situation where I need one specific file from private repo A as an import in private repo B, but the file I need is created as part of repo A's build step, so I don't need the other 5MB of stuff that it contains...

I've created a package.json in repo A with the relevant files key, but as other people in this thread have discovered, npm install limits the files that are retrieved, while yarn add does not. looking through the output from yarn add --help, it seems there are no command flags to control this, so is this the behaviour we're stuck with now?

[ovidiuch]

Not sure if it's related, but I'm experiencing a situation where yarn ignores package.json:files from a workspace package.

You can see the src dir being included when running yarn pack (as well as when installing the dep from npm), even though it's not included in package.json:files.

cat package.json

{
  "name": "react-cosmos-test",
  "version": "4.0.0-beta.0",
  "description": "APIs for testing Cosmos fixtures in headless environments",
  "repository": "https://github.com/react-cosmos/react-cosmos/tree/master/packages/react-cosmos-test",
  "license": "MIT",
+  "files": [
+    "lib",
+    "enzyme.js",
+    "generic.js"
+  ],
  "dependencies": {
    "react-cosmos-config": "^4.0.0-beta.0",
    "react-cosmos-loader": "^4.0.0-beta.0",
    "react-cosmos-shared": "^4.0.0-beta.0",
    "traverse": "^0.6.6"
  },
  "xo": false
}

yarn pack
tar -tf react-cosmos-test-v4.0.0-beta.0.tgz

package
package/enzyme.js
package/generic.js
package/lib
package/package.json
+package/src
package/lib/enzyme.js
package/lib/generic.js
+package/src/__tests__
+package/src/enzyme.js
+package/src/generic.js
+package/src/__tests__/enzyme.js
+package/src/__tests__/generic.js

I appreciate any help with this one as it's annoying for react-cosmos users to download unnecessary source files, which among other things can cause Flow errors in the dependent project. Thanks!

[zastavnitskiy]

Hi everyone, I'm going to fix it.

After the initial research I have two questions.

I plan to modify https://github.com/yarnpkg/yarn/blob/master/src/fetchers/copy-fetcher.js#L10, and then I have several options:

Option one:

• read files field from package.json
• delete all the files that do not match regexp (using the logic from https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/pack.js#L75)

Option two:

• run pack command for this directory

Which one is better?

And the second question – does doing it in the fetcher make sense?

@BYK @kittens I will appreciate your feedback on how to proceed here.

[tommedema]

This problem occurs to me when depending on local packages using Yarn workspaces, too.

Could someone please confirm if this is the same issue? Or should I create a new issue for yarn workspaces specifically?

E.g.

workspace/packages/a has files: ['good.js]
workspace/packages/b depends on package a through yarn workspaces

yarn install --production results in package b's node_modules/a having the full contents of package a (even when package b has been set as nohoist), while it should only include good.js

If this is a separate issue please let me know so I can create it.

[kitfit-dave]

Any word on this? I am running into this when developing a package the specifies the babel transpiled dist directory in files. So when someone who gets it from the repository uses it they will do import something from 'mypackage/somefolder'. But as the yarn workspaces link is to the root of the package not the dist folder, I have to do import something from 'mypackage/dist/somefolder'

[mattvb91]

experiencing the same issue

[aemre]

any update on this?

[n1kk]

I'm having same issue, trying to test a package before publishing and using folder reference yarn gets all the contents including node_modules folder, which creates issues for typescript. Using yarn pack and installing tarbal seems to solve it but then I run into the issue of yarn caching that tarball, so I have to clean yarns cache every time and I run into errors in there, some directories don't exist etc etc... so either way is not working for me.

[hl037]

@merceyz this has been tagged as "fixed-in-v2" but it is not (v2.4.0). When I specify a dep as "file:...", yarn copies the full tree, tatally ignoring the dep package.json files entry. This is particulary anoying for sample / example projects in a subdir of the main project, since it get recursively copied...

[bsberry]

This is still an issue on yarn 1.23.0. Tried yarn 2 and it did not appear to be fixed there either as hl037 states.

[leonbubova]

Any updates on this? As of now I am still encountering this issue. I am trying to install a private package in a project and only need the /dist folder in the package, but also have some other dependencies in the same repo that I am using for the documentation of the package, that I dont want in the project I am using the package in. It's particularly annoying that this feature works fine with npm, but not with yarn.

[akramer24]

So... this might be insanely silly but I added a prepare script that did nothing -- just "prepare": "echo something" and the install began respecting the my package.json's files property. Hope this hack works for others!

[leonbubova]

So... this might be insanely silly but I added a prepare script that did nothing -- just "prepare": "echo something" and the install began respecting the my package.json's files property. Hope this hack works for others!

Awesome that worked! Thank you

[alexisloiselle]

So... this might be insanely silly but I added a prepare script that did nothing -- just "prepare": "echo something" and the install began respecting the my package.json's files property. Hope this hack works for others!

@akramer24 In my experience, this only works for git repos. Was that your use case?

No solutions nor workarounds yet for local dependencies...

[akramer24]

So... this might be insanely silly but I added a prepare script that did nothing -- just "prepare": "echo something" and the install began respecting the my package.json's files property. Hope this hack works for others!

@akramer24 In my experience, this only works for git repos. Was that your use case?

No solutions nor workarounds yet for local dependencies...

It was a git repo in my case

[bsssshhhhhhh]

Looks like there's been a PR open for this issue since last year. Can someone with access get around to merging it please?

This is not a novel use case and should be supported because npm supports it.

[mattvb91]

Looks like there's been a PR open for this issue since last year. Can someone with access get around to merging it please?

This is not a novel use case and should be supported because npm supports it.

FYI (and no idea why this decision was thought to be good or why they still allow issues to be open here) this repo is no longer maintained and the new version is over here: https://github.com/yarnpkg/berry

Trying to upgrade has been an absolute mess with yarn and we have since moved back to npm and it has fixed all of our issues. Unfortunately I do think yarn has shot itself in the foot by messing up the upgrade path & moving to a different repository but thats the beauty of open source :)