New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use --modules-folder flag? #1684

Open
borela opened this Issue Nov 4, 2016 · 51 comments

Comments

Projects
None yet
@borela
Copy link

borela commented Nov 4, 2016

Sorry if this was asked before but I couldn't find an example in the documentation/google search.

Does this flag mean that I could have a "vendor" directory instead of "node_modules"? How to use it?
I tried something like yarn install --modules-folder ./vendor/node_modules and it just installed the modules in ./node_modules.

Please mention your node.js, yarn and operating system version.
Windows 7
yarn 0.16.1

@dhruvdutt

This comment has been minimized.

Copy link
Contributor

dhruvdutt commented Nov 4, 2016

There's a bug actually. Make sure you delete the node_modules folder, then it should work fine.
If node_modules is present then it would always say Already up-to-date.

This thing isn't documented on the website. I'll do it within next couple of days.
Meanwhile, you can refer this.
https://github.com/yarnpkg/yarn/blob/master/src/cli/index.js#L61-L62

@borela

This comment has been minimized.

Copy link
Author

borela commented Nov 4, 2016

Thank you @dhruvdutt it worked. Is there any way to set the new modules folder as default?

@Promixius

This comment has been minimized.

Copy link

Promixius commented Nov 5, 2016

a way to set the default module path would be favorable as running yarn upgrade later when a package has an update will again pull it into node_modules instead of ./vendor/node_modules

@dhruvdutt

This comment has been minimized.

Copy link
Contributor

dhruvdutt commented Nov 5, 2016

@Promixius I was thinking about the same. To make a configurable file or something for this purpose.

@nifgraup

This comment has been minimized.

Copy link

nifgraup commented Nov 7, 2016

broke in 5300b48

@nifgraup

This comment has been minimized.

Copy link

nifgraup commented Nov 7, 2016

any way to recover branch clean-workflow from #281?

@nifgraup

This comment has been minimized.

Copy link

nifgraup commented Nov 9, 2016

I would be able to bisect this further if you push clean-workflow again @kittens

@nifgraup

This comment has been minimized.

Copy link

nifgraup commented Nov 17, 2016

result of further bisecting:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
f5f77ed
61890e9
ec7b1e5
f13537f
We cannot bisect more!

@nifgraup

This comment has been minimized.

Copy link

nifgraup commented Nov 17, 2016

After working around skipped commits I found this to be the trouble maker:

commit ec7b1e5
Author: Sebastian McKenzie kittens@users.noreply.github.com
Date: Mon Sep 5 15:45:55 2016 +0100

Add integrity hash check to `kpm install`

nifgraup added a commit to nifgraup/yarn that referenced this issue Nov 18, 2016

@nsgundy

This comment has been minimized.

Copy link

nsgundy commented Nov 30, 2016

If I run yarn install --production --modules-folder production to have a folder that can be deployed to the production environment and I then run yarn install to get a local node_modules folder to run the build with (in this particular order because of the bug mentioned here and addressed by the PR above), yarn removes all modules from the folder production. This can be worked around by renaming the folder before the second run of yarn install, but I would prefer if yarn would just ignore that folder as long as it isn't specified again using --modules-folder.
Could that be changed or is the current implementation on purpose and required?

@jodigiordano

This comment has been minimized.

Copy link

jodigiordano commented Feb 11, 2017

IMO --modules-folder should be available on all commands.
Simple use case:

yarn install --modules-folder tmp/modules
yarn run [command] --modules-folder tmp/modules
@pciavald

This comment has been minimized.

Copy link

pciavald commented Feb 22, 2017

specifying in package.json would be amazing. Any ETA on this ?
http://stackoverflow.com/questions/40488421/how-to-change-yarn-default-packages-directory

@bestander

This comment has been minimized.

Copy link
Member

bestander commented Mar 3, 2017

This is a great opportunity to contribute to the project.
Please feel free to send a PR with tests and we'll get it in

@StevenDoesStuffs

This comment has been minimized.

Copy link

StevenDoesStuffs commented Apr 13, 2017

Any progress? I'm not familiar enough with the yarn codebase, and this feature would be amazing!

@rcelha

This comment has been minimized.

Copy link

rcelha commented Apr 20, 2017

@jodigiordano If this API is accepted by the maintainers I'd implement it... it'd make my life easier

@JesseObrien

This comment has been minimized.

Copy link

JesseObrien commented May 1, 2017

In regards to #1684 (comment), I'm curious as well. It seems that running yarn with --modules-folder will wipe out/modify any existing node_modules folders that yarn can see. Is this intended behaviour?

@jimtom2713

This comment has been minimized.

Copy link

jimtom2713 commented May 3, 2017

We have also had this issue mentioned above and referenced by @JesseObrien. We use webpack to package code up and create our distribution folder. I notice that running the following command to generate production node_modules folder in our distribution directory deletes the other modules within other exisiting node_modules folders.

yarn install --production --modules-folder ./dist/node_modules

Is this intended? If so, what's the rationale behind removing other node_modules contents?

@bestander

This comment has been minimized.

Copy link
Member

bestander commented May 12, 2017

@jimtom2713, yes this is intended, Yarn guarantees that node_modules is clean every time you install.

@nsgundy

This comment has been minimized.

Copy link

nsgundy commented May 12, 2017

@bestander would love to be able to run yarn install and yarn install --production --modules-folder ./dist/node_modules (or some new version of that) one after the other and as a result have all modules (both dev-dependencies and production-dependencies) in node_modules and only the production dependencies in ./dist/node_modules.

Currently that seems only possible if I rename the node_modules folder after running the first command and then renaming it back after running the second command.

@bestander

This comment has been minimized.

Copy link
Member

bestander commented May 12, 2017

@jimtom2713

This comment has been minimized.

Copy link

jimtom2713 commented May 12, 2017

@bestander The behavior @nsgundy is looking for is similar to the one we have. We have webpack bundle and package the code into dist. Then we run yarn install --modules-folder ./dist/node_modules to install the distribution modules under the dist folder. However, this means that the modules in the current working directory are removed. In development, and staging grounds this wipes the development modules away and we've gotten around it by changing the name of the folder and the order of the testing steps, but it's a lot of extra steps. Also, in development boxes, testing the build process locally means that without those changes the node modules are removed and require a fresh install every time.

@SMotaal

This comment has been minimized.

Copy link

SMotaal commented Nov 14, 2017

A fun fact about .yarnrc syntax:

Instead of:

--add.modules-folder …
--install.modules-folder …
… …

Try:

--*.modules-folder …

@hkdobrev this does not result in a lingering node_modules although it might still be created and deleted instantaneously as yarn does its thing.

edit: Seems that yarn remove does not honour --*. the same as --remove. (so maybe other quirks)

@brunofin

This comment has been minimized.

Copy link

brunofin commented Nov 23, 2017

@SMotaal How can I find all the available options for .yarnrc? It seems the options aren't documented anywhere.

Specifically, I can't find anything mentioning the modules folder options.

@SMotaal

This comment has been minimized.

Copy link

SMotaal commented Nov 23, 2017

@brunofin That was just a fluke… Yes, there are hardly any breadcrumbs to follow (I think this is a sign of the times, everyone is doing that no-docs ≠ poor-docs thing now)

@BYK

This comment has been minimized.

Copy link
Member

BYK commented Nov 23, 2017

I think this is documented here: https://yarnpkg.com/en/docs/yarnrc#toc-cli-arguments

Basically, you can put any CLI flag into your yarnrc file, either as tehy are or prefixed with a yarn command to scope it to only that command.

@SMotaal

This comment has been minimized.

Copy link

SMotaal commented Nov 23, 2017

@BYK Yeah, came across that but then --modules-folder in .yarnrc did not function per docs — would pre-bind a --modules-folder arg for all commands where it applies, but sadly for this arg the behaviour was not on par with others I used in prior occasions.

Whereas for some un* [documented, supported, intended, …] — can only assume its just the temporary useful kinda bug/hidden feature — using --*.modules-folder instead seems to do a little better than --modules-folder in a .yarnrc.

But even then, it seems that certain commands just seem too self-important and want to be mentioned individually as they don't fancy themselves being lumped with the rest of em 😱

@BYK

This comment has been minimized.

Copy link
Member

BYK commented Nov 23, 2017

@SMotaal that actually sounds like a bug. We already know that some commands which should honor --modules-folder does not. That said --*.whatever should be equivalent to --whatever and I'm pretty sure the code paths are either the same or very similar. It is here:

yarn/src/rc.js

Lines 30 to 64 in f69cdda

function buildRcArgs(cwd: string): Map<string, Array<string>> {
const config = getRcConfigForCwd(cwd);
const argsForCommands: Map<string, Array<string>> = new Map();
for (const key in config) {
// args can be prefixed with the command name they're meant for, eg.
// `--install.check-files true`
const keyMatch = key.match(/^--(?:([^.]+)\.)?(.*)$/);
if (!keyMatch) {
continue;
}
const commandName = keyMatch[1] || '*';
const arg = keyMatch[2];
const value = config[key];
// create args for this command name if we didn't previously have them
const args = argsForCommands.get(commandName) || [];
argsForCommands.set(commandName, args);
// turn config value into appropriate cli flag
const option = commander.optionFor(`--${arg}`);
// If commander doesn't recognize the option or it takes a value after it
if (!option || option.optional || option.required) {
args.push(`--${arg}`, value);
} else if (value === true) {
// we can't force remove an arg from cli
args.push(`--${arg}`);
}
}
return argsForCommands;
}

@SMotaal

This comment has been minimized.

Copy link

SMotaal commented Nov 23, 2017

@BYK thats amazing, thanks!

My apologies, I would retest this to make certain but I cannot recall which project/context I encountered it for replicating... @brunofin do you think you're hitting this same inconsistency and if so, can you provide a recipe that is reproducible.

It is very clear from the code that commands involving buildRcArgs would be bound equally with --*.… as --… so if nothing has changed in this function since we last encountered that issue, I wonder if maybe there is a difference some point upstream or even downstream.

@brunofin

This comment has been minimized.

Copy link

brunofin commented Nov 24, 2017

What happened to me is quite specific.

We use Vagrant in our project for reasons, and previously we had node6/npm, but then I migrated the project to node9/Yarn, and later on, I found out that Yarn allows me to configure the node_modules folder location. This was great, because of a known problem with Vagrant and symbolic links that are created under the node_modules/.bin folder during linking phase this flag could help me solve it. The problem is that Vagrant/Windows won't allow symlinks in a network-shared folder, node then throws a protocol error and things break.

For this reason, I had my project set up in a weird way where package.json, bower.json and other files that should be in the root folder of the project were actually in a subfolder called config. The contents of this folder were linked to the root folder in the VM, all that so node_modules would not be a part of the network-shared folder but still in the root of the project, at least in the VM. Very ugly.

But with Yarn, I can configure the location of node_modules, so I changed it to be somewhere else in the VM, moved the config files to the root of the project where they belong, great.

Using --modules-folder ..., I still got modules downloaded to my project root node_modules folder.
So I used --*.modules-folder ... instead and the modules are now downloaded to the location I wanted them to be.

But still I get a node_modules folder in the root of my project when Yarn tries to create symbolic links to the installed modules' binaries, and that's the reason I am making those changes in the first place. I'm not sure if this is a bug, but I think node_modules/.bin symlinks should also respect the --*.modules-folder flag.

I solved my problem by also enabling --no-bin-links true in .yarnrc, that works except bower, gulp or any other module which has binaries won't work because they are not in PATH. Manually linking the binaries to a folder in PATH (like ~/bin) solves the problem for some modules, but Gulp, for example, won't accept that, claiming there is no local gulp installed.

It's a complicated situation a bit out of the scope of the talk, but that's why I asked @SMotaal about the CLI documentation, in case I was missing some other command that could help me solve that situation. I just feel that not all the available commands are documented, or the documentation isn't clear enough about its flexibility and .yarnrc.

I also found out that Yarn won't respect the --production true flag in .yarnrc and will download all the devDependencies anyway.

@BYK

This comment has been minimized.

Copy link
Member

BYK commented Nov 27, 2017

@brunofin thanks for sharing your use case and experience. I was also struck by that symlink issue with Vagrant and Windows some time ago and I can definitely relate to that. --modules-folder and --*.modules-folder should do the same thing so not sure what's going on there. Rest, though, is probably related to this bug. We know that --modules-folder is not used everywhere where it should be used etc. and this bug is kept open to keep track of this.

I also found out that Yarn won't respect the --production true flag in .yarnrc and will download all the devDependencies anyway.

This also shouldn't be the case. You are either using an old version of yarn which has a bug regarding CLI options in .yarnrc which would explain the --*.modules-folder and --modules-folder difference above or there is something else going on with env variables or something. We have this whole test suite after that:

describe('production', () => {
test('it should be true when NODE_ENV=production', async () => {
const cwd = await makeTemp();
const options = {cwd, env: {YARN_SILENT: 1, NODE_ENV: 'production'}};
const [stdoutOutput, _] = await runYarn(['config', 'current'], options);
expect(JSON.parse(stdoutOutput.toString())).toHaveProperty('production', true);
});
test('it should default to false', async () => {
const cwd = await makeTemp();
const options = {cwd, env: {YARN_SILENT: 1, NODE_ENV: ''}};
const [stdoutOutput, _] = await runYarn(['config', 'current'], options);
expect(JSON.parse(stdoutOutput.toString())).toHaveProperty('production', false);
});
test('it should prefer CLI over NODE_ENV', async () => {
const cwd = await makeTemp();
const options = {cwd, env: {YARN_SILENT: 1, NODE_ENV: 'production'}};
const [stdoutOutput, _] = await runYarn(['--prod', 'false', 'config', 'current'], options);
expect(JSON.parse(stdoutOutput.toString())).toHaveProperty('production', false);
});
test('it should prefer YARN_PRODUCTION over NODE_ENV', async () => {
const cwd = await makeTemp();
const options = {cwd, env: {YARN_SILENT: 1, YARN_PRODUCTION: 'false', NODE_ENV: 'production'}};
const [stdoutOutput, _] = await runYarn(['config', 'current'], options);
expect(JSON.parse(stdoutOutput.toString())).toHaveProperty('production', false);
});
test('it should prefer CLI over YARN_PRODUCTION', async () => {
const cwd = await makeTemp();
const options = {cwd, env: {YARN_SILENT: 1, YARN_PRODUCTION: 'false', NODE_ENV: 'production'}};
const [stdoutOutput, _] = await runYarn(['--prod', '1', 'config', 'current'], options);
expect(JSON.parse(stdoutOutput.toString())).toHaveProperty('production', true);
});
});

@brunofin

This comment has been minimized.

Copy link

brunofin commented Nov 27, 2017

@BYK thanks a lot for clarifying. I'll make sure I'll check my versions, upgrade everything possible and try again. I'll let you know.

btw, any advice you can give me with the Vagrant situation? (sorry if I'm leading to an off-topic conversation).

@BYK

This comment has been minimized.

Copy link
Member

BYK commented Nov 27, 2017

@brunofin

btw, any advice you can give me with the Vagrant situation? (sorry if I'm leading to an off-topic conversation).

I think the newer versions of Windows allow the creation of symlinks without admin permissions: https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/

If you want to chat more, come to our Discord server ;)

@miks

This comment has been minimized.

Copy link

miks commented Dec 12, 2017

What's the rationale behind --modules-folder flag if it's works only for install command?
Sure, I can install modules in specified folder, but that's all.
It's not working for example with run command.
Am I missing something?

@bdurrani

This comment has been minimized.

Copy link

bdurrani commented Dec 25, 2017

I found this issue via this SO thread.

What if there are packages that you need to install for the server side (e.g. express) others that you need from browser side e.g. jquery, bootstrap, font-awesome?

I used NPM for server and bower for jquery, bootstrap etc ... now with Yarn alone I still don't know how to change destination for a particular library that I want in my assets/vendor folder instead or node_modules

@kwhitejr

This comment has been minimized.

Copy link

kwhitejr commented Jan 25, 2018

@nsgundy , @zeroblaz3 , @AndersDJohnson , @jimtom2713 : Anyone figure out a more satisfactory way of preventing the base level node_modules from being blown away when using yarn install --production --modules-folder other/directory/? Seems like the most straightforward workaround was to rename original node_modules before and after the yarn install?

@nsgundy

This comment has been minimized.

Copy link

nsgundy commented Feb 7, 2018

@kwhitejr, unfortunately I have not.

@jimtom2713

This comment has been minimized.

Copy link

jimtom2713 commented Feb 7, 2018

@kwhitejr I have not found a different way.

@chakradhar2

This comment has been minimized.

Copy link

chakradhar2 commented Feb 8, 2018

I am using --modules-folder, which download's module for me in different folder. But not able to run the commands which I use to run before using --modules-folder, like gulp .
Is there any way, I can ask to look the modules in the new custome folder and do not have to change the all paths's in require() ?

@njlr

This comment has been minimized.

Copy link

njlr commented Feb 13, 2018

@miks The reason I use the --modules-folder flag is to create a node_modules folder that contains only production dependencies, which are then bundled with my app. If I didn't do this then I would be bundling modules like babel, which bloats the binary.

@miks

This comment has been minimized.

Copy link

miks commented Feb 13, 2018

@njlr how you can run your code against custom node_modules folder after install?

@njlr

This comment has been minimized.

Copy link

njlr commented Feb 13, 2018

@miks I bundle the whole thing (my code, node_modules and Node itself), for example using pkg.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment