New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yarn: Move node_modules, package.json, and yarn.lock file to vendor #27245

Merged
merged 3 commits into from Dec 1, 2016

Conversation

Projects
None yet
8 participants
@dhh
Member

dhh commented Dec 1, 2016

And then provide a binstub for yarn that runs in that directory.

Move node_modules, package.json, and yarn.lock file to vendor
And then provide a binstub for yarn that runs in that directory.

@dhh dhh referenced this pull request Dec 1, 2016

Merged

Add Yarn support in new apps using --yarn option #26836

7 of 7 tasks complete

dhh added some commits Dec 1, 2016

Don't use manually maintained vendor subdirectories when using yarn
We should drop the vendor CSS directory anyway, I think.

@dhh dhh merged commit 3dac36b into master Dec 1, 2016

0 of 3 checks passed

codeclimate Code Climate is analyzing this code.
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details

@dhh dhh deleted the yarn-from-vendor branch Dec 1, 2016

@@ -0,0 +1,2 @@
VENDOR_PATH = File.expand_path('../vendor', __dir__)
Dir.chdir(VENDOR_PATH) { exec "yarnpkg #{ARGV.join(" ")}" }

This comment has been minimized.

@ianks

ianks Dec 4, 2016

Contributor

Booting a whole ruby interpreter just to shell out to yarn seems unnecessary. This is especially painful for jruby folks.

@ianks

ianks Dec 4, 2016

Contributor

Booting a whole ruby interpreter just to shell out to yarn seems unnecessary. This is especially painful for jruby folks.

This comment has been minimized.

@dhh

dhh Dec 4, 2016

Member

Booting Ruby isn't time consuming with MRI. But happy to take a patch with another approach that works on all platforms (and thus doesn't rely on Bash or whatever).

@dhh

dhh Dec 4, 2016

Member

Booting Ruby isn't time consuming with MRI. But happy to take a patch with another approach that works on all platforms (and thus doesn't rely on Bash or whatever).

This comment has been minimized.

@ianks

ianks Dec 4, 2016

Contributor

FWIW, the yarn binstub relies on sh already.

@ianks

ianks Dec 4, 2016

Contributor

FWIW, the yarn binstub relies on sh already.

@@ -1,7 +1,7 @@
* Add Yarn support in new apps using --yarn option. This adds a package.json
* Add Yarn support in new apps using --yarn option. This adds a yarn binstop, vendor/package.json,

This comment has been minimized.

@ianks

ianks Dec 4, 2016

Contributor

binstub*

@ianks

ianks Dec 4, 2016

Contributor

binstub*

ianks added a commit to ianks/rails that referenced this pull request Dec 4, 2016

Use sh instead of Ruby script for yarn binstub
Seeing as yarn already uses sh for their binstub
(https://github.com/yarnpkg/yarn/blob/master/bin/yarn), we opt to use `sh` as
well. The main reason for this being that boot times for some interpreters
(i.e. jruby) is rather slow. This makes for a much better UX for these users.

See: rails#27245

ianks added a commit to ianks/rails that referenced this pull request Dec 4, 2016

Use sh instead of Ruby script for yarn binstub
Seeing as yarn already uses sh for their binstub
(https://github.com/yarnpkg/yarn/blob/master/bin/yarn), we opt to use `sh` as
well. The main reason for this being that boot times for some interpreters
(i.e. jruby) is rather slow. This makes for a much better UX for these users.

See: rails#27245
@lemonmade

This comment has been minimized.

Show comment
Hide comment
@lemonmade

lemonmade Dec 5, 2016

I am really interested why this decision was made. It will make actually configuring any JavaScript tooling extremely difficult, and causes it to behave differently than it would in any other project that uses npm packages. As the simplest example, editor integration with a popular JavaScript linter like ESLint will not work anymore as they all rely on a "traditional" structure that has node_modules at the root, and that follows the basic node resolution algorithm.

At Shopify, we've had no trouble with node_modules at the root of the app (plus additional node_modules for separate bundles within the main application). It's been easier for new developers to understand where dependencies are coming from since it's just like any other npm-using project. If the only benefit is that the root directory is cleaned up a little bit, I don't see that as being worth the cost, particularly when other dependency information for the application (the Gemfile) is already stored at root.

I am really interested why this decision was made. It will make actually configuring any JavaScript tooling extremely difficult, and causes it to behave differently than it would in any other project that uses npm packages. As the simplest example, editor integration with a popular JavaScript linter like ESLint will not work anymore as they all rely on a "traditional" structure that has node_modules at the root, and that follows the basic node resolution algorithm.

At Shopify, we've had no trouble with node_modules at the root of the app (plus additional node_modules for separate bundles within the main application). It's been easier for new developers to understand where dependencies are coming from since it's just like any other npm-using project. If the only benefit is that the root directory is cleaned up a little bit, I don't see that as being worth the cost, particularly when other dependency information for the application (the Gemfile) is already stored at root.

@matthewd

This comment has been minimized.

Show comment
Hide comment
@matthewd

matthewd Dec 5, 2016

Member

extremely difficult

@lemonmade can't you just create a top level node_modules directory if that works better for you? 😕

Member

matthewd commented Dec 5, 2016

extremely difficult

@lemonmade can't you just create a top level node_modules directory if that works better for you? 😕

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Dec 5, 2016

Member
Member

dhh commented Dec 5, 2016

@ianks

This comment has been minimized.

Show comment
Hide comment
@ianks

ianks Dec 5, 2016

Contributor

Having worked extensively with frontend tooling, I very much agree with @lemonmade. Pretty much all files that are in traditionally in will not be easily discoverable by tooling. It will end up just causing headaches for users by breaking conventions that already exist to create a new convention.

I think by default, node_modules and yarn.lock should be in the root directory. This way, the default experience will be painless as all node tooling can happily assume that executable will be in node_modules/.bin, etc. If someone wants move these to vendor, that should be fine; but IMO, don't cause integration pains for other users because you want a subjectively "cleaner" directory structure.

Contributor

ianks commented Dec 5, 2016

Having worked extensively with frontend tooling, I very much agree with @lemonmade. Pretty much all files that are in traditionally in will not be easily discoverable by tooling. It will end up just causing headaches for users by breaking conventions that already exist to create a new convention.

I think by default, node_modules and yarn.lock should be in the root directory. This way, the default experience will be painless as all node tooling can happily assume that executable will be in node_modules/.bin, etc. If someone wants move these to vendor, that should be fine; but IMO, don't cause integration pains for other users because you want a subjectively "cleaner" directory structure.

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Dec 5, 2016

Member
Member

dhh commented Dec 5, 2016

@ianks

This comment has been minimized.

Show comment
Hide comment
@ianks

ianks Dec 5, 2016

Contributor

As a counter-argument, convention over configuration is also a core value; which this implementation will invariably break. I do not think it is feasible to change the conventions of all the tooling around node to accommodate for this use case.

Here is what will happen in my case:

  1. My editor will look for a local version of mocha, tsc (typescript, or sass-lint inside of node_modules in the project root directory.
  2. When it does not find it, it will fall back to the global version of these commands
  3. These global versions will either:
    a. Not exist (best case)
    b. Exist, but potentially be a different version that what I have specified in my yarn.lock (worst case). Now, my editor is telling me things that are potentially wrong or outdated.
  4. This will cause me to waste my time attempting to reconfigure things/convincing the upstream editor to apply a patch which will fix my use case. None of this sounds like a pleasant experience (breaking core value number 1, Optimize for programmer happiness)
Contributor

ianks commented Dec 5, 2016

As a counter-argument, convention over configuration is also a core value; which this implementation will invariably break. I do not think it is feasible to change the conventions of all the tooling around node to accommodate for this use case.

Here is what will happen in my case:

  1. My editor will look for a local version of mocha, tsc (typescript, or sass-lint inside of node_modules in the project root directory.
  2. When it does not find it, it will fall back to the global version of these commands
  3. These global versions will either:
    a. Not exist (best case)
    b. Exist, but potentially be a different version that what I have specified in my yarn.lock (worst case). Now, my editor is telling me things that are potentially wrong or outdated.
  4. This will cause me to waste my time attempting to reconfigure things/convincing the upstream editor to apply a patch which will fix my use case. None of this sounds like a pleasant experience (breaking core value number 1, Optimize for programmer happiness)
@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Dec 5, 2016

Member
Member

dhh commented Dec 5, 2016

@nathanmarks

This comment has been minimized.

Show comment
Hide comment
@nathanmarks

nathanmarks Dec 6, 2016

The issue here is that in order not to have a node_modules folder at the root (which doesn't get checked in anyways), conventions are being broken across the board.

If we're using a framework like rails to begin with, it stands to reason that we enjoy the comfortable conventions that the tools we use offer. I see no reason why rails has to break these JS conventions in order to accommodate JS, a critical component of web development.

  1. As @ianks mentioned, it'll mess with editor plugins and other related tools. Everything will need custom configuration (or potentially patches) just to work with Rails. Isn't convention over configuration a big part of The Rails Way?
  2. Take a simple feature such as Heroku buildpack inference, or CI inference -- they work on the basis that there is a package.json present in the project root. Point being... these are conventions that are respected across the board. All of a sudden, further configuration here is required to get started for the simplest use cases.
  3. Historically, (and not just in rails), /vendor folders were/are used to place 3rd party dependencies that are used in the application itself, not the dev tooling used to build the project. In fact, before package management was around for JS (bower, npm, etc), using a /vendor folder was also very common for non-rails projects.
  4. Unless I'm missing something, the only way to make node module resolution work for the /vendor/node_modules folder is either by:
    a. Using $NODE_PATH which is not recommended unless there's absolutely no other way. (Especially if it means that node module resolution is changed out of the box for anyone using rails for the sake of a folder that isn't checked into version control or controlled by rails... we're not talking about moving the assets/javascripts folder to the root or anything crazy)
    b. Monkey patching node's Module object at runtime
    c. Using relative requires to get things from the node_modules folder...

I'm loving that JS is getting some love in Rails, but my personal opinion is that this particular move is going to break the "Just Works" thing.

nathanmarks commented Dec 6, 2016

The issue here is that in order not to have a node_modules folder at the root (which doesn't get checked in anyways), conventions are being broken across the board.

If we're using a framework like rails to begin with, it stands to reason that we enjoy the comfortable conventions that the tools we use offer. I see no reason why rails has to break these JS conventions in order to accommodate JS, a critical component of web development.

  1. As @ianks mentioned, it'll mess with editor plugins and other related tools. Everything will need custom configuration (or potentially patches) just to work with Rails. Isn't convention over configuration a big part of The Rails Way?
  2. Take a simple feature such as Heroku buildpack inference, or CI inference -- they work on the basis that there is a package.json present in the project root. Point being... these are conventions that are respected across the board. All of a sudden, further configuration here is required to get started for the simplest use cases.
  3. Historically, (and not just in rails), /vendor folders were/are used to place 3rd party dependencies that are used in the application itself, not the dev tooling used to build the project. In fact, before package management was around for JS (bower, npm, etc), using a /vendor folder was also very common for non-rails projects.
  4. Unless I'm missing something, the only way to make node module resolution work for the /vendor/node_modules folder is either by:
    a. Using $NODE_PATH which is not recommended unless there's absolutely no other way. (Especially if it means that node module resolution is changed out of the box for anyone using rails for the sake of a folder that isn't checked into version control or controlled by rails... we're not talking about moving the assets/javascripts folder to the root or anything crazy)
    b. Monkey patching node's Module object at runtime
    c. Using relative requires to get things from the node_modules folder...

I'm loving that JS is getting some love in Rails, but my personal opinion is that this particular move is going to break the "Just Works" thing.

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Dec 7, 2016

Member

@nathanmarks Thanks for your considerations. Good to have all this spelled out when we evaluate the current approach a bit further down the line. We're indeed using $NODE_PATH and I currently consider that to be a fully justified use. But as stated earlier: This isn't set in stone. We're going to try it for real and make it work, but if we can't, we'll reconsider. Let's resume this discussion in a month or two. In the mean time, eager to have everyone's help improving things.

#27288 is now available.

Member

dhh commented Dec 7, 2016

@nathanmarks Thanks for your considerations. Good to have all this spelled out when we evaluate the current approach a bit further down the line. We're indeed using $NODE_PATH and I currently consider that to be a fully justified use. But as stated earlier: This isn't set in stone. We're going to try it for real and make it work, but if we can't, we'll reconsider. Let's resume this discussion in a month or two. In the mean time, eager to have everyone's help improving things.

#27288 is now available.

@olivierlacan

This comment has been minimized.

Show comment
Hide comment
@olivierlacan

olivierlacan Apr 5, 2017

Contributor

For what it's worth, it might be possible to add the option to set the --modules-folder inside of a .yarnrc file (so the Rails install directory works without a binstub) to vendor/node_modules instead of the node_modules/ default.

The maintainers seem open to it.

Contributor

olivierlacan commented Apr 5, 2017

For what it's worth, it might be possible to add the option to set the --modules-folder inside of a .yarnrc file (so the Rails install directory works without a binstub) to vendor/node_modules instead of the node_modules/ default.

The maintainers seem open to it.

@gsamokovarov

This comment has been minimized.

Show comment
Hide comment
@gsamokovarov

gsamokovarov Apr 6, 2017

Contributor

In the end, David set it to the root Rails application folder in #28093.

Contributor

gsamokovarov commented Apr 6, 2017

In the end, David set it to the root Rails application folder in #28093.

@boompig

This comment has been minimized.

Show comment
Hide comment
@boompig

boompig Jun 14, 2017

Is there any documentation on whether modules-folder can be set in .yarnrc? All the google searches for me result in github issues asking for this feature.

boompig commented Jun 14, 2017

Is there any documentation on whether modules-folder can be set in .yarnrc? All the google searches for me result in github issues asking for this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment