New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to avoid relative paths hell #767

Closed
slorber opened this Issue May 16, 2014 · 25 comments

Comments

Projects
None yet
8 participants
@slorber

slorber commented May 16, 2014

The paths option, that forwards to https://github.com/defunctzombie/node-browser-resolve is not documented here

In my browserify app, for my private JS code, I don't want to use relative paths because it's annoying when you move some code from one place to another to have to update all the relative paths.
See http://stackoverflow.com/a/23608416/82609

It seems the paths option permits to make available your own js files like normal node modules.
As far as I have seen it works pretty fine and the browserify app is being build without using relative paths.

The problem is that in my paths folders I have JSX React files that must be compiled to JS before being integrated by browserify.
It seems browserify doesn't transform these files unless I reference them with their relative paths.
I don't know if this is a bug or an intended behavior.
I think it could be nice, for each path, to specify if we want it to be transformed by browserify or not.
Currently my browserify bundle contains raw uncompiled JSX code, and it breaks at runtime.

Until now I did not find any clear solution on how to handle the relative paths everywhere problem. I think it's a pretty common problem that can be really annoying on large code base when refactoring. It's also solved easily with requirejs so I guess it would be nice to document how to do it with browserify.

@substack

This comment has been minimized.

Show comment
Hide comment
@substack

substack May 16, 2014

Collaborator

The problem is that there are a ton of ways to do what you're trying to do. For me, the best thing is just to put code in node_modules/ so that require() will resolve them without any configuration necessary.

Collaborator

substack commented May 16, 2014

The problem is that there are a ton of ways to do what you're trying to do. For me, the best thing is just to put code in node_modules/ so that require() will resolve them without any configuration necessary.

@substack substack closed this May 16, 2014

@slorber

This comment has been minimized.

Show comment
Hide comment
@slorber

slorber May 19, 2014

@substack ok I understand your point but still, if I put JSX or CoffeeScript files inside node_modules I don't think Browserify will process them -> this forces me to process the files outside of browserify, and move the processed files dynamically to the npm_modules

Would you do this?

slorber commented May 19, 2014

@substack ok I understand your point but still, if I put JSX or CoffeeScript files inside node_modules I don't think Browserify will process them -> this forces me to process the files outside of browserify, and move the processed files dynamically to the npm_modules

Would you do this?

@substack

This comment has been minimized.

Show comment
Hide comment
@substack

substack May 19, 2014

Collaborator

For each of your modules in node_modules/, you can configure the package.json with its own browserify transform field. Then your transforms will be applied automatically and in such a way that changes to modules will be isolated from one another. That is really handy when you need to evolve a project over time so you don't need to switch everything over at once. You can just update the pieces as you need to.

Collaborator

substack commented May 19, 2014

For each of your modules in node_modules/, you can configure the package.json with its own browserify transform field. Then your transforms will be applied automatically and in such a way that changes to modules will be isolated from one another. That is really handy when you need to evolve a project over time so you don't need to switch everything over at once. You can just update the pieces as you need to.

@slorber

This comment has been minimized.

Show comment
Hide comment
@slorber

slorber May 20, 2014

Thanks will think about it, but I'm not really convinced putting code in node_modules is the right thing to do.

Actually @substack it seems I was able to use the paths option

I have in my package.json

    "browserify": {
        "transform": [
            [
                "reactify",
                {
                    "es6": true
                }
            ],
            "browserify-shim"
        ]
    },

Not sure why but when using the paths option and not requiring through relative paths, the reactify transform doesn't seem to happen in this case with:

    return browserify({
        entries: ['./src/js/app.js'],
        extensions: ['.jsx'],
        paths: ['./node_modules','./src/js/']
    })
        .bundle({debug: true})
        .on('error', handleErrors)
        .pipe(source('app.js'))
        .pipe(gulp.dest('./www'));

But actually adding the transform directly in my Gulp code and this time it works fine!

    return browserify({
        entries: ['./src/js/app.js'],
        extensions: ['.jsx'],
        paths: ['./node_modules','./src/js/']
    })
        .transform('reactify')
        .bundle({debug: true})
        .on('error', handleErrors)
        .pipe(source('app.js'))
        .pipe(gulp.dest('./www'));

Maybe it's a bug, don't know, just wanted to let you know ;)

slorber commented May 20, 2014

Thanks will think about it, but I'm not really convinced putting code in node_modules is the right thing to do.

Actually @substack it seems I was able to use the paths option

I have in my package.json

    "browserify": {
        "transform": [
            [
                "reactify",
                {
                    "es6": true
                }
            ],
            "browserify-shim"
        ]
    },

Not sure why but when using the paths option and not requiring through relative paths, the reactify transform doesn't seem to happen in this case with:

    return browserify({
        entries: ['./src/js/app.js'],
        extensions: ['.jsx'],
        paths: ['./node_modules','./src/js/']
    })
        .bundle({debug: true})
        .on('error', handleErrors)
        .pipe(source('app.js'))
        .pipe(gulp.dest('./www'));

But actually adding the transform directly in my Gulp code and this time it works fine!

    return browserify({
        entries: ['./src/js/app.js'],
        extensions: ['.jsx'],
        paths: ['./node_modules','./src/js/']
    })
        .transform('reactify')
        .bundle({debug: true})
        .on('error', handleErrors)
        .pipe(source('app.js'))
        .pipe(gulp.dest('./www'));

Maybe it's a bug, don't know, just wanted to let you know ;)

@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Aug 13, 2014

@substack i went straight to the node_modules solution as well, the problem i'm running into is when i have cross tree dependencies, like src/main/app/node_modules/module1 relies on something in src/main/util/node_modules/module2 do you have a clever solution for this?

scamden commented Aug 13, 2014

@substack i went straight to the node_modules solution as well, the problem i'm running into is when i have cross tree dependencies, like src/main/app/node_modules/module1 relies on something in src/main/util/node_modules/module2 do you have a clever solution for this?

@slorber

This comment has been minimized.

Show comment
Hide comment
@slorber

slorber Aug 13, 2014

@scamden you can try the paths option as well maybe.

But you probably want to publish your NPM package to a private repo if they are shared between multiple projects

NPM has an entreprise offer to host private packages as SaaS, and also Gemfurry which is less expensive I think. There are also some open source solutions.

We do this for years in the Java world with Archiva / Nexus in big companies and it works fine

slorber commented Aug 13, 2014

@scamden you can try the paths option as well maybe.

But you probably want to publish your NPM package to a private repo if they are shared between multiple projects

NPM has an entreprise offer to host private packages as SaaS, and also Gemfurry which is less expensive I think. There are also some open source solutions.

We do this for years in the Java world with Archiva / Nexus in big companies and it works fine

@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Aug 13, 2014

i appreciate the reply @slorber but isn't the paths option deprecated?

thanks for the idea on publishing but doesn't it seem like overkill to have to publish your util code to it's own node module if that code is only used through your package?

scamden commented Aug 13, 2014

i appreciate the reply @slorber but isn't the paths option deprecated?

thanks for the idea on publishing but doesn't it seem like overkill to have to publish your util code to it's own node module if that code is only used through your package?

@slorber

This comment has been minimized.

Show comment
Hide comment
@slorber

slorber Aug 13, 2014

@scamden yes this seems overkill :)

I'd like to know where you see it deprecated because I actually use it and it works perfectly well.

@substack please consider not deprecating it, as people are enthusiast about this solution.

http://stackoverflow.com/questions/20158401/how-do-i-manage-relative-path-aliasing-in-multiple-grunt-browserify-bundles/23608416#23608416

vigetlabs/blendid#17 (comment)

slorber commented Aug 13, 2014

@scamden yes this seems overkill :)

I'd like to know where you see it deprecated because I actually use it and it works perfectly well.

@substack please consider not deprecating it, as people are enthusiast about this solution.

http://stackoverflow.com/questions/20158401/how-do-i-manage-relative-path-aliasing-in-multiple-grunt-browserify-bundles/23608416#23608416

vigetlabs/blendid#17 (comment)

@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Aug 13, 2014

from the browserify handbook: node also has a mechanism for searching an
array of paths, but this mechanism is deprecated and you should be using
node_modules/ unless you have a very good reason not to.

i assumed this was referring to the same thing

On 13 August 2014 12:34, Sébastien Lorber notifications@github.com wrote:

@scamden https://github.com/scamden yes this seems overkill :)

I'd like to know where you see it deprecated because I actually use it and
it works perfectly well.

@substack https://github.com/substack please consider not deprecating
it, as people are enthusiast about this solution.

http://stackoverflow.com/questions/20158401/how-do-i-manage-relative-path-aliasing-in-multiple-grunt-browserify-bundles/23608416#23608416

vigetlabs/blendid#17 (comment)
vigetlabs/blendid#17 (comment)


Reply to this email directly or view it on GitHub
#767 (comment)
.

Sterling Camden
Software Engineer
502 Emerson Street | Palo Alto, CA 94301
303.520.2968 | sterling@relateiq.com

See RelateIQ in the Wall Street Journal
http://online.wsj.com/news/articles/SB10001424127887323949904578539983425941490,

TechCrunch
http://techcrunch.com/2013/06/12/relateiq-launches-with-29m-from-formation-8-dustin-moskovitz-and-more-to-be-your-next-gen-relationship-manager/,
Forbes
http://www.forbes.com/sites/michaelsimmons/2014/01/14/the-one-thing-you-should-do-after-meeting-anyone-new/,
and More http://blog.relateiq.com/tag/relateiqinthenews/!

scamden commented Aug 13, 2014

from the browserify handbook: node also has a mechanism for searching an
array of paths, but this mechanism is deprecated and you should be using
node_modules/ unless you have a very good reason not to.

i assumed this was referring to the same thing

On 13 August 2014 12:34, Sébastien Lorber notifications@github.com wrote:

@scamden https://github.com/scamden yes this seems overkill :)

I'd like to know where you see it deprecated because I actually use it and
it works perfectly well.

@substack https://github.com/substack please consider not deprecating
it, as people are enthusiast about this solution.

http://stackoverflow.com/questions/20158401/how-do-i-manage-relative-path-aliasing-in-multiple-grunt-browserify-bundles/23608416#23608416

vigetlabs/blendid#17 (comment)
vigetlabs/blendid#17 (comment)


Reply to this email directly or view it on GitHub
#767 (comment)
.

Sterling Camden
Software Engineer
502 Emerson Street | Palo Alto, CA 94301
303.520.2968 | sterling@relateiq.com

See RelateIQ in the Wall Street Journal
http://online.wsj.com/news/articles/SB10001424127887323949904578539983425941490,

TechCrunch
http://techcrunch.com/2013/06/12/relateiq-launches-with-29m-from-formation-8-dustin-moskovitz-and-more-to-be-your-next-gen-relationship-manager/,
Forbes
http://www.forbes.com/sites/michaelsimmons/2014/01/14/the-one-thing-you-should-do-after-meeting-anyone-new/,
and More http://blog.relateiq.com/tag/relateiqinthenews/!

@substack

This comment has been minimized.

Show comment
Hide comment
@substack

substack Aug 13, 2014

Collaborator

@substack i went straight to the node_modules solution as well, the
problem i'm running into is when i have cross tree dependencies, like
src/main/app/node_modules/module1 relies on something in
src/main/util/node_modules/module2 do you have a clever solution for
this?

In module1, just do require('module2'). Here's a writeup of how the node_modules algorithm works.

In module1 when you do require('module2') these paths are searched, in order:

  1. src/main/app/node_modules/module1/node_modules/module2 (not found)
  2. src/main/app/node_modules/module2 (found! stop looking)
Collaborator

substack commented Aug 13, 2014

@substack i went straight to the node_modules solution as well, the
problem i'm running into is when i have cross tree dependencies, like
src/main/app/node_modules/module1 relies on something in
src/main/util/node_modules/module2 do you have a clever solution for
this?

In module1, just do require('module2'). Here's a writeup of how the node_modules algorithm works.

In module1 when you do require('module2') these paths are searched, in order:

  1. src/main/app/node_modules/module1/node_modules/module2 (not found)
  2. src/main/app/node_modules/module2 (found! stop looking)
@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Aug 13, 2014

@substack i think you misread my example, module2 is actually under src/main/util/ not src/main/app/

scamden commented Aug 13, 2014

@substack i think you misread my example, module2 is actually under src/main/util/ not src/main/app/

@substack

This comment has been minimized.

Show comment
Hide comment
@substack

substack Aug 13, 2014

Collaborator

I see. In that case, don't do that. Your approach does not play well with how modules work and you should move some directories around.

Collaborator

substack commented Aug 13, 2014

I see. In that case, don't do that. Your approach does not play well with how modules work and you should move some directories around.

@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Aug 13, 2014

fair enough, in the meantime how do you handle checking in to git browser code that's been put into node_modules directories for browserify without checking in "real" node_modules. I've started by git ignoring just my top level node_modules directory and any under bower_components, but wondering if you have a better thought

scamden commented Aug 13, 2014

fair enough, in the meantime how do you handle checking in to git browser code that's been put into node_modules directories for browserify without checking in "real" node_modules. I've started by git ignoring just my top level node_modules directory and any under bower_components, but wondering if you have a better thought

@slorber

This comment has been minimized.

Show comment
Hide comment
@slorber

slorber Sep 4, 2014

@scamden @substack I have updated my SO answer here as I have more insights.

http://stackoverflow.com/a/23608416/82609

NODE_PATH is not deprecated, and Browserify paths options reflect the NODE_PATH (as far as I know?)

So I don't understand why paths would be deprecated if NODE_PATH is not.
See also: https://groups.google.com/forum/#!topic/nodejs/EENwg8GAJd8

slorber commented Sep 4, 2014

@scamden @substack I have updated my SO answer here as I have more insights.

http://stackoverflow.com/a/23608416/82609

NODE_PATH is not deprecated, and Browserify paths options reflect the NODE_PATH (as far as I know?)

So I don't understand why paths would be deprecated if NODE_PATH is not.
See also: https://groups.google.com/forum/#!topic/nodejs/EENwg8GAJd8

@hughsk

This comment has been minimized.

Show comment
Hide comment
@hughsk

hughsk Sep 4, 2014

Member

@scamden use symlinks :)

These days I take the approach of having a folder of local packages, then linking the parent folder into node_modules, e.g.

ln -s ../app ./node_modules/\@myapp

Then I can just do this:

require('@myapp/box')
require('@myapp/circle')
Member

hughsk commented Sep 4, 2014

@scamden use symlinks :)

These days I take the approach of having a folder of local packages, then linking the parent folder into node_modules, e.g.

ln -s ../app ./node_modules/\@myapp

Then I can just do this:

require('@myapp/box')
require('@myapp/circle')
@slorber

This comment has been minimized.

Show comment
Hide comment
@slorber

slorber Sep 4, 2014

yes, the symlink solution is pretty good but it does not work if you have developers working on Windows :(

slorber commented Sep 4, 2014

yes, the symlink solution is pretty good but it does not work if you have developers working on Windows :(

@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Sep 9, 2014

i really like this solution. thank you guys!

On 4 September 2014 05:42, Sébastien Lorber notifications@github.com
wrote:

yes, the symlink solution is pretty good but it does not work if you have
developers working on Windows :(


Reply to this email directly or view it on GitHub
#767 (comment)
.

Sterling Camden
Software Engineer
502 Emerson Street | Palo Alto, CA 94301
303.520.2968 | sterling@relateiq.com

See RelateIQ in the Wall Street Journal
http://online.wsj.com/news/articles/SB10001424127887323949904578539983425941490,

TechCrunch
http://techcrunch.com/2013/06/12/relateiq-launches-with-29m-from-formation-8-dustin-moskovitz-and-more-to-be-your-next-gen-relationship-manager/,
Forbes
http://www.forbes.com/sites/michaelsimmons/2014/01/14/the-one-thing-you-should-do-after-meeting-anyone-new/,
and More http://blog.relateiq.com/tag/relateiqinthenews/!

scamden commented Sep 9, 2014

i really like this solution. thank you guys!

On 4 September 2014 05:42, Sébastien Lorber notifications@github.com
wrote:

yes, the symlink solution is pretty good but it does not work if you have
developers working on Windows :(


Reply to this email directly or view it on GitHub
#767 (comment)
.

Sterling Camden
Software Engineer
502 Emerson Street | Palo Alto, CA 94301
303.520.2968 | sterling@relateiq.com

See RelateIQ in the Wall Street Journal
http://online.wsj.com/news/articles/SB10001424127887323949904578539983425941490,

TechCrunch
http://techcrunch.com/2013/06/12/relateiq-launches-with-29m-from-formation-8-dustin-moskovitz-and-more-to-be-your-next-gen-relationship-manager/,
Forbes
http://www.forbes.com/sites/michaelsimmons/2014/01/14/the-one-thing-you-should-do-after-meeting-anyone-new/,
and More http://blog.relateiq.com/tag/relateiqinthenews/!

@hughsk

This comment has been minimized.

Show comment
Hide comment
@hughsk

hughsk Sep 11, 2014

Member

@slorber I just released a small tool which should work on Windows: https://github.com/hughsk/district

If it doesn't then open an issue over there and I'll look into it! :)

Member

hughsk commented Sep 11, 2014

@slorber I just released a small tool which should work on Windows: https://github.com/hughsk/district

If it doesn't then open an issue over there and I'll look into it! :)

@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden Sep 23, 2014

ok quick question here. i'm loving the symlink solution but, neither browserify-istanbul or istanbulify seem to be able to map back to the original file for requires that use the symlink. have you guys had any luck getting test coverage with this method?

scamden commented Sep 23, 2014

ok quick question here. i'm loving the symlink solution but, neither browserify-istanbul or istanbulify seem to be able to map back to the original file for requires that use the symlink. have you guys had any luck getting test coverage with this method?

@STRML

This comment has been minimized.

Show comment
Hide comment
@STRML

STRML Oct 20, 2014

@hughsk When using symlinks, do you have any ideas for avoiding having to use {global: true} in all of your transforms? I find that browserify will refuse to transform any files that are required using the symlinks.

Also, in general, I think it's kind of crazy that there isn't any node-core fix for this yet. I would really like to see syntax like require('~/app/src/foo.js') supported in core, that simply walks up the tree to the nearest package.json and resolves from there.

STRML commented Oct 20, 2014

@hughsk When using symlinks, do you have any ideas for avoiding having to use {global: true} in all of your transforms? I find that browserify will refuse to transform any files that are required using the symlinks.

Also, in general, I think it's kind of crazy that there isn't any node-core fix for this yet. I would really like to see syntax like require('~/app/src/foo.js') supported in core, that simply walks up the tree to the nearest package.json and resolves from there.

@hughsk

This comment has been minimized.

Show comment
Hide comment
@hughsk

hughsk Oct 20, 2014

Member

@STRML I just use the browserify.transform property in package.json – it means you need a package.json file for each of your subpackages but I see that as a good thing. You only need the following in there to make it work:

{
  "name": "@app/package-name",
  "version": "1.0.0",
  "browserify": {
    "transform": ["brfs"]
  }
}

There's been lots of discussion about adding a feature like you suggested to core, and as far as I'm aware it's been decided that (for better or worse) it's not going to change from what we've got now.

Member

hughsk commented Oct 20, 2014

@STRML I just use the browserify.transform property in package.json – it means you need a package.json file for each of your subpackages but I see that as a good thing. You only need the following in there to make it work:

{
  "name": "@app/package-name",
  "version": "1.0.0",
  "browserify": {
    "transform": ["brfs"]
  }
}

There's been lots of discussion about adding a feature like you suggested to core, and as far as I'm aware it's been decided that (for better or worse) it's not going to change from what we've got now.

@serapath

This comment has been minimized.

Show comment
Hide comment
@serapath

serapath May 20, 2015

Contributor

@scamden
My projects look like:

  • moduleName/
    • node_modules/ // npm install goes here
    • package.json
    • SOURCE/
      • node_modules/ // custom code goes here (potential candidates for new npm modules)
        • _config.js // my only place for relative require's (e.g. package.json)
      • index.js
    • .gitignore

In order to exclude ./node_modules but include ./node_modules/SOURCE
I do in:

#  [.gitignore]
node_modules
!SOURCE/node_modules
Contributor

serapath commented May 20, 2015

@scamden
My projects look like:

  • moduleName/
    • node_modules/ // npm install goes here
    • package.json
    • SOURCE/
      • node_modules/ // custom code goes here (potential candidates for new npm modules)
        • _config.js // my only place for relative require's (e.g. package.json)
      • index.js
    • .gitignore

In order to exclude ./node_modules but include ./node_modules/SOURCE
I do in:

#  [.gitignore]
node_modules
!SOURCE/node_modules
@scamden

This comment has been minimized.

Show comment
Hide comment
@scamden

scamden May 20, 2015

ya i did that, but i just found it confusing to have src in node_modules
directories

On Wed, 20 May 2015 at 11:19 Alexander Praetorius notifications@github.com
wrote:

@scamden https://github.com/scamden
My projects look like:

  • moduleName/
    • node_modules/ // npm install goes here
    • package.json
    • SOURCE/
      • node_modules/ // custom code goes here (potential candidates
        for new npm modules)
        • _config.js // my only place for relative require's
          (e.g. package.json)
          • index.js
          • .gitignore

In order to exclude ./node_modules but include ./node_modules/SOURCE
I do in:

[.gitignore]

node_modules
!SOURCE/node_modules


Reply to this email directly or view it on GitHub
#767 (comment)
.

scamden commented May 20, 2015

ya i did that, but i just found it confusing to have src in node_modules
directories

On Wed, 20 May 2015 at 11:19 Alexander Praetorius notifications@github.com
wrote:

@scamden https://github.com/scamden
My projects look like:

  • moduleName/
    • node_modules/ // npm install goes here
    • package.json
    • SOURCE/
      • node_modules/ // custom code goes here (potential candidates
        for new npm modules)
        • _config.js // my only place for relative require's
          (e.g. package.json)
          • index.js
          • .gitignore

In order to exclude ./node_modules but include ./node_modules/SOURCE
I do in:

[.gitignore]

node_modules
!SOURCE/node_modules


Reply to this email directly or view it on GitHub
#767 (comment)
.

@marrs

This comment has been minimized.

Show comment
Hide comment
@marrs

marrs Aug 14, 2015

@serapath That solution is already looking complicated and it doesn't show where your unit test code would go.

If NODE_PATH is a thing that's not going anywhere and already works then it seems a bit silly to discourage people from using it. I really don't care about performance. I'm already linting the entire codebase, running unit tests, who knows what else besides.

marrs commented Aug 14, 2015

@serapath That solution is already looking complicated and it doesn't show where your unit test code would go.

If NODE_PATH is a thing that's not going anywhere and already works then it seems a bit silly to discourage people from using it. I really don't care about performance. I'm already linting the entire codebase, running unit tests, who knows what else besides.

@rjdlee

This comment has been minimized.

Show comment
Hide comment
@rjdlee

rjdlee Oct 28, 2015

How do you guys deal with browserify/ babelify caching the files in the node_modules?

rjdlee commented Oct 28, 2015

How do you guys deal with browserify/ babelify caching the files in the node_modules?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment