This repository has been archived by the owner. It is now read-only.

[feature] NPM Bundles/Packs #3429

Closed
arei opened this Issue May 9, 2013 · 29 comments

Comments

Projects
None yet
@arei

arei commented May 9, 2013

I would love it if I could use NPM to generate a bundle/pack (whatever you want to call it) that included the package I wanted and all of its dependencies in a single tar/zip file. This pack could then be used as an on the fly repository for installing that package and dependencies in an offline mode.

For example... on the online system do this:

npm pack express

would produce express.0.10.1.tar.gz which contains the express package and all the necessary dependencies.

On the offline system

npm install express.0.10.1.tar.gz

would install express and all its dependencies from the tar.gz file without the need to be online.

Currently, if I want to install an NPM package offline I first have to do an online NPM install, zip the entire contents, and then unzip it on the local system. I lose the benefit of having NPM install on the offline system.

I would also accept any ideas for a work around to this. I did try doing this with the npm cache stuff, but it was proving to be more work than I wanted to do.

@mfncooper

This comment has been minimized.

Show comment
Hide comment
@mfncooper

mfncooper May 10, 2013

Contributor

You can do something close to this for your app by using bundledDependencies to include all your app's dependencies, but there isn't a way to package up a subtree of your app's dependencies by itself. The problem with the latter is that installation of dependencies is not as straightforward as creating a bunch of subtrees.

Consider an app that has dependencies A and B, and where B also depends on A. If the app's dependencies are declared in package.json, npm will install A and B in the app's node_modules directory. It will not install another copy of A under B, because the copy of A that is "above" B will be found on the module load path for B. (This assumes compatible versions, of course.) Obviously this gets a lot more complex with deeper and broader dependency trees.

With the suggested packaged-up dependency subtrees, we'd end up either having a different installation behaviour, with a different result, or having npm deleting portions of the packaged subtrees after the fact to get to where it would have been if they'd been installed in the normal manner. Neither of those seem like good options to me.

Contributor

mfncooper commented May 10, 2013

You can do something close to this for your app by using bundledDependencies to include all your app's dependencies, but there isn't a way to package up a subtree of your app's dependencies by itself. The problem with the latter is that installation of dependencies is not as straightforward as creating a bunch of subtrees.

Consider an app that has dependencies A and B, and where B also depends on A. If the app's dependencies are declared in package.json, npm will install A and B in the app's node_modules directory. It will not install another copy of A under B, because the copy of A that is "above" B will be found on the module load path for B. (This assumes compatible versions, of course.) Obviously this gets a lot more complex with deeper and broader dependency trees.

With the suggested packaged-up dependency subtrees, we'd end up either having a different installation behaviour, with a different result, or having npm deleting portions of the packaged subtrees after the fact to get to where it would have been if they'd been installed in the normal manner. Neither of those seem like good options to me.

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei May 10, 2013

I believe you have missed the intent of my suggestion and for this I apologize.

My suggested npm pack command is merely about packaging a set of files (the package and its dependency tree) up into a single file for usage in an offline manner by a package install. It's not intended for usage by a package owner to help with their own package.

The command simply collects the package and all the dependencies in an uninstalled state (on down the entire dependency tree) and packages them into a single tar file. This tar file can then be used as a temporary repository for the installation of the given package at a later, offline point but still using npm install. It has nothing to do with the actual installation, only where npm is getting those files from.

Here's what I would do today to obtain this behavior...

On the online system...

npm --cache=.cache install express
tar cvfz express.tar.gz .cache
rm -rf express
rm -rf .cache

This creates the express.tar.gz file which is an archive of the .cache folder.

On the offline system...

tar xvfz express.tar.gz
npm --cache=.cache --no-registry install express
rm -rf .cache

This creates the node_modules with express and everything installed correctly within.

What I would rather do is this...

On the online system...

npm pack express

Create the express.tar.gz file.

On the offline system...

npm install express.tar.gz

Creates the node_modules with express and all its dependencies.

Is this clearer?

arei commented May 10, 2013

I believe you have missed the intent of my suggestion and for this I apologize.

My suggested npm pack command is merely about packaging a set of files (the package and its dependency tree) up into a single file for usage in an offline manner by a package install. It's not intended for usage by a package owner to help with their own package.

The command simply collects the package and all the dependencies in an uninstalled state (on down the entire dependency tree) and packages them into a single tar file. This tar file can then be used as a temporary repository for the installation of the given package at a later, offline point but still using npm install. It has nothing to do with the actual installation, only where npm is getting those files from.

Here's what I would do today to obtain this behavior...

On the online system...

npm --cache=.cache install express
tar cvfz express.tar.gz .cache
rm -rf express
rm -rf .cache

This creates the express.tar.gz file which is an archive of the .cache folder.

On the offline system...

tar xvfz express.tar.gz
npm --cache=.cache --no-registry install express
rm -rf .cache

This creates the node_modules with express and everything installed correctly within.

What I would rather do is this...

On the online system...

npm pack express

Create the express.tar.gz file.

On the offline system...

npm install express.tar.gz

Creates the node_modules with express and all its dependencies.

Is this clearer?

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei May 18, 2013

Please see https://github.com/arei/npmbox and http://www.arei.net/archives/274 for a description and an example of the behavior/implementation I am trying to describe in this feature request.

arei commented May 18, 2013

Please see https://github.com/arei/npmbox and http://www.arei.net/archives/274 for a description and an example of the behavior/implementation I am trying to describe in this feature request.

@luk-

This comment has been minimized.

Show comment
Hide comment
@luk-

luk- May 20, 2013

Contributor

We conversed a bit about this on twitter but it will be easier to discuss it here.

I'm curious as to why this is a better solution than checking in node_modules into a git repo and cloning that repo onto the machine. (this of course does not address the use of compiled dependencies that might have been built for a different architecture, version of node, etc).

Unless you're using some platform specific post-install scripts for your modules, I can't think of a reason that you wouldn't want to just check them in to git.

Can you explain the problems that result in you needing the files in an 'uninstalled state' ? My goal here isn't to criticize your feature request or the module you've written — I'm just trying to better understand the use/problem case.

Contributor

luk- commented May 20, 2013

We conversed a bit about this on twitter but it will be easier to discuss it here.

I'm curious as to why this is a better solution than checking in node_modules into a git repo and cloning that repo onto the machine. (this of course does not address the use of compiled dependencies that might have been built for a different architecture, version of node, etc).

Unless you're using some platform specific post-install scripts for your modules, I can't think of a reason that you wouldn't want to just check them in to git.

Can you explain the problems that result in you needing the files in an 'uninstalled state' ? My goal here isn't to criticize your feature request or the module you've written — I'm just trying to better understand the use/problem case.

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei May 20, 2013

I believe I have failed to state my case clearly. This may be a result of my using the word 'offline' in a different manner than might be normal. Here is the scenario with which I am faced...

Imagine two networks: NETWORK A and NETWORK B.

NETWORK A is a traditional on the Internet network we all use every day. There are no limitations to what we can do on machines connected to NETWORK A.

NETWORK B is a completely disconnected and isolated network that is not on the Internet and can never be connected to the Internet. Ever.

The only way to move data between A and B is by physical media and sneakernet.

When some npm package is required on NETWORK B here is the process we go through...

1). NETWORK A: Create a new folder somewhere.
2). NETWORK A: Install said package into new folder using node install <package>
3). NETWORK A: Tar the installed node_modules folder.
4). NETWORK A: Copy tar file onto our moveable media.
5). NETWORK A: Delete everything, clean up NETWORK A.

6). SNEAKERNET: Physically move moveable media.

7). NETWORK B: Create new folder.
8). NETWORK B: Copy file from moveable media to new folder.
9). NETWORK B: Untar tar file.
10). NETWORK B: Copy node_modules into new or existing project.
11). NETWORK B: Delete tar file, clean up NETWORK B.

The problem with this process is threefold:

A). This does not allow for npm to be used as the installing agent on NETWORK B.
B). It does not allow for global installs.
C). It does not create a very clean/repeatable process.

The goal of this feature is to do steps 1 to 5 using the npm box <package> command, and to do steps 7 to 11 using the npm unbox <npmbox-file> command.

It has the advantage of directly answering problems A, B and C as outline above.

npm box essentially creates a micro-repository from which something can be installed on NETWORK B or wherever the .npmbox file is used. Regardless of whether or not other people might have the same problems as I do, I think these kind of micro-repositories can have a real usefulness in terms of sharing. Yes, it does somewhat limit the great upgrade ability of the npm package, but sometimes that is useful.

arei commented May 20, 2013

I believe I have failed to state my case clearly. This may be a result of my using the word 'offline' in a different manner than might be normal. Here is the scenario with which I am faced...

Imagine two networks: NETWORK A and NETWORK B.

NETWORK A is a traditional on the Internet network we all use every day. There are no limitations to what we can do on machines connected to NETWORK A.

NETWORK B is a completely disconnected and isolated network that is not on the Internet and can never be connected to the Internet. Ever.

The only way to move data between A and B is by physical media and sneakernet.

When some npm package is required on NETWORK B here is the process we go through...

1). NETWORK A: Create a new folder somewhere.
2). NETWORK A: Install said package into new folder using node install <package>
3). NETWORK A: Tar the installed node_modules folder.
4). NETWORK A: Copy tar file onto our moveable media.
5). NETWORK A: Delete everything, clean up NETWORK A.

6). SNEAKERNET: Physically move moveable media.

7). NETWORK B: Create new folder.
8). NETWORK B: Copy file from moveable media to new folder.
9). NETWORK B: Untar tar file.
10). NETWORK B: Copy node_modules into new or existing project.
11). NETWORK B: Delete tar file, clean up NETWORK B.

The problem with this process is threefold:

A). This does not allow for npm to be used as the installing agent on NETWORK B.
B). It does not allow for global installs.
C). It does not create a very clean/repeatable process.

The goal of this feature is to do steps 1 to 5 using the npm box <package> command, and to do steps 7 to 11 using the npm unbox <npmbox-file> command.

It has the advantage of directly answering problems A, B and C as outline above.

npm box essentially creates a micro-repository from which something can be installed on NETWORK B or wherever the .npmbox file is used. Regardless of whether or not other people might have the same problems as I do, I think these kind of micro-repositories can have a real usefulness in terms of sharing. Yes, it does somewhat limit the great upgrade ability of the npm package, but sometimes that is useful.

@lrowe

This comment has been minimized.

Show comment
Hide comment
@lrowe

lrowe Oct 24, 2013

In the Python world we have a tool called buildout that offers the option of performing offline / repeatable installs through two methods:

  1. Configuration option find-links=<url or path to local directory>. When an exact version requirement is specified (i.e. through the equivalent of npm shrinkwrap) and this version exists in the local directory / local apache index listing then the request to the remote repository is not made.

Package downloads can then be restricted to your local mirror using the allow-links configuration option. Populating a local mirror is as simple as copying the files from the buildout download cache as they're stored under their original filenames which include the package name and version number.

https://pypi.python.org/pypi/zc.buildout/2.2.1

  1. Utility https://pypi.python.org/pypi/zc.sourcerelease which creates a tarball of all used source tgzs from the download cache so that the resulting buildout can be run offline. (Sounds similar to the npm box tool.)

https://pypi.python.org/pypi/zc.sourcerelease

I really like the find-links/allow-links option as it allows our python code to be repeatably built without resorting to making rpms every time. It would be great to have similar facilities for our growing npm install base.

lrowe commented Oct 24, 2013

In the Python world we have a tool called buildout that offers the option of performing offline / repeatable installs through two methods:

  1. Configuration option find-links=<url or path to local directory>. When an exact version requirement is specified (i.e. through the equivalent of npm shrinkwrap) and this version exists in the local directory / local apache index listing then the request to the remote repository is not made.

Package downloads can then be restricted to your local mirror using the allow-links configuration option. Populating a local mirror is as simple as copying the files from the buildout download cache as they're stored under their original filenames which include the package name and version number.

https://pypi.python.org/pypi/zc.buildout/2.2.1

  1. Utility https://pypi.python.org/pypi/zc.sourcerelease which creates a tarball of all used source tgzs from the download cache so that the resulting buildout can be run offline. (Sounds similar to the npm box tool.)

https://pypi.python.org/pypi/zc.sourcerelease

I really like the find-links/allow-links option as it allows our python code to be repeatably built without resorting to making rpms every time. It would be great to have similar facilities for our growing npm install base.

@Marcoevich

This comment has been minimized.

Show comment
Hide comment
@Marcoevich

Marcoevich Oct 30, 2013

As the creator of the issue above, I can elaborate the need for this tool in the basic npm repo. In the government organisation I work for we have many workstations on a closed intranet network. There's a way to get internet but that doesn't work for every application.

We do like to work with NodeJS. However, it can be difficult to install packages using npm install on our local intranet. A tool like npmbox makes this a lot easier.

Marcoevich commented Oct 30, 2013

As the creator of the issue above, I can elaborate the need for this tool in the basic npm repo. In the government organisation I work for we have many workstations on a closed intranet network. There's a way to get internet but that doesn't work for every application.

We do like to work with NodeJS. However, it can be difficult to install packages using npm install on our local intranet. A tool like npmbox makes this a lot easier.

@jackgill

This comment has been minimized.

Show comment
Hide comment
@jackgill

jackgill Nov 22, 2013

To give another use case for this feature: I need to install the bunyan command line tool on a machine in my production environment intended for log file analysis. The servers in my production environment do not have access to the internet (nor should they). So right now I can't use npm to install bunyan on this machine.

There is a workaround: use npm to install bunyan on my workstation, copy node_modules/bunyan to the global node_modules on the machine in production, and copy the bunyan shell script to node_modules/.bin. Definitely doable, but it seems like there's more friction than there should be. It would be nice if npm handled this, allowing me to export a package with bundled dependencies from my workstation, and then install this package globally with a single command on the production machine.

jackgill commented Nov 22, 2013

To give another use case for this feature: I need to install the bunyan command line tool on a machine in my production environment intended for log file analysis. The servers in my production environment do not have access to the internet (nor should they). So right now I can't use npm to install bunyan on this machine.

There is a workaround: use npm to install bunyan on my workstation, copy node_modules/bunyan to the global node_modules on the machine in production, and copy the bunyan shell script to node_modules/.bin. Definitely doable, but it seems like there's more friction than there should be. It would be nice if npm handled this, allowing me to export a package with bundled dependencies from my workstation, and then install this package globally with a single command on the production machine.

@luk-

This comment has been minimized.

Show comment
Hide comment
@luk-

luk- Nov 22, 2013

Contributor

You should just checkin your node_modules directory to source control then.

On Fri, Nov 22, 2013 at 11:02 AM, Jack Gill notifications@github.comwrote:

To give another use case for this feature: I need to install the bunyan
command line tool on a machine in my production environment intended for
log file analysis. The servers in my production environment do not have
access to the internet (nor should they). So right now I can't use npm to
install bunyan on this machine.

There is a workaround: use npm to install bunyan on my workstation, copy
node_modules/bunyan to the global node_modules on the machine in
production, and copy the bunyan shell script to node_modules/.bin.
Definitely doable, but it seems like there's more friction than there
should be. It would be nice if npm handled this, allowing me to export a
package with bundled dependencies from my workstation, and then install
this package globally with a single command on the production machine.


Reply to this email directly or view it on GitHubhttps://github.com/npm/npm/issues/3429#issuecomment-29099532
.

Contributor

luk- commented Nov 22, 2013

You should just checkin your node_modules directory to source control then.

On Fri, Nov 22, 2013 at 11:02 AM, Jack Gill notifications@github.comwrote:

To give another use case for this feature: I need to install the bunyan
command line tool on a machine in my production environment intended for
log file analysis. The servers in my production environment do not have
access to the internet (nor should they). So right now I can't use npm to
install bunyan on this machine.

There is a workaround: use npm to install bunyan on my workstation, copy
node_modules/bunyan to the global node_modules on the machine in
production, and copy the bunyan shell script to node_modules/.bin.
Definitely doable, but it seems like there's more friction than there
should be. It would be nice if npm handled this, allowing me to export a
package with bundled dependencies from my workstation, and then install
this package globally with a single command on the production machine.


Reply to this email directly or view it on GitHubhttps://github.com/npm/npm/issues/3429#issuecomment-29099532
.

@luk-

This comment has been minimized.

Show comment
Hide comment
@luk-

luk- Nov 22, 2013

Contributor

I'm closing this because the solution is to checkin your node_modules to version control. Node modules are javascript, you can check them in to source control without an issue. If you have native dependencies, you can run npm rebuild and rebuild them on the host platform if the architecture differs.

Contributor

luk- commented Nov 22, 2013

I'm closing this because the solution is to checkin your node_modules to version control. Node modules are javascript, you can check them in to source control without an issue. If you have native dependencies, you can run npm rebuild and rebuild them on the host platform if the architecture differs.

@luk- luk- closed this Nov 22, 2013

@Marcoevich

This comment has been minimized.

Show comment
Hide comment
@Marcoevich

Marcoevich Nov 22, 2013

Shouldn't have said this. I apologize.

Marcoevich commented Nov 22, 2013

Shouldn't have said this. I apologize.

@luk-

This comment has been minimized.

Show comment
Hide comment
@luk-

luk- Nov 22, 2013

Contributor

Why don't YOU implement this feature? Fork npm, write the feature, demonstrate its use cases and send a pull request. This is how open source works, and it's a much more efficient way of effecting change than replying with insults because someone wont spend his free time to make your life more convenient and your business more profitable.

Contributor

luk- commented Nov 22, 2013

Why don't YOU implement this feature? Fork npm, write the feature, demonstrate its use cases and send a pull request. This is how open source works, and it's a much more efficient way of effecting change than replying with insults because someone wont spend his free time to make your life more convenient and your business more profitable.

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei Nov 22, 2013

@luk- can you describe to me how you do a global install of an npm package from your source controlled node_modules? (Seriously, is there a way to do this? I don't think there is, but maybe?)

@luk- I believe some npm modules modify the environment that they are installed into (such as creating certain folders, adding to the path, etc). How can you make this occur again from a source controlled node_modules?

@luk- Imagine for a moment that your development system is also on said disconnected network and in order to keep your modules in source control you first have to move them over and need a command like this one is describing.

These are all problems that I have TODAY that could be addressed with this command.

Now, I can certainly understand that 4 people asking for this are a small minority and that such a feature is not a priority for anyone. I wouldn't dare to call closing the ticket lazy and I think the tone of @Marcoevich's comment above unnecessary, but I do think you are letting a perception of how some people work overshadow the idea of how other people work and what we are asking for. At the end of the day I still need an easy way to bundle socket.io (for example) and all of it's dependencies and all of their dependencies and etc into a single file, such that I can install from that file later on. This is what npmbox/npmunbox does for me (mostly) and it would be even more useful is someone who knew npm a little better could integrate that feature into npm more thoroughly.

Please reopen this request.

arei commented Nov 22, 2013

@luk- can you describe to me how you do a global install of an npm package from your source controlled node_modules? (Seriously, is there a way to do this? I don't think there is, but maybe?)

@luk- I believe some npm modules modify the environment that they are installed into (such as creating certain folders, adding to the path, etc). How can you make this occur again from a source controlled node_modules?

@luk- Imagine for a moment that your development system is also on said disconnected network and in order to keep your modules in source control you first have to move them over and need a command like this one is describing.

These are all problems that I have TODAY that could be addressed with this command.

Now, I can certainly understand that 4 people asking for this are a small minority and that such a feature is not a priority for anyone. I wouldn't dare to call closing the ticket lazy and I think the tone of @Marcoevich's comment above unnecessary, but I do think you are letting a perception of how some people work overshadow the idea of how other people work and what we are asking for. At the end of the day I still need an easy way to bundle socket.io (for example) and all of it's dependencies and all of their dependencies and etc into a single file, such that I can install from that file later on. This is what npmbox/npmunbox does for me (mostly) and it would be even more useful is someone who knew npm a little better could integrate that feature into npm more thoroughly.

Please reopen this request.

@jgornick

This comment has been minimized.

Show comment
Hide comment

jgornick commented Nov 22, 2013

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Nov 22, 2013

Member

I think part of the issue is that people in this thread aren't aware that npm install tarballContainingPackageJson.tgz works fine. If they knew that, maybe they wouldn't be so ungrateful and demanding?

Member

domenic commented Nov 22, 2013

I think part of the issue is that people in this thread aren't aware that npm install tarballContainingPackageJson.tgz works fine. If they knew that, maybe they wouldn't be so ungrateful and demanding?

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei Nov 22, 2013

@domenic From my understanding the .tgz version of npm install will install that package from the tgz, but go out to the registry for all of its dependencies. My need is to include all the dependencies as well.

Try this: Your assignment is to install socket.io using npm on a remote system without the internet. You can move the files via a thumb drive, but you cannot connect the remote system to the internet.

arei commented Nov 22, 2013

@domenic From my understanding the .tgz version of npm install will install that package from the tgz, but go out to the registry for all of its dependencies. My need is to include all the dependencies as well.

Try this: Your assignment is to install socket.io using npm on a remote system without the internet. You can move the files via a thumb drive, but you cannot connect the remote system to the internet.

@jgornick

This comment has been minimized.

Show comment
Hide comment
@jgornick

jgornick Nov 22, 2013

@arei Unless I don't understand your use case, but please look at https://npmjs.org/package/pac as this recursively creates an archive of your dependencies.

jgornick commented Nov 22, 2013

@arei Unless I don't understand your use case, but please look at https://npmjs.org/package/pac as this recursively creates an archive of your dependencies.

@jgornick

This comment has been minimized.

Show comment
Hide comment
@jgornick

jgornick commented Nov 22, 2013

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei Nov 22, 2013

@jgornick I just took a look at pac. It's similar to npmbox (and the feature I am fighting for here) except if I read it correctly it doesn't actually install using npm, just compresses and uncompresses a node_modules. npmbox uses the tgz file straight from npm registry, thus allowing the install to take place (much as @domenic describes above). npmbox essentially just creates a mini registry and compresses it. When you do the npmunbox on the other side, it uncompressess the mini registry and just uses it for it's install.

arei commented Nov 22, 2013

@jgornick I just took a look at pac. It's similar to npmbox (and the feature I am fighting for here) except if I read it correctly it doesn't actually install using npm, just compresses and uncompresses a node_modules. npmbox uses the tgz file straight from npm registry, thus allowing the install to take place (much as @domenic describes above). npmbox essentially just creates a mini registry and compresses it. When you do the npmunbox on the other side, it uncompressess the mini registry and just uses it for it's install.

@jackgill

This comment has been minimized.

Show comment
Hide comment
@jackgill

jackgill Nov 22, 2013

@luc - My use case is not about deploying a project which uses the bunyan library. What I want to do is install the bunyan module globally so that I can use it to parse log files generated on other machines. There is no project, and no git repo, associated with this requirement -- I just need the bunyan CLI available on this machine.

@jgornick - Pac is half the solution. The other half of the solution, which I think should come from npm, is the ability to install such an archive globally.

@domenic - npm install tarball still requires an internet connection to pull in dependencies. What we want is to be able to install a package, and all its dependencies, from a single archive without requiring an internet connection.

I chose this example because I would like to create more command line tools using Node, but right now I can't because there is no good way to install them in an offline environment. Node is heavily oriented towards creating server applications, and people tend to assume that there will always be a git repo associated with a project. The "install everything locally" approach is really good for full-blown applications, but for standalone command line utilities like a log file parser (with no associated project) you really do want to be able to install modules globally. Which npm can, as long as there is an internet connection. I don't think it's unreasonable to ask that npm also be able to install modules globally without an internet connection, using a mechanism such as that provided by npmbox.

jackgill commented Nov 22, 2013

@luc - My use case is not about deploying a project which uses the bunyan library. What I want to do is install the bunyan module globally so that I can use it to parse log files generated on other machines. There is no project, and no git repo, associated with this requirement -- I just need the bunyan CLI available on this machine.

@jgornick - Pac is half the solution. The other half of the solution, which I think should come from npm, is the ability to install such an archive globally.

@domenic - npm install tarball still requires an internet connection to pull in dependencies. What we want is to be able to install a package, and all its dependencies, from a single archive without requiring an internet connection.

I chose this example because I would like to create more command line tools using Node, but right now I can't because there is no good way to install them in an offline environment. Node is heavily oriented towards creating server applications, and people tend to assume that there will always be a git repo associated with a project. The "install everything locally" approach is really good for full-blown applications, but for standalone command line utilities like a log file parser (with no associated project) you really do want to be able to install modules globally. Which npm can, as long as there is an internet connection. I don't think it's unreasonable to ask that npm also be able to install modules globally without an internet connection, using a mechanism such as that provided by npmbox.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Nov 22, 2013

Member

@jackgill there is nothing stopping you from putting dependencies into the tarball as well. That seems to be what pac does. And there's nothing stopping you from installing pac-created tarballs (or manually created ones) with the -g flag.

Member

domenic commented Nov 22, 2013

@jackgill there is nothing stopping you from putting dependencies into the tarball as well. That seems to be what pac does. And there's nothing stopping you from installing pac-created tarballs (or manually created ones) with the -g flag.

@jackgill

This comment has been minimized.

Show comment
Hide comment
@jackgill

jackgill Nov 22, 2013

@domenic I tried that, but npm still attempts to resolve the dependencies via the online registry, even though they are bundled in the pac archive. Try this in a temporary directory:

npm install -g pac
npm install bunyan
echo '{ "dependencies": { "bunyan": "~0.22.0" } } ' > package.json
pac
npm install -g .modules/bunyan-v0.22.0.tgz

What I see after the last command is that npm downloads the dependencies from the registry, which fails in an offline environment. I did check that pac is bundling the dependencies in the tarball and it is. At this point I'm not sure if this is a bug or a missing feature. At any rate, I still can't install bunyan with npm in an offline environment.

jackgill commented Nov 22, 2013

@domenic I tried that, but npm still attempts to resolve the dependencies via the online registry, even though they are bundled in the pac archive. Try this in a temporary directory:

npm install -g pac
npm install bunyan
echo '{ "dependencies": { "bunyan": "~0.22.0" } } ' > package.json
pac
npm install -g .modules/bunyan-v0.22.0.tgz

What I see after the last command is that npm downloads the dependencies from the registry, which fails in an offline environment. I did check that pac is bundling the dependencies in the tarball and it is. At this point I'm not sure if this is a bug or a missing feature. At any rate, I still can't install bunyan with npm in an offline environment.

@edef1c

This comment has been minimized.

Show comment
Hide comment
@edef1c

edef1c Nov 24, 2013

Member

@jackgill See npm link. With bundleDependencies that should cover the entire thing.

Member

edef1c commented Nov 24, 2013

@jackgill See npm link. With bundleDependencies that should cover the entire thing.

@jackgill

This comment has been minimized.

Show comment
Hide comment
@jackgill

jackgill Nov 25, 2013

@nathan7 I'm not sure how npm link would help here -- the first problem is that npm won't create an archive containing all bundled dependencies. The second problem is that even if we have a third party utility that will do that (pac), npm won't install that archive correctly. Maybe I'm missing something but I don't see how creating symbolic links between packages will help with any of that.

As for bundleDependencies, it doesn't work (#2639 and #2442). Even if it did work, I'm not trying to bundle dependencies for my own app, I want to be able to install a third party app (bunyan) that doesn't have bundleDependencies listed in its package.json. So bundleDependencies won't really help me.

I've been looking into this further and I think it's a problem with the way npm install adds packages to the cache. I'll keep digging and see if I can come up with a definitive bug, and hopefully a definitive bugfix to go with it.

jackgill commented Nov 25, 2013

@nathan7 I'm not sure how npm link would help here -- the first problem is that npm won't create an archive containing all bundled dependencies. The second problem is that even if we have a third party utility that will do that (pac), npm won't install that archive correctly. Maybe I'm missing something but I don't see how creating symbolic links between packages will help with any of that.

As for bundleDependencies, it doesn't work (#2639 and #2442). Even if it did work, I'm not trying to bundle dependencies for my own app, I want to be able to install a third party app (bunyan) that doesn't have bundleDependencies listed in its package.json. So bundleDependencies won't really help me.

I've been looking into this further and I think it's a problem with the way npm install adds packages to the cache. I'll keep digging and see if I can come up with a definitive bug, and hopefully a definitive bugfix to go with it.

@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei Nov 27, 2013

I have rewritten and resubmitted this feature request. The new version can be found at #4210

Please post any additional thoughts or comments over there.

arei commented Nov 27, 2013

I have rewritten and resubmitted this feature request. The new version can be found at #4210

Please post any additional thoughts or comments over there.

@rlidwka

This comment has been minimized.

Show comment
Hide comment
@rlidwka

rlidwka Nov 28, 2013

Contributor

Also, don't deploy using npm. http://www.codinginthecrease.com/news_article/show/307636

@jgornick

  1. there is nothing wrong with using npm if you have a private registry
  2. there is nothing wrong with using npm if you packed all your deps into a git or a tarball

As for pac, it's simpler to just use tar cvz imho.

Contributor

rlidwka commented Nov 28, 2013

Also, don't deploy using npm. http://www.codinginthecrease.com/news_article/show/307636

@jgornick

  1. there is nothing wrong with using npm if you have a private registry
  2. there is nothing wrong with using npm if you packed all your deps into a git or a tarball

As for pac, it's simpler to just use tar cvz imho.

@jfromaniello

This comment has been minimized.

Show comment
Hide comment
@jfromaniello

jfromaniello Dec 11, 2013

I know this issue is closed and a lot has been already said, but I'd like to add my own experience:

  1. npm pack doesn't pack dependencies that are not bundled.
  2. When you use bundleDependencies, only the inmediate native dependencies are rebuild on npm install. I am not sure if this is a bug, but we are forced to run npm rebuild everytime in order to rebuild native sub dependencies.
  3. We use bundle-deps on CI before npm pack it moves all "dependencies" to "bundleDependencies" in the package.json.

jfromaniello commented Dec 11, 2013

I know this issue is closed and a lot has been already said, but I'd like to add my own experience:

  1. npm pack doesn't pack dependencies that are not bundled.
  2. When you use bundleDependencies, only the inmediate native dependencies are rebuild on npm install. I am not sure if this is a bug, but we are forced to run npm rebuild everytime in order to rebuild native sub dependencies.
  3. We use bundle-deps on CI before npm pack it moves all "dependencies" to "bundleDependencies" in the package.json.
@arei

This comment has been minimized.

Show comment
Hide comment
@arei

arei Dec 11, 2013

@jfromaniello FYI... I have rewritten and resubmitted this feature request. The new version can be found at #4210

Please post any additional thoughts or comments over there.

arei commented Dec 11, 2013

@jfromaniello FYI... I have rewritten and resubmitted this feature request. The new version can be found at #4210

Please post any additional thoughts or comments over there.

@kenberkeley

This comment has been minimized.

Show comment
Hide comment
@kenberkeley

kenberkeley Apr 15, 2016

  1. In online environment, npm install --no-bin-link. You will have a entire flattened node_modules
  2. Then, bundle this flawless node_modules with tar / zip / rar / 7z etc
  3. In offline environment, extract the bundle, that's it

P.S node-pac is another option, but it can't deal with the packages which still need downloading something for installation.

kenberkeley commented Apr 15, 2016

  1. In online environment, npm install --no-bin-link. You will have a entire flattened node_modules
  2. Then, bundle this flawless node_modules with tar / zip / rar / 7z etc
  3. In offline environment, extract the bundle, that's it

P.S node-pac is another option, but it can't deal with the packages which still need downloading something for installation.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.