Support npm tags in general

[guybedford]

See jspm/jspm-cli#704.

Basically we need to be reading all the tags in dist-tags and adding them to the versions lookup object.

[lookfirst]

I come back to this idea again... instead of manually downloading files, why not wrap the npm binary (or use their api?). Files get downloaded to ./node_modules. You then copy them into jspm_packages from ./node_modules and do all of your source code transforms at that point.

Basically, what you've done is rewrite npm which seems like a whackamole issue.

[guybedford]

This implementation is only 905 lines, it's hardly rewriting npm, and the API is really simple.

[lookfirst]

130 lines. https://github.com/lookfirst/honk/blob/master/src/NpmRepository.js

[zcregan]

I strongly agree with @lookfirst's idea of wrapping npm. My team constantly comes across edge cases that just don't work the same as npm itself. Things like this bug, package.json's postinstall script, npm scoped modules + npm's new private modules (https://www.npmjs.com/private-modules). I'm sure there are others I can't remember. You would get all of this for free, as well as any new features npm adds in the future.

In addition to this, you also wouldn't be required to run jspm registry config npm. I'd make this exact same argument for the GitHub endpoint as well. Most devs already have git credential API plugins installed and configured on their systems. Jspm bypasses all of this and forces devs to type their GitHub credentials into jspm (devs hate typing their credentials into anything...).

We've had quite a bit of pushback moving to jspm on our team because of all the necessary npm+github configuration only to be met with edge case limitations that don't exist with Browserify.

[lookfirst]

@zcregan 👍

It isn't a comparison about lines of code as that is a silly metric in this case. Re-implementing the mechanism for downloading npm pacakges is just plain wrong.

I just had a friend try to use jspm, the first thing he ran into was the Github api limits. He was so confused as to why all the sudden there was a mess of errors. The error handling in that case is also whacky.

[guybedford]

There's an issue at #58 to move the reading of auth to happen directly from the local npmrc file on startup.

With that, it won't be necessary to run jspm registry config npm.

Similarly we've just added support for the netrc in the GitHub endpoint so if that file is present, jspm doesn't need to store credentials either.

Yes I know the error messages are bad - we need to be able to distinguish between network and other errors. I've created jspm/jspm-cli#718 to track this.

[zcregan]

@guybedford I'm still pretty convinced that the more correct solution would be to use npm directly. Would you be open to starting a discussion on the idea?

[guybedford]

@zcregan my apologies but this is not something I'm willing to discuss at length. A short summary - jspm handles version resolutions. npm will also try to handle version resolutions but downloading duplicated dependencies into hierarchical node_modules folders. Downloading packages from npm is as simple as downloading a tarball, which jspm implements very simply here. Most of the code in this project is actually converting the JS files from npm into something SystemJS can understand. Calling out to npm to install means that npm would be installing version resolutions into node_modules. Immediately we'd have a messy API boundary copying packages out of node_modules with unnecessary downloading happening of deep resolutions which may not even match jspm's resolutions. These costs would need to outweigh a greater benefit, which is really not clear.

[lookfirst]

@guybedford

Ideally you would extract all that code that does the munging away from the download process. Right now the chain of promises is intertwined with the chain of downloading and it makes the code extremely hard to follow, test and re-use.

The way around getting the nested dependencies is to first flatten the tree that you ask npm to download and to let npm do the version resolution for you (using npm/npm-remote-ls). I'm not really sure how you pick which dependencies of projects to leave out, but downloading a couple extra files isn't going to kill anyone, especially since npm handles all the caching for you anyway.

Anyway, the problems you've noted actually have good solutions. I've proven this technique with my little honk experiment. It was amazingly fast to install a bunch of stuff.

[guybedford]

If you're looking to follow the code, the download implementation is just https://github.com/jspm/npm/blob/master/npm.js#L393 and the cached lookup is https://github.com/jspm/npm/blob/master/npm.js#L229. All the rest is the conversion. The issue of duplicate downloads specifically comes up when dealing with multi-version.

[crisptrutski]

JSPM does not support version aliases, which is what npm's dist tags are effectively. @guybedford is skeptical about the usefulness/necessity of aliases, so not planning to support dist tags for now.

[zcregan]

npm tags are extremely useful for development. Especially when used with jspm link. You can jspm link npm:myPackage@dev and then publish updates to your package with npm publish --tag dev and then you don't have to constantly juggle jspm install --link and jspm install --unlink when commiting changes in development.

And to be frank, it really shouldn't matter whether you're "skeptical about the usefulness/necessity" of any npm features. The goal should be for as close to 100% npm package compatibility as reasonably possible.

Another example of things just not working in jspm: jspm install npm:react-components. jspm doesn't support git commit hashes. At my company we have one team using Browserify and another team has been trying to use jspm. The Browserify guys rarely have any issues, things just seem to work for them. On our jspm team, it seems about half the things we try to do with jspm just don't work. Simply installing npm packages is pretty hit or miss.

npm is aware of and committed to fixing it's problems with front-end package management (http://blog.npmjs.org/post/101775448305/npm-and-front-end-packaging). They're doing quite a bit of work to address many of the concerns that jspm solves (https://github.com/npm/npm/wiki/Roadmap) in particular, dependency flattening/deduping. Currently, jspm just completely bypasses all of this work.

@guybedford you said that the benefits of piggy-backing on npm are not clear. The benefits are simply that jspm would have vastly improved npm compatibility. Again, I'll list a number of very useful and widely used npm features that jspm doesn't support:

• npm tags
• package.json lifecycle scripts
• npm scoped modules + npm private modules
• git commit referencing
• using .npmrc configuration directly without having to copy configuration out into jspm

As far as how you would go about implementing such a thing. I would npm install into a node_modules directory inside of the jspm_packages directory, then run npm dedupe, then traverse over node_modules to perform any remaining deduping/flattening and version resolution that npm couldn't handle itself, and then any necessary node transforms, and then write the results into jspm_packages. This would solve all of the issues I listed above (and probably other edge cases as well).

[guybedford]

@zcregan thanks for describing a great use case example of how this feature is useful. Reopening to implement.

In terms of the other features you mention, scoped modules and private modules are supported on master now as well as the .npmrc configuration use directly. Working to get the release out as soon as possible.

Git commit referencing is something we can certainly consider in due course as well.

Thanks for your patience - completely agree parity is the goal as far as is possible, it just takes work to get to that point.

[lookfirst]

👍

[adamdicarlo]

all my 👍 👍 for wrapping npm so that scoped modules can work and the scope configuration (private npm registry) from ~/.npmrc is used.

[guybedford]

Both of these features are in the coming release.

[peteruithoven]

Is there an update on this? I see there was a new release 0.25.0, but I can't find releasenotes.

[glen-84]

I just ran into this while trying to install foundation-sites@beta. Perhaps this should be documented in the meantime.

[guybedford]

This was supported thanks to the PR by @adamburgess.