Improve performance and add support for versions like ~1.2.3 and ^1.2.3 #16
Conversation
It adds support for version numbers like: * `1` - get the latest `1.x.x` * `1.2` - get the latest `1.2.x` * `^1.2.3` - get the latest `1.x.x` but at least `1.2.3` * `~1.2.3` - get the latest `1.2.x` but at least `1.2.3` so this module could be used to test most versions found in package.json dependencies. It adds tests for the new version formats. It adds examples to the readme.
This new test makes sure that the data returned after changes in 5e7ee01 is still the same as before even though the download is significantly faster and many times smaller: time curl -s https://registry.npmjs.org/express | wc -c time curl -s https://registry.npmjs.org/express/4.0.0 | wc -c
rsp
changed the title
rsp
changed the title
| var scope = name.split('/')[0]; | ||
| var url = registryUrl(scope) + | ||
| encodeURIComponent(name).replace(/^%40/, '@'); | ||
| encodeURIComponent(name).replace(/^%40/, '@') + | ||
| (!isScoped && version ? '/' + version : ''); |
There was a problem hiding this comment.
Though I agree this is more efficient, please read issue #7.
They are deprecating the version URL.
There was a problem hiding this comment.
@destroyerofbuilds If that is the case (which would be a very strange decision for a good API, I have to admit - the data for all versions will only get bigger over time and eventually it will become a problem) then I guess the PR shouldn't be merged as is. Thanks for pointing it out. Do you maybe have some idea how to get the real version number that would be installed by npm having a string like "~1.2.3" - that wouldn't use the deprecated API that currently supports such versions as part of the URL? Thanks.
There was a problem hiding this comment.
You could use the semver module to match against the list of versions returned.
There was a problem hiding this comment.
@sindresorhus I'm using semver module in the PR #17, see: https://github.com/sindresorhus/package-json/pull/17/files
|
👎 Because this approach depends on an API end-point that the npm team has indicated may be removed. |
It adds support for wildcard, caret and tilde versions for both scoped and unscoped packages so this module could be used to test most versions found in package.json dependencies: * 1 - get the latest 1.x.x * 1.2 - get the latest 1.2.x * ^1.2.3 - get the latest 1.x.x but at least 1.2.3 * ~1.2.3 - get the latest 1.2.x but at least 1.2.3 But it keeps downloading the entire data for all versions from the npm registry - see comments to PR sindresorhus#16 at sindresorhus#16 It adds tests for the new version formats and makes sure that the data returned after this pull request is still the same as before. It adds examples to the readme.
|
@destroyerofbuilds Thanks for your comments. I prepared a new PR #17 that addresses all your concerns. It adds support for versions like |
|
Wish I could merge this, but don't want to depend on an API the owners has explicitly said will go away at some point. I hope you can add a comment on https://github.com/npm/public-api/issues/11, though. It's really needed. |
|
@sindresorhus OK, I understand. @destroyerofbuilds pointed it out before so I created PR #17 that adds support for versions like ^1.2.3 and ~1.2.3 but without using that API - using the semver module instead. PR #17 adds the functionality to search for wildcard versions and ranges but doesn't improve performance like using the single version API would - e.g. for express you download 500kB just to throw out everything but 2kB immediately, see: It may not seem like a huge problem for some people but I think it's a huge waste of resources and it will only get worse with every new version published. |
|
@rsp I agree, but the alternative is using an API that might disappear any moment and lacks some features. The better solution is to push npm to add an officially supported API for this as mentioned above. |
|
@sindresorhus I agree. I'll comment on npm/public-api#11 that you linked to. Meanwhile there's PR #17 that doesn't use that API but the semver module instead. |
It improves performance by downloading only the data of the version that is requested, instead of downloading data about every published version in the history of the package only to throw most of it away, as was the case before.
The difference is significant, for some modules like express it is 2kB vs 500kB, see:
time curl -s https://registry.npmjs.org/express | wc -c
time curl -s https://registry.npmjs.org/express/4.0.0 | wc -c
This is not only a waste of time and bandwidth from the perspective of the module user but also a significant waste of resources of the npm registry.
This PR also adds support for version numbers like:
1- get the latest1.x.x1.2- get the latest1.2.x^1.2.3- get the latest1.x.xbut at least1.2.3~1.2.3- get the latest1.2.xbut at least1.2.3so this module could be used to test most versions found in package.json dependencies.
It adds tests for the new version formats and makes sure that the data returned after this pull request is still the same as before.
It adds examples to the readme.