This repository has been archived by the owner on Aug 11, 2022. It is now read-only.
shrinkwrap - verifying dependency shasums #2649
Labels
Milestone
Comments
|
The package tarball would only change if the author pushed an update usint |
|
Right – turns out it was the package.tgz's created by npm that were differing across machines – not the original tarballs. It appears that npm stores the shasums of original tarballs in .cache.json, which would work for verification, but they are not always present for each package, or don't always contain a shasum. I'm currently digging into that. Ideally, that shasum would be available from the package.json of each package in the node_modules directory (so we don't have to rely on the cache directory being present/consistent). Currently, package.json's contain shasums of package.tgz, but as mentioned, those aren't consistent. |
|
Hm, that's a good point. The shasums in the registry are only used for verification of the download process, but the tarballs are regenerated That re-generating behavior not strictly necessary any more, but is a relic of needing to clean up broken things in the past, when we had multiple different system tar implementations in play. |
|
would this still be a big pain to tackle, or has there been much refactoring? |
|
This would be something that we can pull in scope for the multi-stage install / dependency tree realization project, but I don't want to commit to it just yet, as there are a bunch of issues I'd need to nail down first, and it may or may not make sense to tackle that within the scope of that project. (Mostly I need to figure out what to do when shasum validation fails, and how to deal with the modifications that npm makes to a package's |
|
What we've done in lockdown is to make npm fail if the shasums did not match. I'm not sure what you mean by modifications to the package.json. |
|
@isaacs npm is a lot more locked down now due to http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy I think hash validation makes sense in core |
|
@graingert It's not the case since still someone can re-publish a package with |
|
@fastman no you can't npm/npm-registry-couchapp#148 |
|
@graingert thank god I'm not using |
zkat
added a commit
that referenced
this issue
Apr 18, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4.
zkat
added a commit
that referenced
this issue
Apr 18, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4.
zkat
added a commit
that referenced
this issue
Apr 19, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4.
zkat
added a commit
that referenced
this issue
Apr 20, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4.
zkat
added a commit
that referenced
this issue
Apr 22, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4.
zkat
added a commit
that referenced
this issue
Apr 23, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4.
zkat
added a commit
that referenced
this issue
Apr 27, 2017
Fixes: #2568 Fixes: #2649 Fixes: #3141 Fixes: #4042 Fixes: #4652 Fixes: #5357 Fixes: #5509 Fixes: #5622 Fixes: #5941 All fetching-related networking is now done through pacote, and the old cache has been entirely replaced by a cacache-based one. Features: * npm now supports a variety of hash algorithms for tarball storage. On registries that support it, npm is able to use sha512sum for verification. * An `integrity` field has been added to `npm-shrinkwrap.json`. * Package integrity will be fully verified on both cache insert and extraction -- if npm installs something, it's going to be exactly what you downloaded, byte-for-byte, or it will fail. * If `npm-shrinkwrap.json` is used, npm will bypass checking package manifests and go straight to the tarball, fetching it by content address if locally cached. * Checksum integrity failures will now retry downloading on error, instead of failing on a single check. * A new npm command, `npm cache verify`, can now be used to verify and garbage collect your local cache. * npm now supports arbitrarily large tarball downloads: tarballs will no longer be loaded entirely into memory before extraction. * packages whose names only differ in casing, and packages from different sources/registries/etc will now correctly be cached separately from each other. * Some performance improvements. * Improved fetch retry logic will try harder to download your packages. BREAKING CHANGE: many shrinkwrap and cache-related things have changed. * Previously-created caches will no longer be used. They will be left in place, but data will need to be re-cached. There is no facility for rebuilding a cache based on an existing one. * `npm cache ls` has been removed for now * `npm cache rm` now always removes the entire cache. There is no granular removal available for now. * git dependencies can now use semver resolution using `#semver:^1.2.3` * `--cache-min` and `--cache-max` have been deprecated. Use `--offline`, `--prefer-offline`, and `--prefer-online instead. `--cache-min=9999+` and `--cache-max=0` have been aliased to `--prefer-offline` and `--prefer-online`, respectively. * npm will now obey HTTP caching headers sent from registries and other remote HTTP hosts, and will use standard HTTP caching rules for its local cache. * `prepublishOnly` now runs *before* packing the tarball. * npm no longer supports node@<4. fix(doctor): updated doctor command and its tests
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I seem to have hit an impasse w.r.t. adding dependency shasum verification to shrinkwrap (npm-seal has this problem as well.) The shasum of package tarballs seems to change over time, indicating that the npm registry, or the original source of the tarballs may be appending a timestamp. I want to check if that is indeed the case before proceeding.
The text was updated successfully, but these errors were encountered: