integrity checksum failed when using sha1 #1230
Comments
|
Are you sure the packages were not republished under the same version? The npm registry used to allow that and I believe the admins still have that ability. Also, if you use sinopia/verdaccio, they allow overriding already published versions of packages. |
|
But in this repro, the shrinkwrap.yaml file has been completely deleted. So essentially PNPM is encountering an integrity mismatch with itself, right? If so that would seem like a PNPM bug, not a registry issue... |
|
Also the error only repros for certain people. Other people are able to build with exactly the same:
The only consistent difference seems to be the computer itself. Some computers have this problem for some weird reason. (?) And then eventually the person fiddles with stuff and eventually it stops happening. We've had 4 different people encounter this in the past couple months. |
|
Where do these checksums come from? Is there a REST endpoint on the registry that returns them? How does PNPM decide which algorithm to use when it (presumably) hashes the tarball that it downloaded? |
|
Integrity verification happens here The integrity checksum is returned by the registry, with the package metafile |
|
Do you use verdaccio? I've seen such issue with verdaccio. What package did it fail on? |
This is a normal install that's failing. Do you want us to provide a verbose log? Also, we have at least one person who is blocked by this error. He can't do any work on his PC. Would a possible workaround be to comment out the integrity check? Promise.all([
Promise.resolve(), // opts.integrity && ssri.checkStream(res.body, opts.integrity),
unpackStream.local(res.body, tempLocation, { |
|
This is the callstack FYI: |
|
yes, commenting that line should solve the issue. But removing shrinkwrap.yaml should solve it as well |
|
I think, as a temporary fix, we can add a Sounds good? |
This is absolutely NOT the case. PNPM fails with this error even if there is no shrinkwrap file.
So this would temporarily disable the checks entirely? Maybe a better hack would be to say "if the hashes cannot be compared because they are different algorithms, then ignore them and don't fail the install." |
|
The whole log file might help, I think it might also be a network issue.
no, the algorithm is fine, there are two integrities: sha1 and sha256. When only sha1 is available then it will be used |
yes, but I just recalled that this check is also used for retrying. Sometimes due to some network issues, not the whole tarball is returned. Checking the integrity helps to catch that issue and redownload |
|
After investigating, we found that there are really two bugs here:
|
|
I am also facing the same issue with sha1. 26018 verbose node v10.1.0 |
|
You are using an unstable release of NodeJs. Might want to check that first. Also are you using PNPM or NPM? It's a little unclear from your log. |
|
@zkochan following up from our previous conversation: People are still hitting this issue occasionally. I got a couple people to confirm that As a workaround (until we finally get this solved), we're thinking about adding a rush command-line option that would set the |
It works fine with npm/yarn. |
|
@pulse14 could you attach the tarball and the metafile of |
|
https://drive.google.com/file/d/1Adle6gzo3gmrSpOy7ouy9mEQe7mfbGpF/view?usp=sharing I tried to reset verdaccio cache multiple times |
|
I couldn't reproduce it. oh, I see node is 8.9.3 |
|
ok, reproduced, I had to install more packages to make some load on verdaccio |
|
ok, I have found it, this is a regression. The fetch retry options have no default values |
|
AWESOME! |
|
BTW we should also fix the error message so it says something like "The network connection was reset" instead of "integrity checksum failed". That would really help the end user perception of the issue: The "integrity checksum failed" sounds like PNPM is corrupting its own data stream, which is not the case. (#1235) |
|
🚢 2.13.4 I'll try to mock the registry and cover it with tests later |
|
Thanks again for fixing @zkochan ! |
|
no problem |
|
People have confirmed that this fixes their repros for this issue. Hurray! |
|
So, is this fixed or not? I'm encountering an integrity error too, even with I wish the error included the failing package, so I could remove it from the cache and refetch. There is no shrinkwrap.yml yet, and I removed the package-lock.json to be sure. Output: The previous run failed here: Based on that I took a look here: filesize matches, so I removed that directory, and Is this a different bug? Maybe the store should automatically retry fetching? Or at the very least not store intermediate downloads at the final location? |
|
@wmertens please create a separate issue I thought we only rename the tarball if the integrity was correct but looking at the code I am not sure |
|
Are you sure it's an actual integrity error? Based on #1235 it seemed that network failures sometimes get reported as "integrity checksum failed" because the original error does not prevent the integrity check from happening and producing a secondary error. |
|
@pgonzal yes, the file was truncated |
|
We can try something like the following: If a tarball download fails, we retry the download (as we currently do, we retry it up to 2 times by default) and we automatically reduce |
|
This GitHub issue is now closed. Should we open a separate GitHub issue to track this? People continue to report this issue to us every week or so (admittedly at a very large company with lots of developers). The It would be helpful to automate the workaround. Long term it would be ideal to add appropriate logging so we can eventually figure out the root cause and fix it. Currently the real error is obscured by #1235 (which seems like it would be pretty simple fix). |
|
If my proposed solution sounds good, we can create an issue for it |
|
but as I reread the issue, initially it was due to fetch-retries having a bad default number. Setting it to 2 solved most of the issues, so maybe setting a bigger number of retries is the solution. Users may set any number they want. We can add more details to the error message, something like: |
|
BTW if you wanted to publish a prototype of an idea, it's easy for me to find a person who's encountering the issue and get them to test whether it solves the problem or not. |
|
sorry I dropped the ball on this. My problem was actually badly downloaded files being treated as correct - downloading them multiple times won't fix that underlying problem. I'll make an issue. |
pnpm version: 2.2.1
Code to reproduce the issue:
Don't have a shareable deterministic repro...
Expected behavior:
When performing an install pnpm should use the same SHA algorithm as server is using when checking for integrity.
Actual behavior:
We get an error like:
We verified that our NPM registry is definitely returning SHA1 hashes that match the expected value. It appears that PNPM is sometimes comparing the wrong hash.
Any ideas/logs I could provide?
This has been affecting several members of our team, but not others. Sometimes we can fix it by cleaning out the store or not using the shrinkwrap file, but today we got a deterministic repro on 2 machines.
Additional information:
node -vprints: 8.9.4The text was updated successfully, but these errors were encountered: