[BUG] EINTEGRITY errors for some github repos with commit hash

[aloisklink]

Update: This issue explained in my comment seems to have been fixed in v7.11.0 (see my update comment), however, another user has described a similar error in v7.20.3, so I'm unsure whether I should close this issue or leave it open.

Using a git URL in the format github:geotiffjs/geotiff.js#commit-hash on NPM v7 fails with sha-512 integrity errors. However, using a branch-name/tag pointing to the same commit works fine.

I can't seem to find another Git repo where this is failing (except for forks of geotiff.js), so I think the issue might either with Github repos with periods in their name and/or with package.json prepare scripts.

This might be related to #2919 which has a similar title, but their logs and behaviour are different.
I'm fairly certain this is an issue with the CDN, since it doesn't happen in Node v6, and if I use a get+ssh URL on a new VM, the code fails before I get a pop-up with a The authenticity of host 'github.com (140.82.121.3)' can't be established.

Current Behavior:

For the geotiff/geotiff.js repo, using a git+https://...#commit-hash URL fails with a EINTEGRITY error, due to the sha512 error, e.g.

ubuntu@gittest:~/geotiff.js$ npm pack "git+https://github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2"
npm WARN tarball tarball data for git+ssh://git@github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2 (sha512-NdxkcAm6FPXBKyhAHuYp0yMyKcBboXzf00h9hqo+yAVhAPVk6d68GH11SV9n4v2OZ9Roe1zQfCzV4TuoeskVLw==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for git+ssh://git@github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2 (sha512-NdxkcAm6FPXBKyhAHuYp0yMyKcBboXzf00h9hqo+yAVhAPVk6d68GH11SV9n4v2OZ9Roe1zQfCzV4TuoeskVLw==) seems to be corrupted. Trying again.
npm ERR! code EINTEGRITY
npm ERR! sha512-NdxkcAm6FPXBKyhAHuYp0yMyKcBboXzf00h9hqo+yAVhAPVk6d68GH11SV9n4v2OZ9Roe1zQfCzV4TuoeskVLw== integrity checksum failed when using sha512: wanted sha512-NdxkcAm6FPXBKyhAHuYp0yMyKcBboXzf00h9hqo+yAVhAPVk6d68GH11SV9n4v2OZ9Roe1zQfCzV4TuoeskVLw== but got sha512-c2I+9X+wgzfOMS8/Fn09R8BjPrjEavxj0r39aO/loMvtWlA6BNxAlatfcnW8WHhOfgbEbIRi9k3WNpJh2yoxKQ==. (709406 bytes)

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ubuntu/.npm/_logs/2021-03-26T14_59_04_649Z-debug.log

Attached logfile: 2021-03-26T14_59_04_649Z-debug.log

The same error occurs if you try to do npm install on the URL, or if you use a git+ssh url, or a short github: url: github:geotiffjs/geotiff.js#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2

Expected Behavior:

Using NPM v6.14.11, the exact same command works fine.

# ON NPM v6
ubuntu@gittest:~/geotiff.js$ npm pack "git+https://github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2"
npm notice 
npm notice 📦  geotiff@1.0.0
npm notice === Tarball Contents === 
npm notice 1.1kB   LICENSE
# ...etc

Additionally, even on NPM v7.7.5, using a branch/tag name works fine: e.g.npm pack "github:github.com/geotiffjs/geotiff.js.git#v1.0.0" works on NPMv7, even though tag v1.0.0 points to the same commit.

Steps To Reproduce:

On NPM v7.7.5, run the following command:

npm pack "git+https://github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2"

You can also try spinning up an Ubuntu VM using minipool:

• multipass launch --name npm-git-test && multipass shell npm-git-test
• install NVM then restart shell curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash
• Try working cmd on NPM v6.14.11: npm pack github:geotiffjs/geotiff.js#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2
• Update to NPM v7.7.5 and see the same command fail: npm install -g npm && npm pack github:geotiffjs/geotiff.js#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2

Environment:

I've tested this on a VM running:

• OS: Ubuntu 20.04.2 LTS
• Node: v14.16.0
• npm: 7.7.5

and a desktop running:

• OS: Ubuntu 20.04.2 LTS
• Node: v12.19.1
• npm: 7.7.5

[wraithgar]

We fixed the compression level for packing in v7.7.0 because the shasums were not consistent across all platforms (i.e. ARM vs Intel). This means you're going to have to rebuild your package-lock if you have git dependencies.

[aloisklink]

Hi @wraithgar, thanks for the info. I don't think it's due to that, I'm not sure if npm pack url even uses package-lock.json files. Plus, I tested it on v7.6.3 (see
2021-03-26T20_16_02_623Z-debug.log) and v7.0.15 and I'm getting the same issue.

Obviously, if this is only happens in a rare number of Git repos, it's probably not a priority. I thought I'd submit it in case anyone else has the same issue.

If I have time, I'll try to debug it. My gut feeling is that the issue is in npm/pacote, and that the npm prepare script in this repo is somehow changing the integrity checksum.

[sorahn]

We fixed the compression level for packing in v7.7.0 because the shasums were not consistent across all platforms (i.e. ARM vs Intel). This means you're going to have to rebuild your package-lock if you have git dependencie

Is there a way to do this that doesn't involve nuking and re-creating the package-lock.json file?

[mattijsf]

Some findings for us:

• We use modules references from github repos with commit hash
• The modules generate files at postinstall
• Entries within the package-lock.json are incompatibly with 7.7.0 and <7.7.0

Take package-lock.json with entries/checksums generated using 7.6.3:
👍 WORKS: npm install -g npm@7.6.3 && rm -rf node_modules && npm i
👎 BREAKS: npm install -g npm@7.7.0 && rm -rf node_modules && npm i

Take package-lock.json with entries/checksums generated using 7.7.0:
👎 BREAKS: npm install -g npm@7.6.3 && rm -rf node_modules && npm i
👍 WORKS: npm install -g npm@7.7.0 && rm -rf node_modules && npm i

[twschiller]

We're getting this error on Heroku with version 7.20.3

Local: Intel Core i7, Big Sur 11.4, npm 7.20.3
GitHub Actions (passing): ubuntu-latest, npm 7.19.1
Heroku (failing): Ubuntu 20.04, npm 7.20.x

Package:

"@pixiebrix/extension": "github:pixiebrix/pixiebrix-extension#5ebb3937fdb7472466e0ef3e626bc0ef0a6ae653",

I also tried pointing it at a tag which points to the same commit:

"@pixiebrix/extension": "github:pixiebrix/pixiebrix-extension#v1.2.6-alpha.1",

It works if we use a version published to npm:

"@pixiebrix/extension": "1.2.7-alpha.0",

The error when using the github repo:

Installing node modules
       npm ERR! code EINTEGRITY
       npm ERR! sha512-nFmzIf35gp4ocXNseE8i+4fwvylozooBOiXbo7fZSnySaEPSjy7k6wssX4k4j7WR60ZZOVrar/nVTjtgYSOW8A== integrity checksum failed when using sha512: wanted sha512-nFmzIf35gp4ocXNseE8i+4fwvylozooBOiXbo7fZSnySaEPSjy7k6wssX4k4j7WR60ZZOVrar/nVTjtgYSOW8A== but got sha512-7f1XYgH8Y+82qQ/fQkIslK3QMJ+g2cKU91taGEj9a4bmRceA8jHJTWZ95SDK5h94kemjv/54xD9ruq+t53+vXQ==. (2019928 bytes)

[jfirebaugh]

Is it possible this is simply a duplicate of #2846?

[wraithgar]

If not a duplicate it's the same family of issues. I'll add this issue to the body of npm/rfcs#525

[mattijsf]

Is it possible this is simply a duplicate of #2846?

I was having this problem on a intel Mac

[aloisklink]

It looks like my example/problem was fixed in NPM v7.11.0
This comment was from NPM v7.20.3 however, so I don't know if I should close this issue.

Is it possible this is simply a duplicate of #2846?

I was having this issue on x86_64 Intel & AMD Ubuntu devices.

(I've run these in a Docker container to ensure that you can reproduce these results)

7.10.0 Fails

$ docker run --rm -it node:14 sh -c 'npm install -g npm@7.10.0 && npm pack github:geotiffjs/geotiff.js#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2'
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/bin/npx -> /usr/local/lib/node_modules/npm/bin/npx-cli.js
+ npm@7.10.0
added 58 packages from 23 contributors, removed 243 packages and updated 194 packages in 9.591s
updated npm
npm WARN tarball tarball data for git+ssh://git@github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2 (sha512-enKD3uDXFcNXKLLAnl2dpc2Ps0BXC6DcHmdoCoL9kKBd34E8bZBRHKU++sctp8Cz5qho1Ii7j8GdCVTno7G6jw==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for git+ssh://git@github.com/geotiffjs/geotiff.js.git#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2 (sha512-enKD3uDXFcNXKLLAnl2dpc2Ps0BXC6DcHmdoCoL9kKBd34E8bZBRHKU++sctp8Cz5qho1Ii7j8GdCVTno7G6jw==) seems to be corrupted. Trying again.
npm notice 
npm notice New major version of npm available! 7.10.0 -> 8.4.1
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.4.1
npm notice Run npm install -g npm@8.4.1 to update!
npm notice 
npm ERR! code EINTEGRITY
npm ERR! sha512-enKD3uDXFcNXKLLAnl2dpc2Ps0BXC6DcHmdoCoL9kKBd34E8bZBRHKU++sctp8Cz5qho1Ii7j8GdCVTno7G6jw== integrity checksum failed when using sha512: wanted sha512-enKD3uDXFcNXKLLAnl2dpc2Ps0BXC6DcHmdoCoL9kKBd34E8bZBRHKU++sctp8Cz5qho1Ii7j8GdCVTno7G6jw== but got sha512-rBh+f8B8N273VbPmeggM3CmvCKdgNLjEvGJh/4RvBIfV18Bn9Xrq8i7W5RjkwUbmMdmvldojOUiCJbW4GWktkg==. (708773 bytes)

7.11.0 Success

$ docker run --rm -it node:14 sh -c 'npm install -g npm@7.11.0 && npm pack github:geotiffjs/geotiff.js#c01b2dcf00d5a324cd27c38709cb6ca97adf01e2
/usr/local/bin/npm -> /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/bin/npx -> /usr/local/lib/node_modules/npm/bin/npx-cli.js
+ npm@7.11.0
added 60 packages from 23 contributors, removed 243 packages and updated 194 packages in 8.856s
npm notice 
npm notice package: geotiff@1.0.0
npm notice === Tarball Contents === 
npm notice 1.1kB   LICENSE                                    
npm notice 16.2kB  README.md                                  
npm notice 64.0kB  dist-browser/decoder.worker.1936c0d9.js    
[...Removed to save space]

[nlf]

this is fixed in npm@8.5.2 where we have stopped evaluating integrity strings for git dependencies

[Rayyan98]

@nlf I am on a mono-repo where in the cicd pipeline some dependencies are built and packed and then installed into the service. I am facing a similar error where after some retires and warning of tarball integrity it throws an npm ERR! code ENOENT. As per your comment integrity string check are stopped for git dependencies but what about for local tar files.

The commit hashes were committed to lock file by building and installing the dependencies into the service on my local setup (mac). The cicd pipeline runs on (linux). The command i am using to install the dependencies in the service in npm ci

[wraithgar]

Integrity checks are done on tarballs, yes. The creation of tarballs is not guaranteed to always have the same integrity, this is a known limitation of node itself due to the subtly different ways it implements things like compression across architectures.