[Bug] Checksum mismatch for package referenced as "git+https://github.com..." #2399
Comments
flipace
changed the title
This comment has been minimized.
This comment has been minimized.
4 similar comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
We couldn't reproduce your issue (all the assertions passed on master). |
|
The lockfiles aren't dependent on the system - it's likely your package that has a different content depending on where it's built. Did you download the CI-compiled version and compared it to your local one? |
|
@arcanis I just downloaded the archives of the yarn cache for both systems (you should be able to download them as well here: https://github.com/flipace/yarn2-checksum-error-giturl/actions/runs/506322343). After inspecting both with Kaleidoscope, I found that there are symbolic links in this repo (eg: https://github.com/ovos/objection.js/blob/master/doc/api/model/README.md) which seem to cause the checksum to be different, depending on the system on which the archive is generated by yarn.
Now I'm not sure whether this can be considered a bug, or it's just "by design" 😅 (since this worked fine with yarn v1) (P.S. Also, again - note that, when I rerun this github action workflow - the checksums for this package always seem to differ, even on the same "OS".) (P.P.S: |
|
@arcanis do you know whether this should be resolved on yarn side, or we should clean up the affected repos? would just like to know whether this is expected behavior and won't be changed :) |
|
Hey! I haven't had time to investigate - it's the first I hear this problem though so I'm not sure what to make of it. We don't do magic on symlinks during packing, so I'd assume that perhaps the problem comes from the representation the system makes of symlinks (for instance I think Windows treats them as regular files) - in which case there's nothing that Yarn can do 🤔 Basically, if you find what exactly Yarn does wrong we can probably fix it, but as it is I'd tend to think it's caused by something outside our control.
This looks really strange - we go to great length to ensure reproducibility, even going as far as normalizing the mtimes to make sure nothing weird can happen there. You should have identical archives for identical sources 🤔 |
I was wondering - how is the archive in this case generated? If you use a github repo url/path as a depdendency, is yarn downloading the archive from github, then extracting and recompressing it, or is it using the archive "as is" and calculates its checksum based on that downloaded zip file? |
|
It clones the repository, runs |
|
Thanks for the details! I'll try to find where exactly that thing goes off the rails this week and either close the issue or open a PR with a fix 🤞 |
|
Hello! This issue affects us, too, because our developers work on different operating systems. I don't know why this was marked unreproducible, because it's very easy to reproduce:
|
|
@josefschabasser late update from me: we've "resolved" it by removing the offending symlinks from the affected GH repo, but I understand that may not be possible for everyone. Couldn't figure it out though where exactly that goes wrong, got a bit lost in the different moving pieces of yarn itself :/ |
|
@flipace We have a monorepo and use exactly one package from github because of a security flaw we fixed ourselves. We can use patch-package to fix it for now. EDIT: ah, I just found https://yarnpkg.com/cli/patch which should make things much easier 😅 |
|
Hi! 👋 This issue looks stale, and doesn't feature the Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃 If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the |
|
Sorry, I'm late to the party. I don't know if this issue still persists as we're using yarn v2 patching now. EDIT: I'm not sure if it is reproducible with Sherlock, as it's a cross platform issue. |
|
Hi @arcanis I'm also facing this issue. I think it is a problem with yarn. The repo in my instance is this one, at the given commit - celo-org/ganache-cli@31b5bbb Referenced like this Installing locally and in a docker container produce different checksums. When I diffed the contents of the 2 packages I found this. {node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/keccak/build/Makefile | 4 ++--
node_modules_f8f6/@celo/ganache-cli/node_modules/keccak/build/Release/.deps/Release/obj.target/keccak/src/addon.o.d.raw => /dev/null | 27 ---------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/keccak/build/config.gypi | 34 +++++++++++++++++-----------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/keccak/build/keccak.target.mk | 28 ++++++++++++++--------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Makefile | 4 ++--
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/addon.o.d | 70 +++++++++++++++++++++++++++++++++++-----------------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/ecdh.o.d | 70 +++++++++++++++++++++++++++++++++++-----------------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/ecdsa.o.d | 70 +++++++++++++++++++++++++++++++++++-----------------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/privatekey.o.d | 70 +++++++++++++++++++++++++++++++++++-----------------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/publickey.o.d | 70 +++++++++++++++++++++++++++++++++++-----------------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/secp256k1-src/contrib/lax_der_parsing.o.d | 2 +-
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/secp256k1-src/contrib/lax_der_privatekey_parsing.o.d | 2 +-
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/secp256k1-src/src/secp256k1.o.d | 2 +-
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/.deps/Release/obj.target/secp256k1/src/signature.o.d | 70 +++++++++++++++++++++++++++++++++++-----------------------------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/addon.o | Bin 34576 -> 32152 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/ecdh.o | Bin 10192 -> 9984 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/ecdsa.o | Bin 21496 -> 21584 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/privatekey.o | Bin 22128 -> 21128 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/publickey.o | Bin 21736 -> 20152 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/secp256k1-src/contrib/lax_der_parsing.o | Bin 3024 -> 2656 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/secp256k1-src/contrib/lax_der_privatekey_parsing.o | Bin 3712 -> 3304 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/secp256k1-src/src/secp256k1.o | Bin 94752 -> 88136 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1/src/signature.o | Bin 11208 -> 10464 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/obj.target/secp256k1.node | Bin 164432 -> 152888 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/Release/secp256k1.node | Bin 164432 -> 152888 bytes
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/config.gypi | 34 +++++++++++++++++-----------------
{node_modules_f8f6 => node_modules_55ca}/@celo/ganache-cli/node_modules/secp256k1/build/secp256k1.target.mk | 28 ++++++++++++++--------------
27 files changed, 279 insertions(+), 306 deletions(-)One of those files is a It seems that yarn is cloning and building these repos including the c code of its dependencies, and since the environments where the repos are being built are different the packages end up with different checksums. I guess this must be a departure from how the yarn 1 cache was structured otherwise it wouldn't have been possible to use packages with dependencies on c code across different machines. Since the problem is due to building on different machines I don't think it can be reproduced by sherlock, I had a go anyway but the sherlock build never completes. |
|
Looking at this issue with a fresh eye, I feel like this issue is simply the consequence of us running the full prepack flow on installs since Yarn 2+, whereas Yarn 1 used to just clone and go. For context, we made this change because git repositories typically contain by design the package sources (for instance TS files) rather than the package artifacts (for instance the transpiled JS files) - meaning that to consume them, they usually need to be built as if you intended to publish them. There isn't a way to disable this build at the moment, so the fix would be one of those:
|
|
Hey @arcanis thanks for the context. I'm not really sure what the 'full prepack flow' is so maybe my understanding is wrong, but it seems to me that the 2 suggested fixes would not solve the problem I am facing. Although the '@celo/ganache-cli' is the problematic module having mismatching cheksums in my yarn.lock file, the reason its checksum is not consistent is because of its dependencies on secp256k1 and keccak, which in turn both make use of c or c++ code. Even if all extraneous files were removed from the final archive for secp256k1, the compiled shared object Adding a "don't build" annotation here doesn't seem to help, because packages with c or c++ code will need to be built, specifically they need to be built for each machine that they are used on and they will very likely produce different compiled shared objects for each machine. So please correct me if I'm wrong, but it seems that the current approach taken by yarn 2 is not compatible with packages that depend on those that use c or c++ code. |
|
I did some investigation on this. But I am not very familiar with the npm and yarn 1 so I may get some stuff wrong. Sound to me that the problem with If you treat a git dependency as "the package (tarball) produced by |
|
@clemyan Thank you for investigating. I'm pretty new to js/npm/yarn and was not aware of the concept of bundled dependencies, and as you point out it looks like the bundling of dependencies in After I stopped bundling those dependencies I no longer faced the same problem, so it does look like that was the source of the problem in my case. |
|
I'm able to reproduce this using yarn 3.0.1. This is the scenario:
Can this issue be re-opened? |
|
Based on the hint in @arcanis‘s comment above, I resolved this by disabling builds for my Git package. You can do so on a per-package basis using the {
"devDependencies": {
"stylelint-config-rational-order": "github:phyllisstein/stylelint-config-rational-order#head=empty-line-threshold"
},
"dependenciesMeta": {
"stylelint-config-rational-order": {
"build": false
}
}
} |
|
Having the same issue when one of the transient dependency reference this directly: Checksum generated is different on every OS, even between different linux distributions |
|
This is still an issue for us as well. It's not quite clear how to resolve it. With yarn 1.x having a dep like this: worked across all platforms (Linux, macOS and Windows). With yarn 2.x the checksums differ for each platform. |
|
Still an issue for us as well. The dependency below is not having the same checksum across all platforms. |
|
I'm having what seems to be a related issue, where the checksum of a Github package is changing but without any apparent intervention. It's referenced as There wasn't a problem until we upgraded to yarn 3.2.0, at that point the checksum started failing both locally (Mac OS) and on CI (Linux). This didn't appear to be a cross-platform issue as the same new checksum worked on both platforms, so I assumed there was some change in how the checksum was calculated (though I couldn't find anything in the changelog). However, without any further changes from us, that checksum has started failing again - the latest checksum, still on 3.2.0 and with the same commit hash for the Github package, is different both from the checksum under yarn 3.1.1 and the one that was generated when we first updated to 3.2.0. I've tried setting |
|
Just tried to migrate our setup to Yarn 3 and experienced the same checksum problem on MacOS. Downgrading to 3.1.1 seems to work. A simple (but slow) regression (I think) would be to install once on a MacOS image, clear the cache and install once more and compare the hash. There's probably a better way though. |
|
I'm having a similar problem, but not exactly the same: Different devs are getting different checksums for packages downloaded from git. For example, we have this in our I asked one of the devs with the "wrong" checksum to send me the file so I can compare with mine. Turns out the content of the Here is an excerpt of the files: |
|
We are having the same issue using Yarn Here's the We are seeing differences on identical versions of macOS (Monterey
Upon extracting the zip files and doing a recursive diff, there's only a single change between the two zip files, at the bottom of the diff -r node_modules-main/react-native-track-player/package.json node_modules-bad/react-native-track-player/package.json
86c86,87
- }
+ },
+ "packageManager": "yarn@3.2.1"So, for whatever reason, my Yarn We have the Yarn I've tried installing Yarn via Homebrew, via corepack in Homebrew, and also globally with Does anyone have any ideas what might be causing this difference in behavior? @arcanis? |
|
@brianlenz could it be related to this #2774 |
|
@purge thanks for sharing the idea! For us, it was happening on identical macOS versions. Thankfully at this point, we no longer have the issue since we no longer are targeting a specific commit hash, but it does seem that there is a bug somewhere in Yarn... |
|
we're seeing the same issue as @brianlenz where packageManager field in the cache files doesn't match the project level yarn version (3.2.2 vs 3.2.1). My guess is that this is being added here because yarn is trying to build the package since it is a git package rather than a registry one. Except that happens at a global scope (for global cache) rather than using the project version of yarn that we have and when team members have a different global version of @yarnpkg/core - they get a different built file, and checksums don't match. This is affecting new members coming on to the project as well as new CI machines, because they are getting latest yarnpkg/core (3.2.2). |
|
We've been experiencing the same behaviour described by @scrungrth above |
This updates dependencies in the package.json to their latest versions. See: yarnpkg/berry#2399
|
Having this issue in windows 10 with the following package in running install on macos works fine. Yarn v3.2.3 |
|
@arcanis Any updates on this? This is still a problem for a lot of people, including our team. We have a dependency that transitively depends on I think adding the ability to selectively ignore the checksum for specific packages can be used as a workaround until the underlying issue is fixed. Right now it's all or nothing with yarn checksum behavior. Either all checksums are checked, or all are ignored/updated which renders yarn.lock useless. This is killing so much of our time. |
|
TL;DR Could this have been fixed by 635ed55 ? I suspect the culprit was this: @brianlenz I observed something very similar to you, but with the gluegun package mentioned by @KholdStare, i.e. gives me --- gluegun-https-7779acd458-7a45a5a606/node_modules/gluegun/package.json 1984-06-22 21:50:00.000000000 +0100
+++ gluegun-https-7779acd458-7a45a5a606.docker/node_modules/gluegun/package.json 1984-06-22 21:50:00.000000000 +0100
@@ -1,11 +1,13 @@
{
"name": "gluegun",
"version": "4.3.1",
"description": "A delightful toolkit for building Node-powered CLIs.",
"repository": "infinitered/gluegun",
- "bin": "bin/gluegun",
+ "bin": {
+ "gluegun": "bin/gluegun"
+ },
"main": "build/index.js",
"types": "build/types/index.d.ts",
"scripts": {
"format": "prettier --write \"{**/*.ts,**/*.js,.circleci/**/*.js}\" --loglevel error && eslint . --fix",
"clean-build": "rimraf ./build && node ./.circleci/removeDirect.js",
@@ -212,8 +214,7 @@
},
"husky": {
"hooks": {
"pre-commit": "lint-staged"
}
- },
- "packageManager": "yarn@1.22.19"
+ }
}and sure enough inside my project which depends on outside my project, so this presumably the version you get when calling |
Confirmed; |
fix: workaround for yarnpkg/berry#2399
Since according to yarnpkg/berry#2399, the OS matters.
|
for anyone having this issue, I believe this is related #2774 (comment) |
We reference this package with `git+https:`. @0bex0 is getting a checksum error when installing it as dependency. This issue has occured before: yarnpkg/berry#2399. I inspected the zipped packages and noticed the difference is `private:` in package.json. I have therefore decided to remove the attribute and see what happens.
We reference this package with `git+https:`. @0bex0 is getting a checksum error when installing it as dependency. This issue has occured before: yarnpkg/berry#2399. I inspected the zipped packages and noticed the difference is `private:` in package.json. I have therefore decided to remove the attribute and see what happens.
and others
☝️ not sure where I'd have to look/start
Describe the bug
We are currently trying to migrate to yarn v2.
We have one package in our package.json that is defined like this:
"objection": "git+https://github.com/ovos/objection.js.git#1.6.10-mod.0"We ran
yarnand it regenerated the lock fileWhen we now run
yarn --immutableduring our CI workflow,yarnends up with an error for this package -YN0018checksum doesn't match.I noticed that yarn seems to build a new "archive" for this package (I guess because it's not packaged and published via a registry), could it be that this package looks slightly different depending on the system and this causes the checksum mismatch? (committed yarn.lock was generated on macos 11.1 whereas the CI flow is running on Debian 10)
With yarn v1 and
--frozen-lockfilethis worked fine 🤔To Reproduce
I created a separate small repo with just this dependency and 2 Github Actions to illustrate the issue:
https://github.com/flipace/yarn2-checksum-error-giturl/runs/1749446131?check_suite_focus=true
I allow the
yarnstep to update the checksums in the lockfile, and as you can see, they are different on the macos and the ubuntu jobs, and when rerunning the job, it always ends up with a different checksum...Reproduction
Screenshots
If applicable, add screenshots to help explain your problem.
Environment if relevant (please complete the following information):
The text was updated successfully, but these errors were encountered: