New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yarn install sporadically fails with integrity check failure #6407

Open
kayneb opened this Issue Sep 19, 2018 · 36 comments

Comments

Projects
None yet
@kayneb

kayneb commented Sep 19, 2018

Do you want to request a feature or report a bug?
bug

What is the current behavior?
When running yarn install (on our CI) we are finding that random dependencies fail to download due to integrity check failures:

e.g.

yarn install --pure-lockfile error https://registry.yarnpkg.com/react-style-proptype/-/react-style-proptype-3.2.1.tgz: Integrity check failed for "react-style-proptype" (computed integrity doesn't match our records, got "sha512-znIMso2zjWI0V3Ne7hQcOB5HOXdPpJ0HsGw4GBs/uEmVLaN46acxHi5B0PXwOnEzSQ46hp0arB+khsFh/sVIxg==")

If the current behavior is a bug, please provide the steps to reproduce.
Have a somewhat yarn project setup with a somewhat large number of dependencies - keep yarn installing until integrity check fails.

What is the expected behavior?
No integrity checks occur

Please mention your node.js, yarn and operating system version.
node: v10.9.0
yarn: v1.10.0
OS: NixOS (Linux)

This could possibly be occurring due to network failures on our end, but I don't consider this likely considering our build machines are EC2 instances with solid internet connections.

As a workaround for now, we are rolling back to v1.9.4

@heupr heupr bot assigned arcanis Sep 19, 2018

@heupr heupr bot added the triaged label Sep 19, 2018

@arcanis

This comment has been minimized.

Member

arcanis commented Sep 19, 2018

As a workaround for now, we are rolling back to v1.9.4

Are you sure this doesn't happen with the 1.9.4? Afaik this is caused by the npm registry sometimes returning 500 at random (the npm client retries until it works, but we don't do this yet), so this should affect all Yarn releases.

@kayneb

This comment has been minimized.

@arcanis

This comment has been minimized.

Member

arcanis commented Sep 20, 2018

I know :) But the integrity errors are caused by the registry returning 500 errors that are not properly reported as such.

@Dahaden

This comment has been minimized.

Dahaden commented Sep 21, 2018

@arcanis We have been testing for flakiness recently, just reverting yarn back to 1.9.4 has taken our builds from a 36% success rate to 88%

@arcanis

This comment has been minimized.

Member

arcanis commented Sep 21, 2018

@Dahaden Can you share details regarding the errors? In particular:

  • do you use --frozen-lockfile?
  • did your lockfile contain the new integrity field, or was it the same one than for the 1.9.4?

The 1.10 ships with the new integrity feature, which requires us to migrate your lockfiles by querying the npm registry for each dependency in order to add the field. If done during CI, it could increase the opportunities for the registry to return an 500 somewhere. If the integrity fields were added before being checked-in, that would alleviate the issue (since the CI wouldn't have to query the network for this anymore) cc @imsnif

If you have the opportunity to retry the 1.10, can you add the following line into a .yarnrc file at the root of your repository? This should prevent Yarn from adding the integrity field if it is missing, which will help us determine if this is the cause of the issue or not.

unsafe-disable-integrity-migration true
@imsnif

This comment has been minimized.

Member

imsnif commented Sep 21, 2018

Yeah, as @arcanis said, the migration to 1.10 is more network intensive. If on a flaky network (or if the npm registry is having networking issues at the time) it might explain it. Following.

@arcanis

This comment has been minimized.

Member

arcanis commented Sep 21, 2018

This should hopefully be mitigated with #6413

@kayneb

This comment has been minimized.

kayneb commented Sep 26, 2018

We did populate the integrity field in the lockfile prior to pushing, so it should be all good.
Awesome thanks for that improvement @arcanis! Will definitely look to bump to 1.11.0, including this fix.

@fqborges

This comment has been minimized.

fqborges commented Oct 5, 2018

If you have the opportunity to retry the 1.10, can you add the following line into a .yarnrc file at the root of your repository? This should prevent Yarn from adding the integrity field if it is missing, which will help us determine if this is the cause of the issue or not.

unsafe-disable-integrity-migration true

In my case, this .yarnrc workaround did the job.

@szrenwei

This comment has been minimized.

szrenwei commented Oct 9, 2018

@Dahaden Can you share details regarding the errors? In particular:

  • do you use --frozen-lockfile?
  • did your lockfile contain the new integrity field, or was it the same one than for the 1.9.4?

The 1.10 ships with the new integrity feature, which requires us to migrate your lockfiles by querying the npm registry for each dependency in order to add the field. If done during CI, it could increase the opportunities for the registry to return an 500 somewhere. If the integrity fields were added before being checked-in, that would alleviate the issue (since the CI wouldn't have to query the network for this anymore) cc @imsnif

If you have the opportunity to retry the 1.10, can you add the following line into a .yarnrc file at the root of your repository? This should prevent Yarn from adding the integrity field if it is missing, which will help us determine if this is the cause of the issue or not.

unsafe-disable-integrity-migration true

add a .yarnrc file also solved my problem, thanks.

@shunia

This comment has been minimized.

shunia commented Oct 18, 2018

unsafe-disable-integrity-migration true

is working

@viviansteller

This comment has been minimized.

viviansteller commented Nov 6, 2018

yarn config set unsafe-disable-integrity-migration true -g

@sibelius

This comment has been minimized.

sibelius commented Nov 7, 2018

One package that keeps failing the integrity check is this one https://github.com/nodkz/mongodb-memory-server

I hope this helps

@sibelius

This comment has been minimized.

sibelius commented Nov 8, 2018

yarn config set unsafe-disable-integrity-migration true -g does not work anymore on version 1.12.3

anyway to fix this? or to disable integrity check?

@arcanis

This comment has been minimized.

Member

arcanis commented Nov 8, 2018

That seems strange, we have a CI test that uses it. Can you share the output of yarn config list --verbose ?

@arcanis

This comment has been minimized.

Member

arcanis commented Nov 8, 2018

Minus your possible npm auth tokens, of course (we maybe should remove it from the output ...)!

@sibelius

This comment has been minimized.

sibelius commented Nov 8, 2018

another strange thing is that if I do the following things:

  • remove yarn.lock
  • run yarn (it will change from sha1 to sha512)
  • remove yarn.lock
  • run yarn (it will change from sha512 to sha1)
  • this will keep happening

config:

yarn config v1.12.3
verbose 0.282 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.npmrc".
verbose 0.283 Checking for configuration file "/Users/sibelius/.npmrc".
verbose 0.283 Found configuration file "/Users/sibelius/.npmrc".
verbose 0.284 Checking for configuration file "/usr/local/etc/npmrc".
verbose 0.284 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/Dev/e/f/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/Dev/e/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/Dev/.npmrc".
verbose 0.285 Checking for configuration file "/Users/sibelius/.npmrc".
verbose 0.285 Found configuration file "/Users/sibelius/.npmrc".
verbose 0.286 Checking for configuration file "/Users/.npmrc".
verbose 0.29 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.29 Found configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.29 Checking for configuration file "/Users/sibelius/.yarnrc".
verbose 0.29 Found configuration file "/Users/sibelius/.yarnrc".
verbose 0.291 Checking for configuration file "/usr/local/etc/yarnrc".
verbose 0.291 Checking for configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.291 Found configuration file "/Users/sibelius/Dev/e/f/f-server/.yarnrc".
verbose 0.291 Checking for configuration file "/Users/sibelius/Dev/e/f/.yarnrc".
verbose 0.291 Found configuration file "/Users/sibelius/Dev/e/f/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/sibelius/Dev/e/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/sibelius/Dev/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/sibelius/.yarnrc".
verbose 0.292 Found configuration file "/Users/sibelius/.yarnrc".
verbose 0.292 Checking for configuration file "/Users/.yarnrc".
verbose 0.301 current time: 2018-11-08T11:36:25.250Z
info yarn config
{ 'version-tag-prefix':
  'v',
 'version-git-tag':
  true,
 'version-commit-hooks':
  true,
 'version-git-sign':
  false,
 'version-git-message':
  'v%s',
 'init-version':
  '1.0.0',
 'init-license':
  'MIT',
 'save-prefix':
  '^',
 'bin-links':
  true,
 'ignore-scripts':
  false,
 'ignore-optional':
  false,
 registry:
  'https://registry.yarnpkg.com',
 'strict-ssl':
  true,
 'user-agent':
  'yarn/1.12.3 npm/? node/v10.13.0 darwin x64',
 lastUpdateCheck:
  1541611519487,
 'unsafe-disable-integrity-migration':
  true,
 'yarn-offline-mirror':
  '/Users/sibelius/Dev/e/f/f-server/yarn-offline-cache',
 'yarn-offline-mirror-pruning':
  true,
 'experimental-pack-script-packages-in-mirror':
  true }
info npm config
{ 'init.author.name':
  'Sibelius Seraphini',
 'init.author.email':
  'sibeliusseraphini@gmai.com',
 'init.author.url':
  'https://github.com/sibelius',
 '//registry.npmjs.org/:_authToken':
  'blah',
 progress:
  true,
 python:
  '/usr/bin/python' }

yarn-offline-mirror could be the problem?

@sibelius

This comment has been minimized.

sibelius commented Nov 8, 2018

prebuiltVariants keeps changing as well

@sibelius

This comment has been minimized.

sibelius commented Nov 9, 2018

here are two prints of what keep happening with mongodb-memory-server

image

image

mongodb-memory-server@2.7.0:
  version "2.7.0"
  resolved "https://registry.yarnpkg.com/mongodb-memory-server/-/mongodb-memory-server-2.7.0.tgz#663345e8fe38e3b76c703fcc94f691c192fcbd66"
- integrity sha512-T9zBEN3/y7/s4F83B2jAlLHtjjSEp50GQ2J0b7QMbAwM/G7Rkxzdf3cCfzOChDBfI0lQto09EOTdDam6mm0REQ==
+ integrity sha1-M2tbFYi0Q8ExxGJmGySYCdh3/qY=
  dependencies:
    "@babel/runtime" "^7.1.2"
    debug "^4.1.0"

then

mongodb-memory-server@2.7.0:
  version "2.7.0"
  resolved "https://registry.yarnpkg.com/mongodb-memory-server/-/mongodb-memory-server-2.7.0.tgz#663345e8fe38e3b76c703fcc94f691c192fcbd66"
-integrity sha512-T9zBEN3/y7/s4F83B2jAlLHtjjSEp50GQ2J0b7QMbAwM/G7Rkxzdf3cCfzOChDBfI0lQto09EOTdDam6mm0REQ==
+integrity sha1-M2tbFYi0Q8ExxGJmGySYCdh3/qY=
  dependencies:
    "@babel/runtime" "^7.1.2"
    debug "^4.1.0"

same machine, same yarn (I'm using 1.12.3 now)

@arcanis

This comment has been minimized.

Member

arcanis commented Nov 9, 2018

Hmm - ping @imsnif?

@imsnif

This comment has been minimized.

Member

imsnif commented Nov 10, 2018

Thanks for the ping @arcanis!

@sibelius - I cannot reproduce this issue with that package. What I do:

  • switch to yarn 1.12.3
  • mkdir foo
  • cd foo
  • yarn init
  • yarn add mongodb-memory-server@2.7.0 ===> the package is populated in yarn.lock with sha512
  • rm -rf node_modules
  • rm yarn.lock
  • yarn ===> still sha512

I repeated this process a few times and still go the same results. A few options:

  1. We get the sha512 integrity from the npm registry. There might have been a temporary issue that caused this?
  2. Are you perhaps using a private registry proxy? (eg. verdaccio, artifactory etc.?)

If this still happens to you consistently, I'd be very happy to be able to reproduce and debug this. For this I need your help: could you trim an example repo down to the minimum needed to reproduce this consistently (eg. package.json, yarn.lock, yarnrc etc.), make sure it reproduces for you if you move it around (ideally to a different box) and then attach it here?

EDIT: (just to add an explanation)
The package manifest we get from the registry (be it the global npm registry or a private one) includes an integrity field. Some of them are sha512, and some (older ones or ones published with yarn before the integrity patch) are sha1. We use that hash algorithm as the source of truth since we have no choice (if we have sha512 locally and for some reason the repository returns sha1 we have to use sha1 for authentication).
It is conceivable that a package would change from sha512 to sha1 across different versions (eg. if it was published by a different client without sha512 capabilities), but I can't think of a reason for it to happen in the same version. My mind jumps to a registry issue (which would be consistent with the sporadicness of this all) but I have nothing to back this up yet. :)

Thanks for reporting this and helping out. I really want to get to the bottom of these because I've seen them pop up here and there and could never reproduce consistently.

@sibelius

This comment has been minimized.

sibelius commented Nov 10, 2018

this is my .yarnrc

yarn-offline-mirror "./yarn-offline-cache"
yarn-offline-mirror-pruning true
experimental-pack-script-packages-in-mirror true

this is happening in a yarn workspaces, I'm gonna try to create a repro

tks for the help

@sibelius

This comment has been minimized.

sibelius commented Nov 10, 2018

here is a repro using workspaces

https://github.com/sibelius/yarn-workspaces-mms

check the commits where I change yarn.lock doing the below commands:

  • delete yarn.lock
  • yarn
  • delete yarn lock
  • yarn
@imsnif

This comment has been minimized.

Member

imsnif commented Nov 10, 2018

The only diff I'm getting is this:

--- a/yarn.lock
+++ b/yarn.lock
@@ -721,7 +721,7 @@ mongodb-memory-server@2.7.0:
     tmp "^0.0.33"
     uuid "^3.2.1"
   prebuiltVariants:
-    mongodb-memory-server-v2.7.0-darwin-x64-64 d5c108b5d7e74032ed4f62b3ec5dc9568b8b0e3b
+    mongodb-memory-server-v2.7.0-linux-x64-64 edf91278f9941e8d2fbd47bf203ec21af50fdb8c

which is not ideal, but out of scope for now. :)

How often does this happen to? I tried 3-4 times and everything stays a-ok.

@sibelius

This comment has been minimized.

sibelius commented Nov 10, 2018

@imsnif

This comment has been minimized.

Member

imsnif commented Nov 10, 2018

For sure. I just need to be able to reproduce it, otherwise I'm just guessing. Does it happen to you every time with this repo?

@sibelius

This comment has been minimized.

sibelius commented Nov 10, 2018

not every yarn install, try to delete node_modules and yarn.lock and try to do yarn again

this is my mac os x version 10.14.1, not sure if this helps or not

node v10.12.0, using nvm 0.31.1

@imsnif

This comment has been minimized.

Member

imsnif commented Nov 10, 2018

I've now run it 10 times and no reproduction. I'd really like to check and see if you're getting a different response from the registry for some reason. If you have the jq json parser, would you mind running this command a few times and seeing if you sometimes get sha1 returned?

curl --silent https://registry.yarnpkg.com/mongodb-memory-server/ | jq '.versions."2.7.0".dist.integrity'

BTW - I assume you don't start out with the local cache folder, right?

@sibelius

This comment has been minimized.

sibelius commented Nov 10, 2018

it keeps returning sha512

I think the problem is with the offline cache, I've disabled this for now

and it keeps on sha512

I've used the cache because native packages always recompiled, but this is another issue

@imsnif

This comment has been minimized.

Member

imsnif commented Nov 11, 2018

Hey @sibelius, that's quite interesting. If you delete the offline cache and then re-enable it, does the issue reoccur?

@sibelius

This comment has been minimized.

sibelius commented Nov 11, 2018

I have many .yarnrc in a lot of different folders, because I work in a lot of different projects

I've deleted all of them, and for now, it is working fine

not sure if this was a problem with some old yarn-offline-cache folder

@imsnif

This comment has been minimized.

Member

imsnif commented Nov 12, 2018

Alright, so the problematic path seems to be here: https://github.com/yarnpkg/yarn/blob/master/src/resolvers/registries/npm-resolver.js#L170

I'm guessing (all I can do right now unfortunately*) that we go to the offline mirror to get the integrity from there, we get the old sha1 integrity and are satisfied with it and thus yarn.lock is updated with the old sha1. What I'm not fully understanding is why this happens sporadically. If this is the case, why is the offline mirror bypassed every now and then. @arcanis - do you have any ideas?

I wouldn't want to solve this by always going to the registry to get the integrity for obvious reasons - but I would love to solve the flakiness here as soon as we understand it better.

*This might theoretically be reproducible by starting an offline mirror, artificially creating a manifest that has only sha1 integrity and running this a few times. If there aren't any ideas, I'll try to get to that soon.

@sibelius

This comment has been minimized.

sibelius commented Nov 12, 2018

can we do not update yarn.lock when resolving from offline cache?

@imsnif

This comment has been minimized.

Member

imsnif commented Nov 12, 2018

@sibelius - perhaps, but I'd first like to make sure this is indeed the issue before we implement the solution.

@aprilandjan

This comment has been minimized.

aprilandjan commented Dec 1, 2018

Encounter this problems with yarn v1.12.3 when trying to yarn a project provided by private registry. Tried unsafe-disable-integrity-migration true but no luck.

I had to use yvm and manually install previous versions of yarn and found that the version v1.10.1 solved my issue:

yvm install 1.10.1
yvm exec 1.10.1 install/add
@Axure

This comment has been minimized.

Axure commented Dec 1, 2018

It could happen when you are using yarn local cache with the yarn registry set to a 3rd party mirror which does not yet provide the integrity field, e.g. the taobao mirror https://registry.npm.taobao.org.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment