unique file entries in integrity file

[hulkish]

Summary

Fixes #6703

Test plan
After installing dependencies with this modified yarn, run:

cat node_modules/.yarn-integrity | jq -r .files[] | uniq -c

You should only see a 1 next to each entry.

[arcanis]

It seems good, but for posterity can you detail why duplicate entries can even occur? The old code was iterating over readdir, so I'm not sure I understand how it can happen

[buildsize]

File name	Previous Size	New Size	Change
yarn-[version].noarch.rpm	1.1 MB	1.1 MB	-504 bytes (0%)
yarn-[version].js	4.47 MB	4.46 MB	-6.52 KB (0%)
yarn-legacy-[version].js	4.66 MB	4.65 MB	-6.42 KB (0%)
yarn-v[version].tar.gz	1.12 MB	1.11 MB	-3.14 KB (0%)
yarn_[version]all.deb	814.52 KB	813.57 KB	-974 bytes (0%)

[hulkish]

It seems good, but for posterity can you detail why duplicate entries can even occur? The old code was iterating over readdir, so I'm not sure I understand how it can happen

@arcanis @rally25rs So, I've reached the root cause of this issue. Which, lead me to a better solution for this. Basically, modulesFolders was incorrectly being populated with 2 entries of the workspace root dir. I think after you see my new changes it will become more clear.

[arcanis]

I think I'd prefer to keep using the for...of style that the previous code was using. This particular chaining of the array methods makes the code more confusing imo and doesn't help Flow.

[hulkish]

@arcanis So, im happy to restore the sequential awaits. I just thought this would be preferred in terms of perf. Flow barfs on comparing custom types with Object.values() entries.

[arcanis]

There is no await here, right?

[arcanis]

Shouldn't this lambda return loc && (await fs.exists(loc)) instead?

[hulkish]

@arcanis you would be correct, if it were using filter instead of map

[arcanis]

You've removed this part of the code without replacement - won't that break the modulesFolder settings?

[hulkish]

@arcanis this was actually causing the issue, since these dirs are included already.

For ex, i added the following log:

    console.log(
      this.config.modulesFolder,
      path.join(this.config.lockfileFolder, constants.NODE_MODULES_FOLDER),
      Object.values(workspaceLayout.workspaces)
          .filter(Boolean).map(({loc}) => loc)
    );

which yields:

undefined '/Users/myuser/dev/repos/my-repo/node_modules' [ '/Users/myuser/dev/repos/my-repo/components',
  '/Users/myuser/dev/repos/my-repo/contexts',
  '/Users/myuser/dev/repos/my-repo/modules',
  '/Users/myuser/dev/repos/my-repo/pages',
  '/Users/myuser/dev/repos/my-repo/packages/browser-dev-tools',
  '/Users/myuser/dev/repos/my-repo/packages/dal',
  '/Users/myuser/dev/repos/my-repo/packages/dev-certs',
  '/Users/myuser/dev/repos/my-repo/packages/env',
  '/Users/myuser/dev/repos/my-repo/packages/koa-auth-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/koa-error-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/koa-transform-multistats-assets',
  '/Users/myuser/dev/repos/my-repo/packages/koa-wc-headers-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/koa-wc-health-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/koa-wc-logging-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/koa-wc-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/koa-webpack-dev-middleware',
  '/Users/myuser/dev/repos/my-repo/packages/logger',
  '/Users/myuser/dev/repos/my-repo/packages/module-interface',
  '/Users/myuser/dev/repos/my-repo/packages/react-render-to-stream',
  '/Users/myuser/dev/repos/my-repo/packages/redux-create-store',
  '/Users/myuser/dev/repos/my-repo/packages/server',
  '/Users/myuser/dev/repos/my-repo/services/akamai-headers',
  '/Users/myuser/dev/repos/my-repo/services/hello-world',
  '/Users/myuser/dev/repos/my-repo/services/redux-dal',
  '/Users/myuser/dev/repos/my-repo/services/storybook',
  '/Users/myuser/dev/repos/my-repo/tools/cli',
  '/Users/myuser/dev/repos/my-repo/tools/lib-src-resolver',
  '/Users/myuser/dev/repos/my-repo/tools/platform-cache',
  '/Users/myuser/dev/repos/my-repo/tools/platform-config',
  '/Users/myuser/dev/repos/my-repo/tools/platform-core',
  '/Users/myuser/dev/repos/my-repo/tools/progress-webpack-plugin',
  '/Users/myuser/dev/repos/my-repo/tools/storybook-helpers',
  '/Users/myuser/dev/repos/my-repo/tools/webpack-chunk-assets-plugin',
  '/Users/myuser/dev/repos/my-repo' ]

[jeanregisser]

@hulkish are you still interested in getting this PR merged?

I came across this problem too as detailed in celo-org/celo-monorepo#2798