oao bootstrap fails in parallel mode with "ENOENT: no such file or directory" for tarball in yarn cache

[edmorley]

STR:

1. Download this testcase: https://github.com/edmorley/neutrino-dev/archive/testcase-oao-parallel.zip
2. Push to a new test repo that has Travis enabled

Expected:
The Travis run passes.

Actual:
The Travis run fails with errors like:

2017-10-10T16:16:07.815Z           storyboard WARN  ┌── REVEALED PAST LOGS BEGIN HERE (due to warning/error)
2017-10-10T16:16:02.162Z                 main TRACE ┌── Run cmd yarn install --frozen-lockfile at packages/neutrino-middleware-clean [CREATED]
2017-10-10T16:16:03.870Z                 yarn INFO  | yarn install v0.28.4
2017-10-10T16:16:05.368Z                 yarn INFO  | [1/4] Resolving packages...
2017-10-10T16:16:05.719Z                 yarn INFO  | [2/4] Fetching packages...
2017-10-10T16:16:07.816Z           storyboard WARN  └── REVEALED PAST LOGS END HERE
2017-10-10T16:16:07.815Z                 yarn ERROR | error An unexpected error occurred: "https://registry.yarnpkg.com/deepmerge/-/deepmerge-1.5.2.tgz: ENOENT: no such file or directory, open '/home/travis/.cache/yarn/v1/npm-deepmerge-1.5.2-10499d868844cdad4fee0842df8c7f6f0c95a753/.yarn-tarball.tgz'".

Examples:

• Using node 6: https://travis-ci.org/edmorley/neutrino-dev/jobs/286111788
• Using node 7: https://travis-ci.org/edmorley/neutrino-dev/jobs/286111789
• Using node 8 (and a clean yarn cache): https://travis-ci.org/edmorley/neutrino-dev/jobs/286111790

This was using yarn 0.28.4 and oao 0.10.3.

Other observations:

• I can't repro locally on: Windows 10 x64 + MSYS2 bash + yarn 0.28.4 + node 8.6.0 + oao 0.10.3.
• Travis fails whether the yarn cache is empty or not

I believe this is due to yarnpkg/yarn#2629 - the summary of which is:

• yarn's cache isn't currently safe for concurrent use
• therefore callers of yarn who are doing concurrent things need to either:
  • use the --mutex option for locking (like lerna does after Use network mutex when bootstrapping with yarn lerna/lerna#764)
  • manually handle cases where things go astray and fix up the cache/retry/... -> yuck

As such I think oao needs to either:

• stop making parallel the default, since it can break the yarn cache
• start using --mutex to avoid the concurrency issues. (I'm presuming this will slow things down slightly, but presumably not as much as disabling parallel mode entirely)

[guigrpa]

Did you check whether --mutex made the problem go away?

[edmorley]

There wasn't a quick way to check it out, since it's not a one line change - a free port needs to be picked etc (if going with network mode, which seems to be preferred over file mode) - see lerna/lerna#764.

[guigrpa]

Ooops, my bad, I hadn't look at the flag's syntax. I'll look into it as soon as I'm able to breathe :)

[edmorley]

No rush - I just thought it worth filing this since we'd been using --no-parallel for a while since "otherwise it didn't work", but I couldn't remember why :-)

[guigrpa]

I believe the first errors we got when we introduced parallel mode were different, but my memory is a bit weak…

[guigrpa]

Closed due to lack of activity, reopen if necessary!