[Bug] Berry fails to install itself due to sharp package build

[olingern]

Describe the bug

When installing berry dependencies from the root directory, installation fails due to node-gyp build issue.

Removing the gatsby package.json allows the build to finish for me. Not sure what package is using sharper. I took a guess and removed gatsby-transformer-sharp, but the build issue persists.

Running yarn why sharper also hangs indefinitely for me in the repository. Current nightly works as expected in projects with symlinked node_modules installed via yarn v1.x.x / npm. I'll open a separate issue if someone can confirm.

Log output:

# This file contains the result of Berry building a package (sharp@npm:0.21.3)

info sharp Using cached /home/olingern/.npm/_libvips/libvips-8.7.0-linux-x64.tar.gz
(node:8947) ExperimentalWarning: The fs.promises API is experimental
(node:8978) ExperimentalWarning: The fs.promises API is experimental
gyp info it worked if it ends with ok
gyp info using node-gyp@3.8.0
gyp info using node@10.1.0 | linux | x64
(node:8997) ExperimentalWarning: The fs.promises API is experimental
gyp info spawn /usr/bin/python2
gyp info spawn args [ '/home/olingern/code/berry/.yarn/unplugged/node-gyp-npm-3.8.0-8f62f4346137bf0043db31ef1d9cfacb368b52800dda250d652445f0308ebef7/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/home/olingern/code/berry/.yarn/unplugged/sharp-npm-0.21.3-70100537f349f4285924b9143813ff30cacd20d0556b34e835aa8dbe7189a20d/node_modules/sharp/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/olingern/code/berry/.yarn/unplugged/node-gyp-npm-3.8.0-8f62f4346137bf0043db31ef1d9cfacb368b52800dda250d652445f0308ebef7/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/olingern/.node-gyp/10.1.0/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/olingern/.node-gyp/10.1.0',
gyp info spawn args   '-Dnode_gyp_dir=/home/olingern/code/berry/.yarn/unplugged/node-gyp-npm-3.8.0-8f62f4346137bf0043db31ef1d9cfacb368b52800dda250d652445f0308ebef7/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/olingern/.node-gyp/10.1.0/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/home/olingern/code/berry/.yarn/unplugged/sharp-npm-0.21.3-70100537f349f4285924b9143813ff30cacd20d0556b34e835aa8dbe7189a20d/node_modules/sharp',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
(node:9011) ExperimentalWarning: The fs.promises API is experimental
gyp: Call to 'node -e "console.log(require('./lib/libvips').pkgConfigPath())"' returned exit status 0 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/home/olingern/code/berry/.yarn/unplugged/node-gyp-npm-3.8.0-8f62f4346137bf0043db31ef1d9cfacb368b52800dda250d652445f0308ebef7/node_modules/node-gyp/lib/configure.js:345:16)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:235:12)
gyp ERR! System Linux 4.15.0-48-generic
gyp ERR! command "/home/olingern/.nvm/versions/node/v10.1.0/bin/node" "/home/olingern/code/berry/.yarn/unplugged/node-gyp-npm-3.8.0-8f62f4346137bf0043db31ef1d9cfacb368b52800dda250d652445f0308ebef7/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/olingern/code/berry/.yarn/unplugged/sharp-npm-0.21.3-70100537f349f4285924b9143813ff30cacd20d0556b34e835aa8dbe7189a20d/node_modules/sharp
gyp ERR! node -v v10.1.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 

To Reproduce

• Navigate to root of berry project
• Run yarn

Environment if relevant (please complete the following information):

OS: Ubuntu 18.04.2 LTS
Node version: 10.1.0
Yarn version: v2.0.0

[arcanis]

Something weird I notice on your logs is that you have all those fs.promises API is experimental. Looking at the node-gyp source code, it seems likely this is what causes node-gyp to crash (it sees something in stderr, it aborts even though the exit code is 0, as evidenced by the log message).

Can you try building with Node 10.15.1? It works on my side (and I can confirm it crashes on v10.1.0).

[olingern]

I tried both 10.15.1 and 12.x.x. I'm guess this is an Ubuntu or node-gyp / python issue. I'll debug for a bit.

Also, I've tried to get the project building in an isolated environment ( Docker ), but that is tricky, too.

Here's the Dockerfile I'm working with:

FROM node:10.15.1

# Yarn ships with Node docker image.
# Use yarn to switch to berry
RUN yarn policies set-version nightly
RUN yarn policies set-version berry

# Create app directory
WORKDIR /berry

# Copy all files from curr dir to workdir
COPY . .

You can use it by:

• Placing it in berry directory root
• docker build -t berry .
• To access bash in the container: docker run -v $(pwd):/berry -it berry /bin/bash

---

Doing this, I get a configuration issue:

root@1c39b56c6bb4:/berry# yarn
Error: Unrecognized configuration settings found: version - run "yarn config -v" to see the list of settings supported in Yarn (in <environment>)

Usage: install [--frozen-lockfile] [--inline-builds]
root@1c39b56c6bb4:/berry# yarn config -v
➤ YN0034: Invalid configuration key "version" in <environment>

➤ YN0000: bstatePath               Path of the file where the current state of the built packages must be stored         '/berry/.yarn/build-state.yml'
➤ YN0000: cacheFolder              Folder where the cache files must be written                                          '/berry/.yarn/cache'
➤ YN0000: checksumBehavior         Enumeration defining what to do when a checksum doesn't match expectations            'throw'
➤ YN0000: defaultLanguageName      Default language mode that should be used when a package doesn't offer any insight    'node'
➤ YN0000: defaultProtocol          Default resolution protocol used when resolving pure semver and tag ranges            'npm:'
➤ YN0000: enableAbsoluteVirtuals   If true, the virtual symlinks will use absolute paths if required [non portable!!]    false
➤ YN0000: enableColors             If true, the CLI is allowed to use colors in its output                               true
➤ YN0000: enableGlobalCache        If true, the system-wide cache folder will be used regardless of `cache-folder`       false
➤ YN0000: enableNetwork            If false, the package manager will refuse to use the network if required to           true
➤ YN0000: enableScripts            If true, packages are allowed to have install scripts by default                      true
➤ YN0000: enableTimers             If true, the CLI is allowed to print the time spent executing commands                true
➤ YN0000: frozenInstalls           If true, prevents the install command from modifying the lockfile                     false
➤ YN0000: globalFolder             Folder where are stored the system-wide settings                                      '/root/.yarn/berry'
➤ YN0000: httpProxy                URL of the http proxy that must be used for outgoing http requests                    null
➤ YN0000: httpsProxy               URL of the http proxy that must be used for outgoing https requests                   null
➤ YN0000: ignorePath               If true, the local executable will be ignored when using the global one               true
➤ YN0000: initLicense              License used when creating packages via the init command                              null
➤ YN0000: initScope                Scope used when creating packages via the init command                                'berry'
➤ YN0000: initVersion              Version used when creating packages via the init command                              null
➤ YN0000: lastUpdateCheck          Last timestamp we checked whether new Yarn versions were available                    null
➤ YN0000: lockfileFilename         Name of the files where the Yarn dependency tree entries must be stored               'yarn.lock'
➤ YN0000: npmAlwaysAuth            URL of the selected npm registry (note: npm enterprise isn't supported)               false
➤ YN0000: npmAuthIdent             Authentication identity for the npm registry (_auth in npm and yarn v1)               null
➤ YN0000: npmAuthToken             Authentication token for the npm registry (_authToken in npm and yarn v1)             null
➤ YN0000: npmPublishAccess         Default access of the published packages                                              null
➤ YN0000: npmRegistries            Settings per registry                                                                 Map {}
➤ YN0000: npmRegistryServer        URL of the selected npm registry (note: npm enterprise isn't supported)               'https://registry.yarnpkg.com'
➤ YN0000: npmScopes                Settings per package scope                                                            Map {}
➤ YN0000: pnpDataPath              Path of the file where the PnP data (used by the loader) must be written              '/berry/.pnp.data.json'
➤ YN0000: pnpEnableInlining        If true, the PnP data will be inlined along with the generated loader                 true
➤ YN0000: pnpIgnorePattern         Regex defining a pattern of files that should use the classic resolution              null
➤ YN0000: pnpPath                  Path of the file where the PnP loader must be written                                 '/berry/.pnp.js'
➤ YN0000: pnpShebang               String to prepend to the generated PnP script                                         '#!/usr/bin/env node'
➤ YN0000: pnpUnpluggedFolder       Folder where the unplugged packages must be stored                                    '/berry/.yarn/unplugged'
➤ YN0000: preferInteractive        If true, the CLI will automatically use the interactive mode when called from a TTY   false
➤ YN0000: rcFilename               Name of the files where the configuration can be found                                '.yarnrc'
➤ YN0000: virtualFolder            Folder where the symlinks generated for virtual packages must be written              '/berry/.yarn/virtual'
➤ YN0000: yarnPath                 Path to the local executable that must be used over the global one                    '/berry/scripts/run-yarn.js'

[olingern]

Madness. @arcanis You're right about the output being the reason for failure.

Disabling node warnings via env var fixes the issue.
NODE_NO_WARNINGS=1 yarn

[arcanis]

Glad it worked! Curious that I didn't get the problem with 10.15 though 🤔

Regarding the second issue it means that there's an environment variable called YARN_VERSION (all YARN_-prefixed environment variables are injected in the configuration). If that's something Docker automatically adds we certainly can special-case it.

[olingern]

Yah, I'm not sure about 10.15.x ... is there a target Node version range berry will support, i.e. 10.x.x - 12.x.x? I guess it's only an issue if it needs to be fixed 🙂

RE env var: Indeed, it being set here.

[arcanis]

We run our tests against 8 and 10 (12 had a problem until 12.2 but I'll add it soon). We don't have a blanket policy yet, but we aim to at least be compatible with the latest releases of each major branch, and ideally more than that - with the older they get the more it becomes a best-effort basis.

RE env var: Indeed, it being set here.

Gotcha, fixed by #173