Ruby version 3.2.1 build failed due to Psych error in on macOS

[yonkeltron]

Hey I am writing to let you know about a build issue with Ruby 3.2.1. I am on macOS and here's the relevant output:

Error building 3.2.1:

$ ruby-build 3.2.1 ~/tmp
To follow progress, use 'tail -f /var/folders/yv/phpj4jg95zj_9pjy_wgh88rm0000gp/T/ruby-build.20230329103305.93390.log' or pass --verbose
Downloading openssl-3.0.8.tar.gz...
-> https://dqw8nmjcqpjn7.cloudfront.net/6c13d2bf38fdf31eac3ce2a347073673f5d63263398f1f69d0df4a41253e4b3e
Installing openssl-3.0.8...
Installed openssl-3.0.8 to /Users/USERNAME/tmp

Downloading ruby-3.2.1.tar.gz...
-> https://cache.ruby-lang.org/pub/ruby/3.2/ruby-3.2.1.tar.gz
Installing ruby-3.2.1...
ruby-build: using readline from homebrew
ruby-build: using gmp from homebrew

BUILD FAILED (macOS 12.6.3 using ruby-build 20230309)

Inspect or clean up the working tree at /var/folders/yv/phpj4jg95zj_9pjy_wgh88rm0000gp/T/ruby-build.20230329103305.93390.GXEpqM
Results logged to /var/folders/yv/phpj4jg95zj_9pjy_wgh88rm0000gp/T/ruby-build.20230329103305.93390.log

Last 10 log lines:
  --with-openssl-dir=/Users/USERNAME/tmp/openssl
  --enable-shared
  --with-readline-dir=/usr/local/opt/readline
  --with-gmp-dir=/usr/local/opt/gmp
  --enable-yjit
  --with-libyaml-dir=/usr/local/opt/libyaml
  --disable-install-doc
  CC=clang
  LDFLAGS=-L/Users/USERNAME/tmp/lib
  CPPFLAGS=-I/Users/USERNAME/tmp/include

$ tail -n 20 /var/folders/yv/phpj4jg95zj_9pjy_wgh88rm0000gp/T/ruby-build.20230329103305.93390.log
                                    rbs 2.8.2
                                    typeprof 0.21.3
                                    debug 1.7.1
installing bundled gem cache:       /Users/USERNAME/tmp/lib/ruby/gems/3.2.0/cache
/Users/USERNAME/tmp/lib/ruby/3.2.0/yaml.rb:3: warning: It seems your ruby installation is missing psych (for YAML output).
To eliminate this warning, please install libyaml and reinstall your ruby.
The Ruby yaml extension was not compiled.
ERROR: Ruby install aborted due to missing extensions
Configure options used:
  --prefix=/Users/USERNAME/tmp
  --with-openssl-dir=/Users/USERNAME/tmp/openssl
  --enable-shared
  --with-readline-dir=/usr/local/opt/readline
  --with-gmp-dir=/usr/local/opt/gmp
  --enable-yjit
  --with-libyaml-dir=/usr/local/opt/libyaml
  --disable-install-doc
  CC=clang
  LDFLAGS=-L/Users/USERNAME/tmp/lib
  CPPFLAGS=-I/Users/USERNAME/tmp/include 

$ tmp/bin/ruby -e "require 'psych'; require 'yaml'; puts YAML.dump('abc')"
<internal:/Users/USERNAME/tmp/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:151:in `require': dlopen(/Users/USERNAME/.gem/ruby/3.2.0/gems/psych-5.1.0/lib/psych.bundle, 0x0009): Library not loaded: '/Users/USERNAME/.asdf/installs/ruby/3.2.1/lib/libruby.3.2.dylib' (LoadError)
  Referenced from: '/Users/USERNAME/.gem/ruby/3.2.0/gems/psych-5.1.0/lib/psych.bundle'
  Reason: tried: '/Users/USERNAME/.asdf/installs/ruby/3.2.1/lib/libruby.3.2.dylib' (no such file), '/usr/local/lib/libruby.3.2.dylib' (no such file), '/usr/lib/libruby.3.2.dylib' (no such file) - /Users/USERNAME/.gem/ruby/3.2.0/gems/psych-5.1.0/lib/psych.bundle
	from <internal:/Users/USERNAME/tmp/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:151:in `require'
	from /Users/USERNAME/.gem/ruby/3.2.0/gems/psych-5.1.0/lib/psych.rb:13:in `<top (required)>'
	from <internal:/Users/USERNAME/tmp/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:99:in `require'
	from <internal:/Users/USERNAME/tmp/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:99:in `require'
	from -e:1:in `<main>'

No problem building 3.0.5 (as asked to test in #2101):

ruby-build 3.0.5 ~/tmp
To follow progress, use 'tail -f /var/folders/yv/phpj4jg95zj_9pjy_wgh88rm0000gp/T/ruby-build.20230329104957.90324.log' or pass --verbose
Downloading openssl-1.1.1t.tar.gz...
-> https://dqw8nmjcqpjn7.cloudfront.net/8dee9b24bdb1dcbf0c3d1e9b02fb8f6bf22165e807f45adeb7c9677536859d3b
Installing openssl-1.1.1t...
Installed openssl-1.1.1t to /Users/USERNAME/tmp

Downloading ruby-3.0.5.tar.gz...
-> https://cache.ruby-lang.org/pub/ruby/3.0/ruby-3.0.5.tar.gz
Installing ruby-3.0.5...
ruby-build: using readline from homebrew
ruby-build: using gmp from homebrew
Installed ruby-3.0.5 to /Users/USERNAME/tmp

Some info on my environment:

$ uname -a
Darwin HOSTNAME.tld 21.6.0 Darwin Kernel Version 21.6.0: Mon Dec 19 20:44:01 PST 2022; root:xnu-8020.240.18~2/RELEASE_X86_64 x86_64

$ brew info ruby-build
==> ruby-build: stable 20230309 (bottled), HEAD
Install various Ruby versions and implementations
https://github.com/rbenv/ruby-build
/usr/local/Cellar/ruby-build/20230309 (574 files, 297.5KB) *
  Poured from bottle using the formulae.brew.sh API on 2023-03-09 at 09:16:31
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/ruby-build.rb
License: MIT
==> Dependencies
Required: autoconf ✔, pkg-config ✔, readline ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
ruby-build installs a non-Homebrew OpenSSL for each Ruby version installed and these are never upgraded.

To link Rubies to Homebrew's OpenSSL 1.1 (which is upgraded) add the following
to your shell profile e.g. ~/.profile or ~/.zshrc:
  export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@1.1)"

Note: this may interfere with building old versions of Ruby (e.g <2.4) that use
OpenSSL <1.1.
==> Analytics
install: 1,502 (30 days), 213,582 (90 days), 2,395,453 (365 days)
install-on-request: 596 (30 days), 118,303 (90 days), 1,887,672 (365 days)
build-error: 0 (30 days)

$ ruby-build --version
ruby-build 20230309

What else can I provide in terms of information and/or context?

Thanks,
+Jonathan

[mislav]

Hi, could you nuke /Users/USERNAME/.gem/ruby/3.2.0 and try this again?

rm -rf ~/.gem/ruby/3.2.0
~/tmp/bin/ruby -e "require 'psych'; require 'yaml'; puts YAML.dump('abc')"

If that doesn't change anything, can you check your environment variables for any that have GEM or RUBY in the name

[yonkeltron]

@mislav Oddly, that totally worked! Let me see if I can reproduce the failure now that I blew away the gem dir there.

[mislav]

@hsbt: I've managed to reproduce this; do you think it might be a Ruby build bug? A broken psych gem install under ~/.gem can sabotage Ruby installation:

1. Use any Ruby 3.2 version to gem i psych --user-install;
2. Manually “break” the ~/.gem/ruby/3.2.0/gems/psych-*/lib/psych.rb file by prepending raise LoadError.new("BOOM") to its code;
3. Now try to make a fresh install of Ruby 3.2.1;
4. The make install step fails with “It seems your ruby installation is missing psych”, which is misleading. Psych was installed into the fresh Ruby version just fine! It was the previously broken user-installed gem that's giving trouble.

I think that Ruby make && make install should ideally work even if there is a broken gem under ~/.gem/ruby/<version>/gems. Gems installed into user install location can get broken for a variety of reasons (maybe they're linked to libraries that don't exist on the system anymore), but that shouldn't abort Ruby installation. Maybe Ruby should isolate itself from gem user-install location during make install?

[hsbt]

In my understanding, Abort of ERROR: Ruby install aborted due to missing extensions is happend by ruby-build, not ruby installer like make && make install. ruby-build try to check openssl, readline, zlib, yaml extensions from user environment.

Should we ignore GEM_HOME environmental variable on build_package_verify_openssl()?

[mislav]

@hsbt It's true that ruby-build additionally checks whether things like openssl and libyaml have been linked well, but this particular failure isn't from ruby-build, because in my reproduction case I got it from make install. https://github.com/ruby/ruby/blob/v3_1_2/lib/yaml.rb#L3-L6

[yonkeltron]

This seems to have resolved this issue, thanks for your help. It completely baffled me, so I am super grateful for the assistance you were able to offer. Please accept my gratitude for this and for all your efforts in service of the community!

Thanks,
+Jonathan