Ruby can't compile with psych when libyaml isn't installed #552

eileencodes · 2022-03-30T19:58:03Z

Hello! Since I pulled Ruby master yesterday I'm unable to configure or run make successfully due to an error that the digest file is missing. This only happens when libyaml isn't installed - if I install libyaml then everything works correctly. However the code that's failing is supposed to download libyaml and instead raises the following error:

/Users/eileencodes/open_source/ruby/tool/extlibs.rb:6:in `require': cannot load such file -- digest (LoadError)
	from /Users/eileencodes/open_source/ruby/tool/extlibs.rb:6:in `<top (required)>'
	from /Users/eileencodes/open_source/ruby/ext/psych/extconf.rb:22:in `require_relative'
	from /Users/eileencodes/open_source/ruby/ext/psych/extconf.rb:22:in `<top (required)>'
	from ./ext/extmk.rb:214:in `load'
	from ./ext/extmk.rb:214:in `block in extmake'
	from /Users/eileencodes/open_source/ruby/lib/mkmf.rb:324:in `open'
	from ./ext/extmk.rb:210:in `extmake'
	from ./ext/extmk.rb:572:in `block in <main>'
	from ./ext/extmk.rb:568:in `each'
	from ./ext/extmk.rb:568:in `<main>'
make[1]: *** [ext/psych/exts.mk] Error 1
make[1]: *** Waiting for unfinished jobs....

I know that there were a lot of recent changes to psych and libyaml is no longer bundled with it. On my machine where libyaml is not installed when I run make on my Ruby checkout (or configure) it will hit this line https://github.com/ruby/ruby/blob/master/ext/psych/extconf.rb#L22 which loads tool/extlibs.rb. That file then has a require 'digest' which is where it is failing to build and cannot load the file. Since it fails to require the digest file, the download_cache is never used. I can consistently reproduce if I don't have libyaml installed. If I manually install it with homebrew, Ruby builds fine.

Machine details:

MacOS Monterey, M1 Max
Ruby master as of github.com/ruby/ruby/0cb43034b416aaf28bf57ee34bc095fdcb659f33

Steps to reproduce:

Uninstall libyaml or ensure it is not installed
make clean
Run configure script, mine below:

autoconf
./configure --prefix=$HOME/.rbenv/versions/ruby-trunk --disable-install-rdoc --with-openssl-dir=$(brew --prefix openssl@1.1) optflags='-O3 -DNDEBUG' debug='-ggdb'
make -j16

You can also see it on each rebuild with make -j16 if it failed to configure so you don't need to re-run configure over and over.

The text was updated successfully, but these errors were encountered:

eileencodes · 2022-03-31T11:16:52Z

When I run rake compile in psych with_config("libyaml-source-dir") returns nil and enable_config("bundled-libyaml", false) returns false so my machine falls through to configuring yaml from source here https://github.com/ruby/psych/blob/master/ext/psych/extconf.rb#L33-L55.

So when building psych inside psych it works fine, but when building psych inside Ruby it fails. Inside Ruby, with_config("libyaml-source-dir") returns true.

nobu · 2022-03-31T13:00:11Z

Doesn’t make extract-extlubs after configure make ext/psych/yaml-2.5?
Then ext/psych/extconf.rb should fall back to it.
Note that downloading in extconf.rb is for the separate repositories.

eileencodes · 2022-04-01T15:29:06Z

Doesn’t make extract-extlubs after configure make ext/psych/yaml-2.5?

I think you meant extlibs but I'm sure what's not working right.

When tool/extlibs.rb gets loaded there's a require 'digest' at the top, that's what's causing the download to not happen on my machine.

Note that downloading in extconf.rb is for the separate repositories

I'm not downloading it or running it manually, it's make that's running it.

XrXr · 2022-04-02T00:01:04Z

I get the same error on FreeBSD 13 when libyaml is not installed. make configure-ext fails, which make runs. Looks like it's because make configure-ext uses miniruby which doesn't have digest.

Running make extract-extlibs first let's me get past make configure-ext into make build-ext, but that fails due to libyaml having a Makefile BSD make dislikes:

--- ext/psych/all ---
--- libyaml.a ---
cd libyaml && make
make[3]: "/usr/home/ec2-user/ruby/ext/psych/libyaml/Makefile" line 884: Need an operator
make[3]: "/usr/home/ec2-user/ruby/ext/psych/libyaml/Makefile" line 888: Need an operator
make[3]: "/usr/home/ec2-user/ruby/ext/psych/libyaml/Makefile" line 890: Need an operator
make[3]: Fatal errors encountered -- cannot continue
make[3]: stopped in /usr/home/ec2-user/ruby/ext/psych/libyaml
*** [libyaml.a] Error code 1

nobu · 2022-04-03T04:50:07Z

@eileencodes

Doesn’t make extract-extlubs after configure make ext/psych/yaml-2.5?

I think you meant extlibs but I'm sure what's not working right.

Sorry, yes, it is.

When tool/extlibs.rb gets loaded there's a require 'digest' at the top, that's what's causing the download to not happen on my machine.

make extract-extlibs runs BASERUBY, which should have digest.

$ ruby -w tool/extlibs.rb ext
downloading for ext/fiddle/extlibs
.downloaded-cache/libffi-3.2.1.tar.gz already exists
downloading for ext/psych/extlibs
.downloaded-cache/yaml-0.2.5.tar.gz already exists

Note that downloading in extconf.rb is for the separate repositories

I'm not downloading it or running it manually, it's make that's running it.

I mean we are going to remove all extlibs files.
You will then need to install libffi and libyaml before building ruby.

nobu · 2022-04-03T04:57:14Z

@XrXr
Those lines seem coming from the original libyaml source, Makefile.in.
Could your report it there?

People trying to build CRuby by following the instructions in its [README] have been running into [errors] due to missing `libyaml` on their system. Let's try to present a better error message when it happens. [README]: https://github.com/ruby/ruby/tree/fb5aa31e2d20ea8e1425432672f4de4c8ca2c26b#how-to-compile-and-install [errors]: ruby#552

People trying to build CRuby by following the instructions in its [README] have been running into [errors] due to missing `libyaml` on their system. Let's try to present a better error message when it happens. [README]: https://github.com/ruby/ruby/tree/fb5aa31e2d20ea8e1425432672f4de4c8ca2c26b#how-to-compile-and-install [errors]: ruby/psych#552 ruby/psych@20a633028e

* Enhanced RDoc for String#index (ruby#5759) * ruby_gc_set_params: update malloc_limit when env is set During VM startup, rb_objspace_alloc sets malloc_limit (objspace->malloc_params.limit) before ruby_gc_set_params is called, thus nullifying the effect of RUBY_GC_MALLOC_LIMIT before the initial GC run. The call sequence is as follows: main.c::main() ruby_init ruby_setup Init_BareVM rb_objspace_alloc // malloc_limit = gc_params.malloc_limit_min; ruby_options ruby_process_options process_options ruby_gc_set_params // RUBY_GC_MALLOC_LIMIT => gc_params.malloc_limit_min With ruby_gc_set_params setting malloc_limit, RUBY_GC_MALLOC_LIMIT affects the process sooner. [ruby-core:107170] * [ruby/psych] Improve libyaml source downloading error messages People trying to build CRuby by following the instructions in its [README] have been running into [errors] due to missing `libyaml` on their system. Let's try to present a better error message when it happens. [README]: https://github.com/ruby/ruby/tree/fb5aa31e2d20ea8e1425432672f4de4c8ca2c26b#how-to-compile-and-install [errors]: ruby/psych#552 ruby/psych@20a633028e * Apply timescale configuration for tests of Regexp.timeout * Removed mswin patch for zlib-1.2.11 * Ignore yaml source * Unflag a splatted flagged hash if the method doesn't use ruby2_keywords For a method such as: def foo(*callee_args) end If this method is called with a flagged hash (created by a method flagged with ruby2_keywords), this previously passed the hash through without modification. With this change, it acts as if the last hash was passed as keywords, so a call to: foo(*caller_args) where the last element of caller_args is a flagged hash, will be treated as: foo(*caller_args[0...-1], **caller_args[-1]) As a result, inside foo, callee_args[-1] is an unflagged duplicate of caller_args[-1] (all other elements of callee_args match caller_args). Fixes [Bug #18625] * Use latest RSpec to get rspec-mocks ruby2_keywords fix * Add NEWS entry for Bug #18625 to help adding ruby2_keywords in the missing places * Give some tips on how to find the missing ruby2_keywords * Try to fix NoMethodError on slow environments ``` 1) Failure: TestParallel::TestParallel#test_hungup [/home/user/ruby/tool/test/testunit/test_parallel.rb:215]: Expected /^Retrying hung up testcases\.+$/ to match "Run options: \n" + " --seed=43403\n" + " --ruby\n" + " \"./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems\"\n" + " -j\n" + " t1\n" + " --worker-timeout=1\n" + "\n" + "# Running tests:\n" + "\n" + "/home/user/ruby/tool/lib/test/unit.rb:687:in `block in _run_parallel': undefined method `<' for nil:NilClass (NoMethodError)\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:538:in `block in quit_workers'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:537:in `reject!'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:537:in `quit_workers'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:687:in `_run_parallel'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:810:in `_run_suites'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:849:in `_run_suites'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1479:in `_run_anything'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1263:in `_run_anything'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1654:in `run_tests'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1641:in `block in _run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1640:in `each'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1640:in `_run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1682:in `run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1034:in `run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:882:in `run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:154:in `run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1761:in `run'\n" + "\tfrom /home/user/ruby/tool/lib/test/unit.rb:1765:in `run'\n" + "\tfrom /home/user/ruby/tool/test/testunit/tests_for_parallel/runner.rb:14:in `<main>'\n". ``` * Load fake.rb at `BTESTRUBY` So that `mkmf` checks work from `make run`, and also remove duplicate `$(MINIRUBYOPT)` which is used in `$(MINIRUBY)`. * Fix using anonymous block in method accepting explicit keywords Record block ID before vtable_pop, so the incorrect one doesn't override it. Fixes [Bug #18673] * Document MakeMakefile#append_cflags This method is at least 7 years old and is widely used in the wild. Since we need to support it, let's document it to make it discoverable. Add docs and move it out of the `# :stopdoc:` zone. * * 2022-04-06 [ci skip] * RubyVM.stat constant cache metrics (ruby#5766) Before the new constant cache behavior, caches were invalidated by a single global variable. You could inspect the value of this variable with RubyVM.stat(:global_constant_state). This was mostly useful to verify the behavior of the VM or to test constant loading like in Rails. With the new constant cache behavior, we introduced RubyVM.stat(:constant_cache) which returned a hash with symbol keys and integer values that represented the number of live constant caches associated with the given symbol. Additionally, we removed the old RubyVM.stat(:global_constant_state). This was proven to be not very useful, so it doesn't help you diagnose constant loading issues. So, instead we added the global constant state back into the RubyVM output. However, that number can be misleading as now when you invalidate something like `Foo::Bar::Baz` you're actually invalidating 3 different lists of inline caches. This commit attempts to get the best of both worlds. We remove RubyVM.stat(:global_constant_state) like we did originally, as it doesn't have the same semantic meaning and it could be confusing going forward. Instead we add RubyVM.stat(:constant_cache_invalidations) and RubyVM.stat(:constant_cache_misses). These two metrics should provide enough information to diagnose any constant loading issues, as well as provide a replacement for the old global constant state. * [rubygems/rubygems] Enable mfa on specific keys during gem signin rubygems/rubygems@e787f7f655 * [rubygems/rubygems] Correct mfa level name rubygems/rubygems@a002e351ae * [rubygems/rubygems] Make mfa the default rubygems/rubygems@0b636f6902 * [rubygems/rubygems] Fix lint rubygems/rubygems@a68bfde18e * [rubygems/rubygems] Make changes <2.6 compatible Multiple params to merge was not introduced until Ruby 2.6, so this merges the two additional params together first and then merges that with the request body rubygems/rubygems@870f7e9a1c * [rubygems/rubygems] Remove whitespace rubygems/rubygems@08c2d88137 * [rubygems/rubygems] Update endpoint rubygems/rubygems@a5a7b3ec96 * [rubygems/rubygems] Accomodate gem hosts without profile/me endpoint rubygems/rubygems@31b6dcf5d3 * [rubygems/rubygems] Use YAML rubygems/rubygems@6122e8cac5 * [rubygems/rubygems] Extract default_host method rubygems/rubygems@6e10e75574 * [rubygems/rubygems] Use `ask_yes_no` rubygems/rubygems@1d38e167fa * Fix a typo [ci skip] * Bundle RBS 2.3.2 (ruby#5762) * Update bundled gems list at 8197ae3 [ci skip] * [DOC] Enhanced RDoc for string slices (ruby#5769) Creates file doc/string/slices.rdoc that the string slicing methods can link to. * * 2022-04-07 [ci skip] Co-authored-by: Burdette Lamar <BurdetteLamar@Yahoo.com> Co-authored-by: Eric Wong <e@80x24.org> Co-authored-by: Yusuke Endoh <mame@ruby-lang.org> Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org> Co-authored-by: Jeremy Evans <code@jeremyevans.net> Co-authored-by: Benoit Daloze <eregontp@gmail.com> Co-authored-by: Kazuhiro NISHIYAMA <zn@mbf.nifty.com> Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org> Co-authored-by: git <svn-admin@ruby-lang.org> Co-authored-by: Kevin Newton <kddnewton@gmail.com> Co-authored-by: Ashley Ellis Pierce <anellis12@gmail.com> Co-authored-by: Soutaro Matsumoto <matsumoto@soutaro.com>

c4lliope · 2022-04-14T15:13:49Z

I see this error on Fedora 35, and during make extract-extlibs I had an error concerning the patch library.
sudo dnf install patch libyaml libffi seems to help.

I tried to build Ruby on a system without libyaml today and realized that my attempt from ruby#557 doesn't fix the error in ruby#552. I still got the same LoadError from digest. Since `LoadError` is not a StandardError, a plain `rescue` doesn't catch it. Catch `LoadError` explicitly instead and reduce the scope of the `begin` block. I tested this change in a Ruby build on macOS without libyaml installed and confirmed that `make` continues with a warning instead of aborting: *** Following extensions are not compiled: psych: Could not be configured. It will not be installed. ...

I tried to build Ruby on a system without libyaml today and realized that my attempt from <#557> doesn't fix the error in <#552>. I still got the same `LoadError` from `digest` which stopped the build. Since `LoadError` is not a `StandardError`, a plain `rescue` doesn't catch it. Catch `LoadError` explicitly instead and reduce the scope of the `begin` block. I tested this change in a Ruby build on macOS without libyaml installed and confirmed that `make` continues with a warning instead of aborting: *** Following extensions are not compiled: psych: Could not be configured. It will not be installed. ... This should address <https://bugs.ruby-lang.org/issues/18790>.

I tried to build Ruby on a system without libyaml today and realized that my attempt from <ruby/psych#557> doesn't fix the error in <ruby/psych#552>. I still got the same `LoadError` from `digest` which stopped the build. Since `LoadError` is not a `StandardError`, a plain `rescue` doesn't catch it. Catch `LoadError` explicitly instead and reduce the scope of the `begin` block. I tested this change in a Ruby build on macOS without libyaml installed and confirmed that `make` continues with a warning instead of aborting: *** Following extensions are not compiled: psych: Could not be configured. It will not be installed. ... This should address <https://bugs.ruby-lang.org/issues/18790>. ruby/psych@251289ba83

hsbt · 2022-10-20T05:59:23Z

We completely disabled auto-download feature on ruby/psych and ruby/ruby. I think we will not see this issue in the future.

XrXr mentioned this issue Apr 4, 2022

Improve libyaml source downloading error messages #557

Merged

XrXr mentioned this issue May 26, 2022

Fix libyaml download failure rescue under miniruby #562

Merged

hsbt closed this as not planned Won't fix, can't repro, duplicate, stale Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ruby can't compile with psych when libyaml isn't installed #552

Ruby can't compile with psych when libyaml isn't installed #552

eileencodes commented Mar 30, 2022

eileencodes commented Mar 31, 2022

nobu commented Mar 31, 2022

eileencodes commented Apr 1, 2022

XrXr commented Apr 2, 2022

nobu commented Apr 3, 2022

nobu commented Apr 3, 2022

c4lliope commented Apr 14, 2022

hsbt commented Oct 20, 2022

Ruby can't compile with psych when libyaml isn't installed #552

Ruby can't compile with psych when libyaml isn't installed #552

Comments

eileencodes commented Mar 30, 2022

eileencodes commented Mar 31, 2022

nobu commented Mar 31, 2022

eileencodes commented Apr 1, 2022

XrXr commented Apr 2, 2022

nobu commented Apr 3, 2022

nobu commented Apr 3, 2022

c4lliope commented Apr 14, 2022

hsbt commented Oct 20, 2022