Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update ruby 3.3 support to use 3.3.0 #112

Closed
wants to merge 1 commit into from

Conversation

stanhu
Copy link

@stanhu stanhu commented Dec 25, 2023

@stanhu
Copy link
Author

stanhu commented Dec 25, 2023

Wonder what happened in https://github.com/rake-compiler/rake-compiler-dock/actions/runs/7321958437/job/19942923219. This looks like some error building Ruby 3.1:

#23 232.5 /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/file_utils.rb:67:in `block in create_shell_runner': Command failed with status (2): [make V=1...] (RuntimeError)
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/file_utils.rb:57:in `sh'
#23 232.5 	from /usr/local/rvm/gems/ruby-3.1.3/gems/rake-compiler-1.2.5/tasks/bin/cross-ruby.rake:144:in `block (3 levels) in <top (required)>'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:253:in `block (2 levels) in invoke_prerequisites_concurrently'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/promise.rb:64:in `chore'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/promise.rb:46:in `work'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/thread_pool.rb:104:in `process_queue_item'
#23 232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/thread_pool.rb:124:in `block (2 levels) in start_thread'
#23 ERROR: process "/bin/sh -c bash -c \"       rvm use 3.1.3 &&       export CPPFLAGS='' &&       export CFLAGS='-O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong ' &&       export LDFLAGS='-pipe ' &&              export CC=aarch64-apple-darwin-clang &&       export MAKE='make V=1' &&       rake-compiler cross-ruby VERSION=$XRUBIES HOST=aarch64-apple-darwin &&       rm -rf ~/.rake-compiler/builds ~/.rake-compiler/sources &&       find /usr/local/rvm -type d -print0 | sudo xargs -0 chmod g+sw     \"" did not complete successfully: exit code: 1
------
 > [18/31] RUN bash -c "       rvm use 3.1.3 &&       export CPPFLAGS='' &&       export CFLAGS='-O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong ' &&       export LDFLAGS='-pipe ' &&              export CC=aarch64-apple-darwin-clang &&       export MAKE='make V=1' &&       rake-compiler cross-ruby VERSION=3.3.0:3.2.0:3.1.0:3.0.0:2.7.0 HOST=aarch64-apple-darwin &&       rm -rf ~/.rake-compiler/builds ~/.rake-compiler/sources &&       find /usr/local/rvm -type d -print0 | sudo xargs -0 chmod g+sw     ":
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:253:in `block (2 levels) in invoke_prerequisites_concurrently'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/promise.rb:64:in `chore'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/promise.rb:46:in `work'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/thread_pool.rb:104:in `process_queue_item'
232.5 	from /usr/local/rvm/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/thread_pool.rb:124:in `block (2 levels) in start_thread'
------
WARNING: local cache import at tmp/build-cache not found due to err: could not read tmp/build-cache/index.json: open tmp/build-cache/index.json: no such file or directory
common-bd2fd2960dd6cc272913b89ee50946f995aa78fe:107
--------------------
 106 |     # Build xruby versions, then cleanup all build artifacts
 107 | >>> RUN bash -c " \
 108 | >>>       rvm use 3.1.3 && \
 109 | >>>       export CPPFLAGS='' && \
 110 | >>>       export CFLAGS='-O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong ' && \
 111 | >>>       export LDFLAGS='-pipe ' && \
 112 | >>>        \
 113 | >>>       export CC=aarch64-apple-darwin-clang && \
 114 | >>>       export MAKE='make V=1' && \
 115 | >>>       rake-compiler cross-ruby VERSION=$XRUBIES HOST=aarch64-apple-darwin && \
 116 | >>>       rm -rf ~/.rake-compiler/builds ~/.rake-compiler/sources && \
 117 | >>>       find /usr/local/rvm -type d -print0 | sudo xargs -0 chmod g+sw \
 118 | >>>     "
 119 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c bash -c \"       rvm use 3.1.3 &&       export CPPFLAGS='' &&       export CFLAGS='-O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong ' &&       export LDFLAGS='-pipe ' &&              export CC=aarch64-apple-darwin-clang &&       export MAKE='make V=1' &&       rake-compiler cross-ruby VERSION=$XRUBIES HOST=aarch64-apple-darwin &&       rm -rf ~/.rake-compiler/builds ~/.rake-compiler/sources &&       find /usr/local/rvm -type d -print0 | sudo xargs -0 chmod g+sw     \"" did not complete successfully: exit code: 1
rake aborted!

@flavorjones
Copy link
Collaborator

flavorjones commented Dec 25, 2023

@stanhu Thanks. I'll look into the build.

In the meantime: everyone should be able to use 1.4.0.rc2 to build releases that are compatible with ruby-3.0.0 final, since there were no ABI changes announced by naruse.

@flavorjones
Copy link
Collaborator

flavorjones commented Dec 25, 2023

I'm actually going to hold off on merging this -- even if it goes green -- until we get the rc2 build green on 3.0.0 CI image, see #113

@stanhu
Copy link
Author

stanhu commented Dec 26, 2023

Oh, it looks like a legit failure building Ruby 3.3.0 in arm64-darwin. I can reproduce it locally:

50.28 aarch64-apple-darwin-clang  -O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong  -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wmisleading-indentation -Wundef  -pipe -march=arm64 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -DRUBY_EXPORT -fPIE -I. -I.ext/include/arm64-darwin -I/usr/local/rake-compiler/sources/ruby-3.3.0/include -I/usr/local/rake-compiler/sources/ruby-3.3.0 -I/usr/local/rake-compiler/sources/ruby-3.3.0/prism -I/usr/local/rake-compiler/sources/ruby-3.3.0/enc/unicode/15.0.0   -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -o main.o -c /usr/local/rake-compiler/sources/ruby-3.3.0/main.c
50.29 clang: error: the clang compiler does not support '-march=arm64'
50.30 make: *** [uncommon.mk:1291: main.o] Error 1
$ docker run -it ghcr.io/rake-compiler/rake-compiler-dock-image:1.4.0.rc2-mri-arm64-darwin bash
root@f283b5c36868:/# touch test.c
root@f283b5c36868:/# aarch64-apple-darwin-clang -march=arm64 -o test test.c
clang: error: the clang compiler does not support '-march=arm64'
root@f283b5c36868:/# aarch64-apple-darwin-clang --print-supported-cpus
clang version 10.0.0-4ubuntu1
Target: aarch64-apple-darwin20.2
Thread model: posix
InstalledDir: /usr/bin
Available CPUs for this target:

        apple-a10
        apple-a11
        apple-a12
        apple-a13
        apple-a7
        apple-a8
        apple-a9
        apple-latest
        apple-s4
        apple-s5
        cortex-a35
        cortex-a53
        cortex-a55
        cortex-a57
        cortex-a65
        cortex-a65ae
        cortex-a72
        cortex-a73
        cortex-a75
        cortex-a76
        cortex-a76ae
        cyclone
        exynos-m3
        exynos-m4
        exynos-m5
        falkor
        generic
        kryo
        neoverse-e1
        neoverse-n1
        saphira
        thunderx
        thunderx2t99
        thunderxt81
        thunderxt83
        thunderxt88
        tsv110

Use -mcpu or -mtune to specify the target's processor.
For example, clang --target=aarch64-unknown-linux-gui -mcpu=cortex-a35
root@f283b5c36868:/# aarch64-apple-darwin-clang --version
clang version 10.0.0-4ubuntu1
Target: aarch64-apple-darwin20.2
Thread model: posix
InstalledDir: /usr/bin

I'm not sure yet how the older Ruby versions worked before, but I do see that ruby/ruby@2c2c6bc was added between v3_3_0_preview1..v3_3_0.

UPDATE: It appears that older versions omitted the -march=<arch> parameter entirely.

@stanhu
Copy link
Author

stanhu commented Dec 26, 2023

Even on my native macOS with clang 15.0.0, arm64 isn't a valid option, so I think this might be a regression in upstream Ruby:

% clang --print-supported-cpus
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Available CPUs for this target:

	a64fx
	ampere1
	apple-a10
	apple-a11
	apple-a12
	apple-a13
	apple-a14
	apple-a15
	apple-a16
	apple-a7
	apple-a8
	apple-a9
	apple-latest
	apple-m1
	apple-m2
	apple-s4
	apple-s5
	carmel
	cortex-a34
	cortex-a35
	cortex-a510
	cortex-a53
	cortex-a55
	cortex-a57
	cortex-a65
	cortex-a65ae
	cortex-a710
	cortex-a72
	cortex-a73
	cortex-a75
	cortex-a76
	cortex-a76ae
	cortex-a77
	cortex-a78
	cortex-a78c
	cortex-r82
	cortex-x1
	cortex-x1c
	cortex-x2
	cyclone
	exynos-m3
	exynos-m4
	exynos-m5
	falkor
	generic
	kryo
	neoverse-512tvb
	neoverse-e1
	neoverse-n1
	neoverse-n2
	neoverse-v1
	neoverse-v2
	saphira
	thunderx
	thunderx2t99
	thunderx3t110
	thunderxt81
	thunderxt83
	thunderxt88
	tsv110

Use -mcpu or -mtune to specify the target's processor.
For example, clang --target=aarch64-unknown-linux-gui -mcpu=cortex-a35

@stanhu
Copy link
Author

stanhu commented Dec 26, 2023

Yeah, I confirmed that Ruby 3.3.0 builds if i revert ruby/ruby#9250.

It seems that target_cpu is set to arm64, but that AS_CASE statement ends up setting -march=arm64, which is incorrect.

@stanhu stanhu mentioned this pull request Dec 26, 2023
@stanhu
Copy link
Author

stanhu commented Dec 26, 2023

@flavorjones
Copy link
Collaborator

Thank you, @stanhu!

@flavorjones
Copy link
Collaborator

flavorjones commented Dec 26, 2023

@stanhu When the following PRs all go green (tests all should pass once CI images are available), I think I'm going to cut a 1.4.0 final release of rake-compiler-dock using Ruby 3.3.0-rc1:

The alternatives I can think of are not as pleasant:

  1. wait for a bugfix release of Ruby
  2. patch ruby 3.3.0 in rake-compiler-dock before building it
  3. patch rake-compiler to pass in appropriate target-cpu info

Are there other options? WDYT?

@stanhu
Copy link
Author

stanhu commented Dec 27, 2023

I think I'm going to cut a 1.4.0 final release of rake-compiler-dock using Ruby 3.3.0-rc2.

Does that mean that gems will still have to build against 3.3.0-rc2, and what does this mean for the required_ruby_version of precompiled gems?

ruby/ruby#9364 looks like it fixes the arm64-darwin build issue for me.

@mudge
Copy link

mudge commented Dec 27, 2023

@stanhu based on https://rubygems.org/gems/re2/versions/2.6.0.rc1-x86_64-linux which was built with rake-compiler-dock 1.4.0.rc2, I believe the required_ruby_version will be >= 2.6, < 3.4.DEV (assuming CROSS_RUBY_VERSIONS of 2.6.0 up to 3.3.0). I have a green build testing against 3.3 final despite compiling with 3.3.0-rc1 so I'm also tempted to put that out as a new version.

@flavorjones
Copy link
Collaborator

@stanhu yes, what @mudge said is correct. using 3.3.0-rc1 to precompile will work with ruby 3.3.0 at runtime (once all the test suites go green we'll know that to be true).

@flavorjones
Copy link
Collaborator

flavorjones commented Dec 27, 2023

OK, for whatever it's worth, rake-compiler-dock, nokogiri, and sqlite3-ruby are all passing on ruby 3.0.0 when precompiling with rcd 1.4.0.rc2 (using ruby 3.0.0-rc1)

(with the exception of what seems to be an instance of https://bugs.ruby-lang.org/issues/20085 in https://github.com/sparklemotion/nokogiri/actions/runs/7325933933/job/19985580656#step:4:379)

So I'm going to cut a 1.4.0 final, and we can circle back on this PR once there's a ruby patch release with ruby/ruby#9364

@flavorjones
Copy link
Collaborator

Going to close this. Ruby 3.3.1 is (reportedly) close to being released, and we can create a new PR at that point since the change is small.

@flavorjones flavorjones closed this Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants