Provide 3.2 build support #212

gjtorikian · 2023-01-06T22:18:07Z

Should close #211

ianks · 2023-01-08T05:33:14Z

ianks · 2023-01-08T07:09:54Z

Looks like the build passed, but the Cargo.lock at the root of the gem isn't included in spec.files. I think adding it to spec.files should resolve the issue.

kivikakk · 2023-01-08T07:11:21Z

Ta! Just giving that a go now.

See rubygems/rubygems#6151.

kivikakk · 2023-01-08T07:57:36Z

Nice! I'll leave this for @gjtorikian to consider, since there might be aspects of this he has in mind that I'm not aware of, but hopefully this gets us there faster.

kivikakk · 2023-01-08T09:48:39Z

Hm! I've noticed that trying to build this branch on Ruby 3.2.0 with something like this:

source "https://rubygems.org"

gem "commonmarker", github: "gjtorikian/commonmarker", branch: "support-32"

succeeds on (e.g.) Linux x86_64, but fails on Darwin arm64:

Fetching https://github.com/gjtorikian/commonmarker.git
Resolving dependencies...Fetching gem metadata from https://rubygems.org/.
...
Using bundler 2.3.25
Using rb_sys 0.9.54
Using commonmarker 1.0.0.pre5 (was 1.0.0.pre4) from https://github.com/gjtorikian/commonmarker.git (at support-32@a18b3e1)
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784/ext/commonmarker
cargo rustc --target-dir /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784 --manifest-path
/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784/ext/commonmarker/Cargo.toml --lib --profile release --locked -- -C linker\=clang
-L native\=/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib -L native\=/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib -L native\=/opt/homebrew/opt/gmp/lib -C
link_arg\=-Wl,-multiply_defined,suppress -l pthread -C debuginfo\=1
   Compiling commonmarker v1.0.0 (/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784/ext/commonmarker)
error: linking with `clang` failed: exit status: 1
  |
= note: "clang" "-Wl,-exported_symbols_list,/var/folders/l1/j6x78td56fq_ctkt8qx5hjdw0000gn/T/rustcZEUPiH/list" "-arch" "arm64"
"/var/folders/l1/j6x78td56fq_ctkt8qx5hjdw0000gn/T/rustcZEUPiH/symbols.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.0.rcgu.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.1.rcgu.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.10.rcgu.o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/commonmarker.commonmarker.21ac97c
f-cgu.11.rcgu.o"
[…]
[…]
[…]
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/liballoc-945b7af0c9f21aac.rlib"
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-0251f0b5857602a2.rlib"
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-9382e8c089006a25.rlib"
"/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-6b5d600bff28faab.rlib" "-lSystem" "-lresolv" "-lc" "-lm"
"-liconv" "-L" "/Users/kivikakk/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "-o"
"/Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/release/deps/libcommonmarker.dylib"
"-Wl,-dead_strip" "-dynamiclib" "-Wl,-dylib" "-nodefaultlibs" "-Wl,-multiply_defined,suppress"
  = note: Undefined symbols for architecture arm64:
            "_rb_any_to_s", referenced from:
                magnus::value::Value::to_s_infallible::hc9ca6d51fb4404b8 in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.2.rcgu.o)
                magnus::value::Value::inspect::hd02724b944f40252 in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.2.rcgu.o)
            "_rb_ary_to_ary", referenced from:
                magnus::error::protect::call::h928de4cd683d6f7b (.llvm.14460302740317915909) in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.0.rcgu.o)
            "_rb_ascii8bit_encindex", referenced from:
[…]
[…]
[…]
                magnus::error::protect::call::h8ee51cf65c05a7c5 (.llvm.14460302740317915909) in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.0.rcgu.o)
                magnus::error::raise::hb1f04e67eecc0f1e in libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.0.rcgu.o)
magnus::r_string::_$LT$impl$u20$core..convert..From$LT$alloc..string..String$GT$$u20$for$u20$magnus..value..Value$GT$::from::h2a2a140ad0df8154 in
libmagnus-c1cefe60a099264b.rlib(magnus-c1cefe60a099264b.magnus.99dacd53-cgu.2.rcgu.o)
          ld: symbol(s) not found for architecture arm64
          clang-15: error: linker command failed with exit code 1 (use -v to see invocation)


error: could not compile `commonmarker` due to previous error

cargo failed, exit code 101

Gem files will remain installed in /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/commonmarker-a18b3e1ed784 for inspection.
Results logged to /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/bundler/gems/extensions/arm64-darwin-22/3.2.0/commonmarker-a18b3e1ed784/gem_make.out

  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:102:in `run'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/cargo_builder.rb:32:in `call'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/cargo_builder.rb:32:in `build_crate'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/cargo_builder.rb:22:in `build'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:171:in `build_extension'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:205:in `block in build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:202:in `each'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:202:in `build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:843:in `build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/source/path/installer.rb:34:in `post_install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/source/path.rb:244:in `generate_bin'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/source/git.rb:194:in `install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/gem_installer.rb:54:in `install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/parallel_installer.rb:186:in `do_install'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:62:in `apply_func'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:57:in `block in process_queue'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:54:in `loop'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:54:in `process_queue'
  /Users/kivikakk/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.3.25/lib/bundler/worker.rb:91:in `block (2 levels) in create_threads'

An error occurred while installing commonmarker (1.0.0.pre5), and Bundler cannot continue.

In Gemfile:
  commonmarker

Looks like possibly magnus's preview 3.2.0 support.

kivikakk · 2023-01-08T10:14:37Z

Looks like it might be related to oxidize-rb/rb-sys#88 maybe not triggering as it should? If I repeat the cargo rustc invocation from the failing build log but append -C link_arg=-Wl,-undefined,dynamic_lookup, it succeeds.

… Actually, It looks like this is to do with RubyGems' own Rust building capabilities running this command for us, which, on further investigation, does actually add this flag these days.

Why does my own Ruby 3.2.0's RubyGems not have this, which is part of the v3_2_0 tag? I have no idea whatsoever — I lack some knowledge about how RubyGems is packaged these days, I guess. gem update --system was all I needed.

ianks · 2023-01-08T15:47:25Z

Looks like it might be related to oxidize-rb/rb-sys#88 maybe not triggering as it should? If I repeat the cargo rustc invocation from the failing build log but append -C link_arg=-Wl,-undefined,dynamic_lookup, it succeeds.

… Actually, It looks like this is to do with RubyGems' own Rust building capabilities running this command for us, which, on further investigation, does actually add this flag these days.

Why does my own Ruby 3.2.0's RubyGems not have this, which is part of the v3_2_0 tag? I have no idea whatsoever — I lack some knowledge about how RubyGems is packaged these days, I guess. gem update --system was all I needed.

Yes it should work now. On another note, this thread is worth reading regarding builtin ruby gems cargo builder. Ling story short, using rb_sys as a dependency will yield more predictable results until some issues with Rubygems are addressed with external builders.

gjtorikian · 2023-01-08T23:02:52Z

Appreciate you both <3

gjtorikian · 2023-01-08T23:07:57Z

Nice! I'll leave this for @gjtorikian to consider, since there might be aspects of this he has in mind that I'm not aware of, but hopefully this gets us there faster.

Thanks for the help y'all, this looks great.

This reverts commit f48eaef.

gjtorikian added 2 commits January 6, 2023 17:17

Provide 3.2 build support

8b7a7c8

test 3.2 build

654e922

This was referenced Jan 6, 2023

Cargo.lock prevents Ruby 3.2.0 from installing commonmarker v1.0.0.pre4 #211

Closed

Support 3.2 by default oxidize-rb/cross-gem-action#129

Merged

gemspec: allow 3.2+

f48eaef

kivikakk force-pushed the support-32 branch from 3680b62 to f48eaef Compare January 8, 2023 05:54

generic-build-and-install: checking if it's this simple

4fc98d7

kivikakk added 2 commits January 8, 2023 18:11

gemspec: try adding Cargo.lock to spec.files

0c22b33

See rubygems/rubygems#6151.

rubocop: appease

0384fda

workflows: push more towards using 3.2 and building for both

a18b3e1

Revert "gemspec: allow 3.2+"

8af6c97

This reverts commit f48eaef.

gjtorikian merged commit ab585be into main Jan 8, 2023

gjtorikian deleted the support-32 branch January 8, 2023 23:09

kivikakk mentioned this pull request Jan 9, 2023

always use rb_sys (don't use Ruby's emerging cargo tooling where available) #213

Merged

gjtorikian mentioned this pull request Jan 9, 2023

Support ruby32 gjtorikian/selma#9

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide 3.2 build support #212

Provide 3.2 build support #212

gjtorikian commented Jan 6, 2023

ianks commented Jan 8, 2023

ianks commented Jan 8, 2023 •

edited

kivikakk commented Jan 8, 2023

kivikakk commented Jan 8, 2023

kivikakk commented Jan 8, 2023

kivikakk commented Jan 8, 2023

ianks commented Jan 8, 2023

gjtorikian commented Jan 8, 2023

gjtorikian commented Jan 8, 2023

Provide 3.2 build support #212

Provide 3.2 build support #212

Conversation

gjtorikian commented Jan 6, 2023

ianks commented Jan 8, 2023

ianks commented Jan 8, 2023 • edited

kivikakk commented Jan 8, 2023

kivikakk commented Jan 8, 2023

kivikakk commented Jan 8, 2023

kivikakk commented Jan 8, 2023

ianks commented Jan 8, 2023

gjtorikian commented Jan 8, 2023

gjtorikian commented Jan 8, 2023

ianks commented Jan 8, 2023 •

edited