binary C extension not built? #148

noraj · 2023-08-17T21:31:47Z

System Information

OS: ArchLinux
Ruby: 3.0.6p216
Version: 0.7.56

Description

The binary lib/iodine/iodine seems to be missing.

Expected behavior

No error loading the gem.

Actual behavior

➜ irb   
irb(main):001:0> require 'iodine'
/home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.56/lib/iodine.rb:6:in `require_relative': cannot load such file -- /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.56/lib/iodine/iodine (LoadError)
        from /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.56/lib/iodine.rb:6:in `<top (required)>'
        from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
        from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
        from (irb):1:in `<main>'                     
        from /usr/lib/ruby/gems/3.0.0/gems/irb-1.4.2/exe/irb:11:in `<top (required)>'
        from /usr/bin/irb:25:in `load'               
        from /usr/bin/irb:25:in `<main>'             
<internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- iodine (LoadError)
        from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from (irb):1:in `<main>'                     
        from /usr/lib/ruby/gems/3.0.0/gems/irb-1.4.2/exe/irb:11:in `<top (required)>'
        from /usr/bin/irb:25:in `load'               
        from /usr/bin/irb:25:in `<main>'             

➜ sed -n 6p /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.56/lib/iodine.rb
require_relative './iodine/iodine' # loading a binary C extension

➜ ls -1 /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.56/lib/iodine
connection.rb
json.rb
mustache.rb
pubsub.rb
rack_utils.rb
tls.rb
version.rb

Notes

Install:

➜ gem install iodine --no-document
Fetching iodine-0.7.56.gem
Building native extensions. This could take a while...
Thank you for installing Iodine 0.7.56.
Remember: if iodine supports your business, it's only fair to give value back (code contributions / donations).
Successfully installed iodine-0.7.56
1 gem installed

I see no error while install, is the C building failing silently?

The text was updated successfully, but these errors were encountered:

boazsegev · 2023-08-19T14:10:38Z

I see no error while install, is the C building failing silently?

Most probably, as I cannot see any reason for the error. The CI on Ruby 3.0 fails with a bundler error stating:

ERROR:  Error installing /home/runner/work/iodine/iodine/pkg/iodine-0.7.56.gem:
	ERROR: Failed to build gem native extension.

    No such file or directory @ dir_s_mkdir - /home/runner/work/iodine/iodine/vendor/bundle/ruby/3.0.0/gems/iodine-0.7.56/ext/iodine/.gem.20230707-1798-pdmzvu

I am not sure why your version fails silently, but there's a bundler issue somewhere that I haven't looked into working around quite yet.

Once I upgraded to Ruby 3.1.3p185, my build issue was resolved. I don't know if that will solve your issue.

Sorry about that.

noraj · 2023-08-21T09:34:46Z

ArchLinux version of ruby has been quite outdated for a long time https://archlinux.org/packages/extra/x86_64/ruby/, they don't seem to care much about it.

If I want to use a more recent version of Ruby I usually use asdf-vm on my dev machines. But on my desktop I wanted to avoid installing dev tools for virtual environments and use only the system ruby.

Based on what you told me, installing asdf-ruby + ruby 3.2+ will probably workaround this compilation issue that appears only on 3.0-3.1.3.

noraj · 2023-09-03T15:53:22Z

gem install iodine --no-document --backtrace -V 9 | tee /tmp/iodine-install.log on ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]

iodine-install.log ❌

gem install iodine --no-document --backtrace -V 9 | tee /tmp/iodine-install-3.2.log on ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

iodine-install-3.2.log ✅

janbiedermann · 2023-09-03T19:18:11Z

@noraj in both logs it says "successfully installed", whats the issue?

janbiedermann · 2023-09-03T19:33:07Z

In addition, in both logs it links iodine.so correctly and installs it without error.

Now, if iodine.so doesn't get loaded, that may be possible because of a name clash in the load path, where it finds iodine.rb first and thus doesn't load iodine.so. Not sure if that is the case here, but that happened to me once, thats why (@boazsegev) i renamed iodine.so to iodine_ext.so in isomorfeus-iodine.

@noraj you can verify that by patching the gem in its path, possibly copying the built extension in its place to iodine_ext.so and tinkering a bit in irb to see if it gets loaded.

noraj · 2023-09-03T19:34:41Z

@noraj in both logs it says "successfully installed", whats the issue?

The issue is what is explained in the original message (#148 (comment)), the gem is installed successfully but the iodine C binary is not built.

You can see both log file have very different size.

➜ lsd -lh iodine-install-3.2.log iodine-install.log
.rw-r--r-- noraj noraj 194 KB Sun Sep  3 18:17:48 2023  iodine-install-3.2.log
.rw-r--r-- noraj noraj  29 KB Sun Sep  3 17:52:08 2023  iodine-install.log

➜ wc iodine-install-3.2.log iodine-install.log     
  3749  20837 198640 iodine-install-3.2.log
   483   2146  29918 iodine-install.log

If you diff both, you'll see there are tons of stuff being build for 3.2 that is not for 3.0.

➜ diff iodine-install.log iodine-install-3.2.log | bat

janbiedermann · 2023-09-03T19:39:09Z

@noraj i see that, if i inspect the files, looking fo "compiling " i see 32 hits in both of them, i see the *.c *.h files are copied correctly.
The difference in log file size is either because of different compiler versions being used or different build options, one printing more warnings, one printing less warnings. I can see no difference in compiled files and i can find no errors. Its built correctly in both cases.

janbiedermann · 2023-09-03T19:40:01Z

So the issue seems to be, that it cannot find the built extension, see above.

janbiedermann · 2023-09-03T19:42:50Z

And sure, you cannot use the gem built for 3.0 with 3.2 or the other way around, if you switch rubies, are the gem paths updated accordingly? Otherwise load will fail.

noraj · 2023-09-03T19:46:34Z

So it's something from the ruby environment.

3.0 being archlinux system ruby with gem installed as user.
3.2 being a asdf-vm shim env.

janbiedermann · 2023-09-03T19:52:34Z

Then check $LOAD_PATH, see if /home/noraj/.local/share/gem/ruby/3.0.0/gems/
is in there on archlinux

janbiedermann · 2023-09-03T19:56:49Z

Oh yes, it tries to load from it, so i'm back on this possibly being a name clash, as described above (@boazsegev)

boazsegev · 2023-09-03T23:39:21Z

Hi @janbiedermann and @noraj ,

Thank you both for hunkering down on this one. I wasn't sure what the root cause was until @janbiedermann brought it up.

I released a patch that names the library file to iodine_ext, avoiding the possible filename conflict.

This patch doesn't solve the CI tests failing on Ruby 3+, but they might solve this issue.

Thanks again and please keep me posted so I know if this help.

Cheers,
Bo.

noraj · 2023-09-04T09:30:07Z

Can someone try with a suer install?

gem install --user-install iodine or this in /etc/gemrc

# Read about the gemrc format at http://guides.rubygems.org/command-reference/#gem-environment

# --user-install is used to install to $HOME/.gem/ by default since we want to separate
#                system (package manager) installed gems and gem installed gems
gem: --user-install

In your .bashrc|.zshrc:

export GEM_HOME="$(gem env user_gemhome)"
export PATH="$PATH:$GEM_HOME/bin"

Maybe it's a path issue when you use --user-install

janbiedermann · 2023-09-04T09:37:48Z

@noraj but it tries to load from the right path
it says "no such file or directory"
please do
ls -l /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine
ls -l /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/

noraj · 2023-09-04T09:43:05Z

~ 
➜ ls -l /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/
ls: cannot access '/home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/': No such file or directory

~ 
➜  ls -l /home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine
ls: cannot access '/home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine': No such file or directory

~ 
➜ ls -l /home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine/     
total 32
-rw-r--r-- 1 noraj noraj 2226 sept.  4 11:40 connection.rb
-rw-r--r-- 1 noraj noraj 2012 sept.  4 11:40 json.rb
-rw-r--r-- 1 noraj noraj 5309 sept.  4 11:40 mustache.rb
-rw-r--r-- 1 noraj noraj 3585 sept.  4 11:40 pubsub.rb
-rw-r--r-- 1 noraj noraj 2234 sept.  4 11:40 rack_utils.rb
-rw-r--r-- 1 noraj noraj  670 sept.  4 11:40 tls.rb
-rw-r--r-- 1 noraj noraj   46 sept.  4 11:40 version.rb

➜ /home/noraj/.gem/ruby/3.0.0/bin/iodine
/home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine.rb:6:in `require_relative': cannot load such file -- /home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine/iodine_ext (LoadError)
        from /home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine.rb:6:in `<top (required)>'
        from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/exe/iodine:4:in `<top (required)>'
        from /home/noraj/.gem/ruby/3.0.0/bin/iodine:25:in `load'
        from /home/noraj/.gem/ruby/3.0.0/bin/iodine:25:in `<main>'

janbiedermann · 2023-09-04T09:51:44Z

Hm, it installs from
/home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.56/ext/iodine
it loads from:
/home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine

and at same time it concludes:
/home/noraj/.local/share/gem/ruby/3.0.0/gems/iodine-0.7.57/lib/ : No such file or directory?

I can only conclude, that your system ruby is broken or your filesystem or there is a symlink somwhere thats broken or something like that.

My recommendation:

uninstall the system ruby
install rvm from rvm.io
use rvm to manage rubies on your system

noraj · 2023-09-04T10:03:46Z

It's maybe confusing but /home/noraj/.local/share/gem/ruby/3.0.0/ was on another machine (original post). On my previous message the user gem home is /home/noraj/.gem/ruby/3.0.0/. and you can see iodine is trying to load from /home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine/iodine_ext so the right path, the issue is there is no iodine_ext when I check ls -l /home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/lib/iodine/ . The original message was the same issue on another machine where the user gem home was /home/noraj/.local/share/gem/ruby/3.0.0. I don't think my system ruby is broken, I have user other native extension gem. And yeah I already worked around it by using asdf-vm (rvm is dangerous) and installing ruby 3.2. I'll try installing a ruby 3.0 on asdf to see if I have the same issue.

noraj · 2023-09-04T10:07:05Z

Also the C extension is built:

➜ find /home/noraj/.gem/ruby/3.0.0 -iname 'iodine_ext*'
/home/noraj/.gem/ruby/3.0.0/extensions/x86_64-linux/3.0.0/iodine-0.7.57/iodine/iodine_ext.so
/home/noraj/.gem/ruby/3.0.0/gems/iodine-0.7.57/ext/iodine/iodine_ext.so

The require tries gems/iodine-0.7.57/lib/iodine/iodine_ext instead of gems/iodine-0.7.57/ext/iodine/iodine_ext

noraj · 2023-09-04T10:11:12Z

On asdf ruby 3.2

➜ find /home/noraj/.asdf/installs/ruby/3.2.0/ -iname 'iodine_ext*'
/home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/extensions/x86_64-linux/3.2.0/iodine-0.7.57/iodine/iodine_ext.so
/home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/iodine-0.7.57/lib/iodine/iodine_ext.so

So in asdf ruby 3.2 tt's put in lib while in system ruby with user install 3.0 it's put in ext. Now I'll try asdf ruby 3.0 to know if the issue is caused by the user install or ruby 3.0 vs 3.2 or something else (weird default at compile time for AL ruby).

janbiedermann · 2023-09-04T10:12:02Z

install should copy the *.so as final step from ext to lib

noraj · 2023-09-04T10:13:17Z

Interestingly with asdf ruby 3.1.0 there is the extension on both lib and ext:

➜ find /home/noraj/.asdf/installs/ruby/3.1.0/ -iname 'iodine_ext*'
/home/noraj/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/iodine-0.7.57/iodine/iodine_ext.so
/home/noraj/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/iodine-0.7.57/ext/iodine/iodine_ext.so
/home/noraj/.asdf/installs/ruby/3.1.0/lib/ruby/gems/3.1.0/gems/iodine-0.7.57/lib/iodine/iodine_ext.so

janbiedermann · 2023-09-04T10:21:17Z

Yes, thats correct, thats how it should be

janbiedermann · 2023-09-04T10:23:40Z

(@boazsegev glad you renamed the ext, although thats not the problem here, i remember, that i had to make that change, when i build iodine withing the ruby tree for debugging)

noraj · 2023-09-04T10:33:02Z

Ruby source	Ruby version	ext build	work	install as
ASDF	3.2.0	`lib`	✅	system
ASDF	3.1.0	`lib` & `ext`	✅	system
ASDF	3.0.6	`lib` & `ext`	✅	system
ArchLinux	3.0.6p216	`ext`	❌	user
ASDF	3.0.6	`lib` & `ext`	✅	user

So now I can conclude this is not an issue with ruby 3.0 or 3.1, not an issue with the system/user install, but seems to be an issue with ArchLinux ruby. I'll check the probably apply a wrong patch or bad compilation flags.

noraj · 2023-09-04T10:46:56Z

Ok found the issue, it related to ArchLinux build.

There is a patch to disable to copy of the .so from ext to lib: https://gitlab.archlinux.org/archlinux/packaging/packages/rubygems/-/blob/main/rubygems_stop_so_duplication.patch?ref_type=heads

Upstream issue https://bugs.archlinux.org/task/70959

noraj changed the title ~~Loading~~ binary C extension not built? Aug 17, 2023

noraj closed this as completed Sep 3, 2023

boazsegev added a commit that referenced this issue Sep 3, 2023

possible fix for #148

5bea4f0

noraj reopened this Sep 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

binary C extension not built? #148

binary C extension not built? #148

noraj commented Aug 17, 2023

boazsegev commented Aug 19, 2023

noraj commented Aug 21, 2023

noraj commented Sep 3, 2023 •

edited

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

noraj commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

noraj commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

boazsegev commented Sep 3, 2023

noraj commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023 •

edited

janbiedermann commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 4, 2023

binary C extension not built? #148

binary C extension not built? #148

Comments

noraj commented Aug 17, 2023

System Information

Description

Expected behavior

Actual behavior

Notes

boazsegev commented Aug 19, 2023

noraj commented Aug 21, 2023

noraj commented Sep 3, 2023 • edited

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

noraj commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

noraj commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

janbiedermann commented Sep 3, 2023

boazsegev commented Sep 3, 2023

noraj commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023 • edited

janbiedermann commented Sep 4, 2023

janbiedermann commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 4, 2023

noraj commented Sep 3, 2023 •

edited

noraj commented Sep 4, 2023 •

edited