What is the process of building native extension?

[hurricup]

I was thinking that it works like this:

1. rubygems unpacks gem archive into appropriate directory, e.g. $GEM_PATH/gems/sqlite3-1.4.2
2. puts gemspec into $GEM_PATH/specifications/
3. processes the gemspec and at some point runs extconf.rb specified in the extensions attribute of the gemspec
4. extconf.rb creates a Makefile inside the $GEM_PATH/gems/sqlite3-1.4.2/ext directory and invokes make
5. make builds the shared library and puts it into the lib. And that's it.

But what we have in real life:

1. we have a gem files in $GEM_PATH/gems/sqlite3-1.4.2
2. we have an object files, Makefile and shared library alongside the source files in $GEM_PATH/gems/sqlite3-1.4.2/ext/, but no building logs.
3. we have a shared library in the lib directory.
4. we have a separate extensions directory: $GEM_PATH/extensions/x86_64-linux/3.1.0/sqlite3-1.4.2/ with a building logs (gem.build_complete, gem_make.out, mkmf.log ) and resulting shared library.

So:

1. What is this extensions directory about?
2. Why do we need 3 copies of shared library? In lib, ext, and extensions directories?
3. Why rubygems adds both paths to the $LOAD_PATH: $GEM_PATH/gems/sqlite3-1.4.2/lib and $GEM_PATH/extensions/x86_64-linux/3.1.0/sqlite3-1.4.2/ if we expecting to have shared library in lib directory?
4. For what reason shared library file may be missing in lib directory, but present in ext/extensions directories? (usually this happens on some system rubies)

[deivid-rodriguez]

Probably does not fully answer your question but gives some hint to investigate. I think there's some backwards compatibility stuff that makes the built extension live in several places:

rubygems/lib/rubygems/ext/ext_conf_builder.rb

Lines 68 to 75 in 5d65b7f

	# TODO remove in RubyGems 3
	if Gem.install_extension_in_lib and lib_dir
	FileUtils.mkdir_p lib_dir
	entries = Dir.entries(full_tmp_dest) - %w[. ..]
	entries = entries.map {|entry| File.join full_tmp_dest, entry }
	FileUtils.cp_r entries, lib_dir, :remove_destination => true
	end

(By the way that TODO should be moved to "RubyGems 4", because it was obviously not removed in RubyGems 3 😅.

[hurricup]

Is there some perspective on how it should work in future?

[hurricup]

Introduced in #749

[hurricup]

Seems the plan is to NOT install so files into the lib

[mohits]

I have been doing quite a bit of looking at how native gems are built in a series of blog posts: https://notepad.onghu.com/2023/learning-by-reversing-s1-native-gems/

I hope that this helps at least a bit?