Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The gem fails to compile from source on Fedora 35 #354

Closed
eregon opened this issue Oct 4, 2022 · 17 comments · Fixed by #355
Closed

The gem fails to compile from source on Fedora 35 #354

eregon opened this issue Oct 4, 2022 · 17 comments · Fixed by #355

Comments

@eregon
Copy link
Contributor

eregon commented Oct 4, 2022

This is on CRuby 3.0.3. I'm trying from source since I saw a very similar error on TruffleRuby.
So it seems an issue in this gem since it also does not work on CRuby.
It could also be due to my environment but it's a pretty much vanilla Fedora 35.

$ ruby -v
ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]
$ gem i --platform=ruby -V sqlite3
HEAD https://index.rubygems.org/
200 OK
GET https://index.rubygems.org/info/sqlite3
200 OK
GET https://index.rubygems.org/info/ffi
200 OK
GET https://index.rubygems.org/info/mini_portile2
200 OK
Downloading gem sqlite3-1.5.2.gem
Downloading gem mini_portile2-2.8.0.gem
GET https://index.rubygems.org/gems/sqlite3-1.5.2.gem
Fetching sqlite3-1.5.2.gem
200 OK
GET https://index.rubygems.org/gems/mini_portile2-2.8.0.gem
Fetching mini_portile2-2.8.0.gem
200 OK
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/.github/FUNDING.yml
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/.github/workflows/ci.yml
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/.gitignore
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/CHANGELOG.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/Gemfile
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/LICENSE.txt
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/README.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/Rakefile
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/SECURITY.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/lib/mini_portile2.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile_cmake.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/lib/mini_portile2/version.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/mini_portile2.gemspec
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/git/config
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/gpg-fixtures/data
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/gpg-fixtures/data.asc
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/gpg-fixtures/data.invalid.asc
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/patch 1.diff
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/test mini portile-1.0.0/configure
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/test-cmake-1.0/CMakeLists.txt
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/test-cmake-1.0/hello.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/assets/test-download-archive.tar.gz
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/helper.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/test_cmake.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/test_cook.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/test_digest.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/test_download.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/test_execute.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/mini_portile2-2.8.0/test/test_proxy.rb
Successfully installed mini_portile2-2.8.0
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/.gemtest
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/API_CHANGES.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/CHANGELOG.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/CONTRIBUTING.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ChangeLog.cvs
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/Gemfile
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/LICENSE
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/LICENSE-DEPENDENCIES
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/README.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/dependencies.yml
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/aggregator.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/aggregator.h
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/backup.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/backup.h
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/database.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/database.h
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/exception.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/exception.h
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/extconf.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/sqlite3.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/sqlite3_ruby.h
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/statement.c
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/statement.h
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/faq/faq.md
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/faq/faq.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/faq/faq.yml
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/constants.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/database.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/errors.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/pragmas.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/resultset.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/statement.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/translator.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/value.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/version.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ports/archives/sqlite-autoconf-3390400.tar.gz
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/helper.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_backup.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_collation.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database_flags.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database_readonly.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database_readwrite.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_deprecated.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_encoding.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_aggregate.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_open_close.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_pending.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_resultset.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_statement.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_result_set.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_sqlite3.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_statement.rb
/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_statement_execute.rb
Building native extensions. This could take a while...
current directory: /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["/home/eregon/.rubies/ruby-3.0.3/bin/ruby", "-I", "/home/eregon/.rubies/ruby-3.0.3/lib/ruby/3.0.0", "-r", "./siteconf20221004-37471-9wjoba.rb", "extconf.rb"]
Building sqlite3-ruby using packaged sqlite3.
Extracting sqlite-autoconf-3390400.tar.gz into tmp/x86_64-redhat-linux/ports/sqlite3/3.39.4... OK
Running 'configure' for sqlite3 3.39.4... OK
Running 'compile' for sqlite3 3.39.4... OK
Running 'install' for sqlite3 3.39.4... OK
Activating sqlite3 3.39.4 (from /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ports/x86_64-redhat-linux/sqlite3/3.39.4)...
checking for whether -lsqlite3 is accepted as LDFLAGS... yes
checking for whether -lz is accepted as LDFLAGS... yes
checking for whether -lm is accepted as LDFLAGS... yes
checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... yes
checking for rb_proc_arity()... yes
checking for rb_integer_pack()... yes
checking for sqlite3_initialize()... yes
checking for sqlite3_backup_init()... yes
checking for sqlite3_column_database_name()... no
checking for sqlite3_enable_load_extension()... yes
checking for sqlite3_load_extension()... yes
checking for sqlite3_open_v2()... yes
checking for sqlite3_prepare_v2()... yes
checking for sqlite3_int64 in sqlite3.h... yes
checking for sqlite3_uint64 in sqlite3.h... yes
creating Makefile
current directory: /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["make", "DESTDIR=", "clean"]

current directory: /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["make", "DESTDIR="]
compiling aggregator.c
compiling backup.c
compiling database.c
compiling exception.c
compiling sqlite3.c
compiling statement.c
linking shared-object sqlite3/sqlite3_native.so
/usr/bin/ld: cannot find -lsqlite3
collect2: error: ld returned 1 exit status
make: *** [Makefile:262: sqlite3_native.so] Error 1
ERROR:  Error installing sqlite3:
	ERROR: Failed to build gem native extension.

    current directory: /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
/home/eregon/.rubies/ruby-3.0.3/bin/ruby -I /home/eregon/.rubies/ruby-3.0.3/lib/ruby/3.0.0 -r ./siteconf20221004-37471-9wjoba.rb extconf.rb
current directory: /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
make DESTDIR\= clean
current directory: /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
make DESTDIR\=
Building has failed. See above output for more information on the failure.
make failed, exit code 2

Gem files will remain installed in /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2 for inspection.
Results logged to /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/extensions/x86_64-linux/3.0.0/sqlite3-1.5.2/gem_make.out
zsh: exit 1     gem i --platform=ruby -V sqlite3
$ cd /home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2
$ find . -name 'libsqlite*'       
./ext/sqlite3/tmp/x86_64-redhat-linux/ports/sqlite3/3.39.4/sqlite-autoconf-3390400/.libs/libsqlite3.a
./ext/sqlite3/tmp/x86_64-redhat-linux/ports/sqlite3/3.39.4/sqlite-autoconf-3390400/.libs/libsqlite3.lai
./ext/sqlite3/tmp/x86_64-redhat-linux/ports/sqlite3/3.39.4/sqlite-autoconf-3390400/.libs/libsqlite3.la
./ext/sqlite3/tmp/x86_64-redhat-linux/ports/sqlite3/3.39.4/sqlite-autoconf-3390400/libsqlite3.la
./ports/x86_64-redhat-linux/sqlite3/3.39.4/lib/libsqlite3.la
./ports/x86_64-redhat-linux/sqlite3/3.39.4/lib/libsqlite3.a
$ which pkg-config
/usr/bin/pkg-config
$ pkg-config --version
1.8.0
$ pkg-config --libs --static ./ports/x86_64-redhat-linux/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc
-L/home/eregon/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib -lsqlite3 -lz -lm 
In ~/.rubies/ruby-3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3 $ make V=1
rm -f sqlite3_native.so
gcc -shared -o sqlite3_native.so aggregator.o backup.o database.o exception.o sqlite3.o statement.o -L. -L/home/eregon/.rubies/ruby-3.0.3/lib -Wl,-rpath,/home/eregon/.rubies/ruby-3.0.3/lib -L. -L/home/eregon/.rubies/ruby-3.0.3/lib  -fstack-protector-strong -rdynamic -Wl,-export-dynamic  -lsqlite3 -lz -lm -L/home/eregon/.rubies/ruby-3.0.3/lib  -Wl,--compress-debug-sections=zlib    -Wl,-rpath,/home/eregon/.rubies/ruby-3.0.3/lib -L/home/eregon/.rubies/ruby-3.0.3/lib -lruby -lsqlite3  -lsqlite3 -lm   -lc
/usr/bin/ld: cannot find -lsqlite3
collect2: error: ld returned 1 exit status
make: *** [Makefile:262: sqlite3_native.so] Error 1
zsh: exit 2     make V=1

So the -L seems gone and then of course it cannot find libsqlite3.

@eregon
Copy link
Contributor Author

eregon commented Oct 4, 2022

The problem seems to be that LIBRARY_PATH is set while calling pkg-config, and that influences pkg-config:

$ pkg-config --libs --static /home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc                                                                              
-L/home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib -lsqlite3 -lz -lm 

$ LIBRARY_PATH="/home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib:" pkg-config --libs --static /home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc 
-lsqlite3 -lz -lm 

LIBRARY_PATH is probably set by mini_portile2?

It also fails by cloning this repo and bundle exec rake compile BTW.

My system has /usr/lib64/libsqlite3.so.0.8.6 locally but no headers.

@eregon
Copy link
Contributor Author

eregon commented Oct 4, 2022

Ah, and while LIBRARY_PATH in set in the process running extconf.rb via https://github.com/flavorjones/mini_portile/blob/6e3b08ce5ab768921f04ae0aa6cc96a149cfb6ff/lib/mini_portile2/mini_portile.rb#L198
It's not set in the generated Makefile and in the make process and its subprocesses.
And so pkg-config seems to notice LIBRARY_PATH and so remove the -L... but it's actually needed in the separate process in which LIBRARY_PATH isn't set.

IOW the gcc -shared ... command above works with LIBRARY_PATH="/home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib:" but that's not set in the make process and its subprocesses.

@eregon
Copy link
Contributor Author

eregon commented Oct 4, 2022

pkg-config's --keep-system-libs seems to do what we want:

$ LIBRARY_PATH="/home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib:" pkg-config --libs --keep-system-libs --static /home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc
-L/home/eregon/code/sqlite3-ruby/ports/x86_64-redhat-linux/sqlite3/3.39.4/lib -lsqlite3 -lz -lm 

But maybe it's a bug of pkg-config to consider LIBRARY_PATH as a "system lib path" and/or a bug of mini_portile2 to set LIBRARY_PATH since that will be lost in the make process anyway?

Extract from my system's man page, where the wording is at least much less confusing than old pkg-config man pages:

     --keep-system-cflags, --keep-system-libs
             Keep CFLAGS or linker flag fragments that would be filtered due to being included by default in the compiler.

@eregon
Copy link
Contributor Author

eregon commented Oct 4, 2022

Seems we're not the firsts to fall into this trap of pkg-config: https://gitlab.kitware.com/cmake/cmake/-/issues/22148#note_980802
PKG_CONFIG_ALLOW_SYSTEM_LIBS=true (or PKG_CONFIG_ALLOW_SYSTEM_LIBS=1) also works to keep the -L.

@eregon
Copy link
Contributor Author

eregon commented Oct 4, 2022

Using PKG_CONFIG_ALLOW_SYSTEM_LIBS=true / --keep-system-libs has the potential reverse problem of adding actual system paths like /usr/lib which might make it harder to prefer other libraries, so it doesn't seem ideal (rust-lang/pkg-config-rs#11).

So the best solution seems to not set LIBRARY_PATH in mini_portile2, or ensure it's also set in the Makefile (e.g. via ENV['LIBRARY_PATH'].split(File::PATH_SEPARATOR).each { append_ldflags "-L#{_1}" }), so that the env is the same in the Makefile and in the extconf.rb process.
Similar for PATH and CPATH too, set in https://github.com/flavorjones/mini_portile/blob/6e3b08ce5ab768921f04ae0aa6cc96a149cfb6ff/lib/mini_portile2/mini_portile.rb#L196-L197

@flavorjones
Copy link
Member

Thanks for the report and investigation. I'll take a look.

@eregon
Copy link
Contributor Author

eregon commented Oct 10, 2022

@flavorjones Could you take a look? This prevents installing sqlite3 from source on Linux (and so always on TruffleRuby).

@flavorjones
Copy link
Member

@eregon Looking now.

I do want to note that this works fine on some Linuxes (CI runs with and without system libs, and here's my local dev system), which are all debian-based systems ... so I'm going to start with trying to repro on Fedora.

@flavorjones
Copy link
Member

flavorjones commented Oct 10, 2022

@eregon I need your help reproducing this.

Here's how I set up the Fedora system I'm starting with the docker image `fedora:35`. Here's my provisioning script:
#! /usr/bin/env bash

set -ex

# https://developer.fedoraproject.org/tech/languages/ruby/ruby-installation.html
dnf install git-core zlib zlib-devel gcc-c++ patch readline readline-devel libyaml-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison curl sqlite-devel

cd
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
source ~/.bashrc

git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
echo 'export PATH="$HOME/.rbenv/plugins/ruby-build/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

rbenv install 3.0.3
rbenv global 3.0.3
ruby -v

echo "gem: --no-document" > ~/.gemrc
gem install bundler

which installs ruby:

[root@5c7eb6203813 ~]# ruby -v
ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]

And here's the output from gem install sqlite3 --platform=ruby --verbose:

HEAD https://index.rubygems.org/
200 OK
GET https://index.rubygems.org/info/sqlite3
200 OK
GET https://index.rubygems.org/quick/Marshal.4.8/sqlite3-1.5.2-x86_64-linux.gemspec.rz
200 OK
GET https://index.rubygems.org/info/ffi
200 OK
GET https://index.rubygems.org/info/mini_portile2
200 OK
GET https://index.rubygems.org/quick/Marshal.4.8/sqlite3-1.5.2.gemspec.rz
200 OK
Downloading gem sqlite3-1.5.2.gem
GET https://index.rubygems.org/gems/sqlite3-1.5.2.gem
200 OK
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/.gemtest
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/API_CHANGES.md
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/CHANGELOG.md
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/CONTRIBUTING.md
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ChangeLog.cvs
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/Gemfile
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/LICENSE
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/LICENSE-DEPENDENCIES
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/README.md
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/dependencies.yml
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/aggregator.c
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/aggregator.h
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/backup.c
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/backup.h
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/database.c
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/database.h
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/exception.c
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/exception.h
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/extconf.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/sqlite3.c
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/sqlite3_ruby.h
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/statement.c
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3/statement.h
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/faq/faq.md
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/faq/faq.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/faq/faq.yml
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/constants.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/database.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/errors.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/pragmas.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/resultset.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/statement.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/translator.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/value.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/lib/sqlite3/version.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ports/archives/sqlite-autoconf-3390400.tar.gz
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/helper.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_backup.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_collation.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database_flags.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database_readonly.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_database_readwrite.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_deprecated.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_encoding.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_aggregate.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_open_close.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_pending.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_resultset.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_integration_statement.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_result_set.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_sqlite3.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_statement.rb
/root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/test/test_statement_execute.rb
Building native extensions. This could take a while...
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["/root/.rbenv/versions/3.0.3/bin/ruby", "-I", "/root/.rbenv/versions/3.0.3/lib/ruby/3.0.0", "-r", "./siteconf20221010-16674-k4uz7v.rb", "extconf.rb"]
Building sqlite3-ruby using packaged sqlite3.
Extracting sqlite-autoconf-3390400.tar.gz into tmp/x86_64-redhat-linux/ports/sqlite3/3.39.4... OK
Running 'configure' for sqlite3 3.39.4... OK
Running 'compile' for sqlite3 3.39.4... OK
Running 'install' for sqlite3 3.39.4... OK
Activating sqlite3 3.39.4 (from /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ports/x86_64-redhat-linux/sqlite3/3.39.4)...
checking for whether -lsqlite3 is accepted as LDFLAGS... yes
checking for whether -lz is accepted as LDFLAGS... yes
checking for whether -lm is accepted as LDFLAGS... yes
checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... yes
checking for rb_proc_arity()... yes
checking for rb_integer_pack()... yes
checking for sqlite3_initialize()... yes
checking for sqlite3_backup_init()... yes
checking for sqlite3_column_database_name()... yes
checking for sqlite3_enable_load_extension()... yes
checking for sqlite3_load_extension()... yes
checking for sqlite3_open_v2()... yes
checking for sqlite3_prepare_v2()... yes
checking for sqlite3_int64 in sqlite3.h... yes
checking for sqlite3_uint64 in sqlite3.h... yes
creating Makefile
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["make", "DESTDIR=", "clean"]

current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["make", "DESTDIR="]
compiling aggregator.c
compiling backup.c
compiling database.c
compiling exception.c
compiling sqlite3.c
compiling statement.c
linking shared-object sqlite3/sqlite3_native.so
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
["make", "DESTDIR=", "install"]
/usr/bin/install -c -m 0755 sqlite3_native.so ./.gem.20221010-16674-7ajazx/sqlite3
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
/root/.rbenv/versions/3.0.3/bin/ruby -I /root/.rbenv/versions/3.0.3/lib/ruby/3.0.0 -r ./siteconf20221010-16674-k4uz7v.rb extconf.rb
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
make DESTDIR\= clean
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
make DESTDIR\=
current directory: /root/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/sqlite3-1.5.2/ext/sqlite3
make DESTDIR\= install
Successfully installed sqlite3-1.5.2
1 gem installed

@flavorjones
Copy link
Member

Ah, ok, if sqlite-devel isn't installed then I get the same failure. I'm unblocked. Continuing.

@flavorjones
Copy link
Member

I believe this has to do with behavior differences in pkg-config between Fedora and Debian.

https://fedoraproject.org/wiki/Changes/pkgconf_as_system_pkg-config_implementation

On Debian, the output of pkg-config --libs --static path/to/sqlite3.pc is:

-L/home/flavorjones/code/oss/sqlite3-ruby/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib -lsqlite3 -lz -lm -ldl -lpthread

but on Fedora it is:

-lsqlite3 -lz -lm

which logically leads to that share object not being found.

@flavorjones
Copy link
Member

flavorjones commented Oct 10, 2022

OK, all caught up and I agree with your diagnosis. I'll take a look at whether mini_portile can work around this without introducing a breaking change for someboday, but I'm not confident.

@eregon
Copy link
Contributor Author

eregon commented Oct 10, 2022

Interesting that this doesn't happen on Debian. IIRC in one of the links there was a mention of 2 pkg-config impls and both seem to have this behavior of considering LIBRARY_PATH as part of the "system path "without --keep-system-libs, so it seems intended behavior (and I'd agree it'd make sense if the env is kept).

I think the fundamental issue here is the different environment in the extconf.rb and the Makefile (#354 (comment)).
Changing the environment in mini_portile2 seems unexpected to me, maybe there could be an alternative to activate or a kwarg to not set in the environment but rather return a Hash or set the state for mkmf but not in ENV, or set both in mkmf and in ENV.

@flavorjones
Copy link
Member

I'm going to work around this in sqlite3-ruby/ext/sqlite3/extconf.rb for now, but will open up a follow-up issue in mini_portile to look a bit deeper at preferring LDFLAGS to LIBRARY_PATH (which I think should be backwards-compatible but deserves thorough testing before shipping a release).

@flavorjones
Copy link
Member

I've created flavorjones/mini_portile#118 upstream to track a permanent fix.

@flavorjones
Copy link
Member

v1.5.3 has been released which fixes this.

Release 1.5.3 / 2022-10-11 · sparklemotion/sqlite3-ruby

@eregon
Copy link
Contributor Author

eregon commented Oct 11, 2022

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants