Curl flake with ^out,dev not adding dynamic libraries #1852

undecided · 2024-02-27T16:41:53Z

What happened?

Having just worked around #1851 and deleted my /nix and reinstalled everything, suddenly my project is broken thanks to ruby's FFI being unable to find libcurl.

Looks like even with the curl line added, .devbox/nix/profile/default/lib (as per $LD_LIBRARY_PATH) does not contain libcurl

Steps to reproduce

With export DEVBOX_USE_VERSION=0.8.7 run the script devbox run problem - it works
unset DEVBOX_USE_VERSION and with the latest devbox, run the same script - it breaks
With 0.9.2-dev it also breaks (with new-style glibc-patch and --output out,dev, of course!)

Command

run, shell

devbox.json

{
  "packages": [
    "ruby@latest",
    "github:nixos/nixpkgs#curl^out,dev"
  ],
  "shell": {
    "scripts": {
      "problem": "gem install typhoeus ; ruby -rtyphoeus -e 'puts :hi'"
    }
  }
}

Devbox version

0.9.1

Nix version

nix (Nix) 2.18.1

What system does this bug occur on?

Linux (x86-64)

Debug logs

❯ DEVBOX_USE_VERSION=0.8.7 devbox run problem
Info: New devbox available: 0.8.7 -> v0.9.1. Please run `devbox version update`.
Ensuring packages are installed.

Installing package: curl.

[1/1] curl
[1/1] curl: Success
✓ Computed the Devbox environment.
Successfully installed typhoeus-1.4.1
Parsing documentation for typhoeus-1.4.1
Done installing documentation for typhoeus after 0 seconds
1 gem installed
hi

❯ DEVBOX_USE_VERSION=0.9.1 devbox run problem
Ensuring packages are installed.
✓ Computed the Devbox environment.
Info: Removing curl@8.6.0
Successfully installed typhoeus-1.4.1
Parsing documentation for typhoeus-1.4.1
Done installing documentation for typhoeus after 0 seconds
1 gem installed
/home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/dynamic_library.rb:65:in `load_library': Could not open library 'libcurl': libcurl: cannot open shared object file: No such file or directory. (LoadError)
Could not open library 'libcurl.so': libcurl.so: cannot open shared object file: No such file or directory.
Could not open library 'libcurl.so.4': libcurl.so.4: cannot open shared object file: No such file or directory.
Searched in <system library path>, /usr/lib, /usr/local/lib, /opt/local/lib
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/library.rb:95:in `block in ffi_lib'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `map'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `ffi_lib'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curls/settings.rb:10:in `<module:Curl>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curls/settings.rb:3:in `<module:Ethon>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curls/settings.rb:2:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curl.rb:28:in `<module:Curl>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curl.rb:14:in `<module:Ethon>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curl.rb:9:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon.rb:16:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/typhoeus-1.4.1/lib/typhoeus.rb:2:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
<internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- typhoeus (LoadError)
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
Error: error running script "problem" in Devbox: exit status 1

Also, this might be useful:

❯ DEVBOX_USE_VERSION=0.8.7 devbox run 'ls $LD_LIBRARY_PATH'
Info: New devbox available: 0.8.7 -> v0.9.1. Please run `devbox version update`.
libcurl.la  libcurl.so	libcurl.so.4  libcurl.so.4.8.0	libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby
 
❯ DEVBOX_USE_VERSION=0.9.1 devbox run 'ls $LD_LIBRARY_PATH'
Ensuring packages are installed.
✓ Computed the Devbox environment.
Info: Removing curl@8.6.0
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

Changing the devbox.json to force it to reinstall (e.g. changing github:nixos/ to github:NixOS/) gives this:

❯ DEVBOX_USE_VERSION=0.9.1 devbox run 'ls $LD_LIBRARY_PATH'
Ensuring packages are installed.
[1/1] curl
[1/1] curl: Success
✓ Computed the Devbox environment.
Info: Removing curl@8.6.0
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

The text was updated successfully, but these errors were encountered:

gcurtis · 2024-02-27T16:47:47Z

Thanks for the thorough bug report. I'll take a look at this today.

gcurtis · 2024-02-27T22:09:33Z

@savil it looks like we're dropping the out output from curl when it's specified as a flake reference. I think this might be a regression after some of the recent lock file changes.

Here's what I see happening with the Devbox nix profile using the devbox.json from the issue (logs snipped to only include relevant messages).

Enter a shell, see that only ruby and curl^dev are installed to the profile (curl^out is missing).

$ devbox version -v
Version:     0.0.0-dev
Platform:    linux_arm64
Commit:      0662f451222fe3195127f6376d49a8a7239192d3
Go Version:  go1.22.0
$ devbox shell
2024/02/27 16:49:15 running command: /run/current-system/sw/bin/nix profile install --profile /home/gcurtis/devbox/nixosvm/issue1852/.devbox/nix/profile/default --impure --priority 6 --extra-experimental-features ca-derivations --option experimental-features nix-command flakes fetch-closure --offline /nix/store/0dqds3mklwg3jygw6gwi7dknag2q7fik-ruby-3.3.0
2024/02/27 16:49:15 running command: /run/current-system/sw/bin/nix profile install --profile /home/gcurtis/devbox/nixosvm/issue1852/.devbox/nix/profile/default --impure --priority 7 --extra-experimental-features ca-derivations --option experimental-features nix-command flakes fetch-closure --offline /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev

See that the curl libraries are missing from the profile's lib directory.

(devbox) $ ls .devbox/nix/profile/default/lib
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

See that only curl^dev is installed to the profile.

(devbox) $ nix profile list --profile .devbox/nix/profile/default
Index:              0
Store paths:        /nix/store/0dqds3mklwg3jygw6gwi7dknag2q7fik-ruby-3.3.0

Index:              1
Store paths:        /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev

Manually add nixpkgs#curl^out,dev to the profile and see that libcurl appears in the profile's lib directory.

(devbox) $ nix profile install --profile .devbox/nix/profile/default nixpkgs#curl^out,dev
(devbox) $ nix profile list --profile .devbox/nix/profile/default
Index:              0
Store paths:        /nix/store/0dqds3mklwg3jygw6gwi7dknag2q7fik-ruby-3.3.0

Index:              1
Store paths:        /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev

Index:              2
Flake attribute:    legacyPackages.aarch64-linux.curl^dev,out
Original flake URL: flake:nixpkgs
Locked flake URL:   github:NixOS/nixpkgs/8520c158aee718c6e87b56881105fc4223c3c723
Store paths:        /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev /nix/store/ilzq62jr6sj7p2vlq0sjsmg1264w1vgf-curl-8.6.0
(devbox) $ ls .devbox/nix/profile/default/lib
libcurl.la  libcurl.so	libcurl.so.4  libcurl.so.4.8.0	libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

Exit and re-enter the devbox shell. See that devbox removed curl^out (but not curl^dev) from the profile.

(devbox) $ exit
$ devbox shell
2024/02/27 17:01:24 Removing packages from nix profile: curl@8.6.0

See that libcurl is missing again.

(devbox) $ ls .devbox/nix/profile/default/lib
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

undecided · 2024-02-28T10:15:15Z

Thanks for investigating. Off the back of that, in case it's any use to anyone else, I've been able to add the following to my devbox.json as a workaround, and it seems to work nicely

  "shell": {
    "init_hook": [
      "nix profile install --profile .devbox/nix/profile/default nixpkgs#curl^out,dev --extra-experimental-features nix-command --extra-experimental-features flakes"

With that, when I do devbox shell, it's all installed correctly

❯ devbox version
0.9.1

❯ ls .devbox/nix/profile/default/lib/*curl*
.devbox/nix/profile/default/lib/libcurl.la  .devbox/nix/profile/default/lib/libcurl.so.4
.devbox/nix/profile/default/lib/libcurl.so  .devbox/nix/profile/default/lib/libcurl.so.4.8.0

## Summary Fixes #1852 Ensure we initialize outputs using flake if package is a flake. Made outputs non-pointer so we don't have to worry about initialization. ## How was it tested? Installed `github:nixos/nixpkgs#curl^out,dev` and verified that flake and profile contain both outputs.

undecided added bug Something isn't working triage Issue needs triage labels Feb 27, 2024

gcurtis removed the triage Issue needs triage label Feb 27, 2024

gcurtis self-assigned this Feb 27, 2024

gcurtis added ruby Support for Ruby, gems, Bundler, etc. linux Linux systems x86-64 x86-64 systems labels Feb 27, 2024

mikeland73 mentioned this issue Feb 28, 2024

[nix-outputs] Fix flake installable with custom outouts #1864

Merged

mikeland73 closed this as completed in #1864 Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Curl flake with ^out,dev not adding dynamic libraries #1852

Curl flake with ^out,dev not adding dynamic libraries #1852

undecided commented Feb 27, 2024

gcurtis commented Feb 27, 2024

gcurtis commented Feb 27, 2024 •

edited

Loading

undecided commented Feb 28, 2024

Curl flake with ^out,dev not adding dynamic libraries #1852

Curl flake with ^out,dev not adding dynamic libraries #1852

Comments

undecided commented Feb 27, 2024

What happened?

Steps to reproduce

Command

devbox.json

Devbox version

Nix version

What system does this bug occur on?

Debug logs

gcurtis commented Feb 27, 2024

gcurtis commented Feb 27, 2024 • edited Loading

undecided commented Feb 28, 2024

gcurtis commented Feb 27, 2024 •

edited

Loading