[BUG] Can not install digest libraries

[nicobayati]

Hello,

in net-pop version 0.1.1 ( https://github.com/ruby/net-pop/releases/tag/v0.1.1 ) digest was added as a dependency, since then it tries to install it for me when I want to update my gems with "gem update".

I want to update the gems in my HOME directory as an unprivileged user. What I notice at first glance is that it tries to delete a file from my system, which of course fails due to lack of permissions.

I am sorry if I am wrong here with my request, maybe it is a net-pop issue? I am wondering why it is trying to install digest as a user when it should be installed under the system context.

Thank you.
gem_environment.txt
gem_update_output.txt

[ahwatts]

I'm getting the same issue when trying to do a gem update to my ~/.gem path. It appears that digest wants to install digest.h to the installed ruby's include path, which should probably be handled like install binaries, but is not, probably because most gems don't need to install header files. But this one does, I guess for backwards compatibility with the version that was in ruby?

awatts@ironic-mullet ~ $ gem update
Updating installed gems
Updating net-pop
Building native extensions. This could take a while...
ERROR:  Error installing net-pop:
	ERROR: Failed to build gem native extension.

    current directory: /home/awatts/.gem/ruby/gems/digest-3.0.0/ext/digest
/usr/bin/ruby -I /usr/share/rubygems -r ./siteconf20210106-373506-2lhso8.rb extconf.rb
creating Makefile

current directory: /home/awatts/.gem/ruby/gems/digest-3.0.0/ext/digest
make "DESTDIR=" clean
rm -f 
rm -f digest.so  *.o  *.bak mkmf.log .*.time

current directory: /home/awatts/.gem/ruby/gems/digest-3.0.0/ext/digest
make "DESTDIR="
gcc -I. -I/usr/include -I/usr/include/ruby/backward -I/usr/include -I.   -fPIC -O2  -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fPIC -m64 -o digest.o -c digest.c
rm -f digest.so
gcc -shared -o digest.so digest.o -L. -L/usr/lib64 -L. -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld   -m64  -lruby  -lm   -lc

current directory: /home/awatts/.gem/ruby/gems/digest-3.0.0/ext/digest
make "DESTDIR=" install
/usr/bin/mkdir -p . ./.gem.20210106-373506-1k6iyj0
exit > .sitearchdir.time
/usr/bin/install -c -m 0755 digest.so ./.gem.20210106-373506-1k6iyj0
/usr/bin/mkdir -p . /usr/include/ruby
exit > .rubyhdrdir.-.ruby.time
/usr/bin/install -c -m 644 ./digest.h /usr/include/ruby
/usr/bin/install: cannot remove '/usr/include/ruby/digest.h': Permission denied
make: *** [Makefile:204: /usr/include/ruby/digest.h] Error 1

make install failed, exit code 2

Gem files will remain installed in /home/awatts/.gem/ruby/gems/digest-3.0.0 for inspection.
Results logged to /home/awatts/.gem/ruby/extensions/x86_64-linux/2.7.0/digest-3.0.0/gem_make.out
Gems updated: digest

[yorickpeterse]

We're also running into this at GitLab, resulting in builds errors such as the following:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory:
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/digest-3.0.0/ext/digest
/usr/bin/ruby -I /usr/lib/ruby/2.7.0 -r ./siteconf20210324-9-1adwezm.rb
extconf.rb
creating Makefile
current directory:
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/digest-3.0.0/ext/digest
make "DESTDIR=" clean
current directory:
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/digest-3.0.0/ext/digest
make "DESTDIR="
compiling digest.c
linking shared-object digest.so
current directory:
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/digest-3.0.0/ext/digest
make "DESTDIR=" install
/usr/bin/install -c -m 0755 digest.so ./.gem.20210324-9-yxcxdn
installing digest libraries
/usr/bin/install: cannot remove '/usr/include/ruby-2.7.0/ruby/digest.h':
Permission denied
make: *** [Makefile:204: /usr/include/ruby-2.7.0/ruby/digest.h] Error 1
make install failed, exit code 2
Gem files will remain installed in
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/digest-3.0.0 for inspection.
Results logged to
/srv/gitlab/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/digest-3.0.0/gem_make.out
An error occurred while installing digest (3.0.0), and Bundler cannot continue.
Make sure that `gem install digest -v '3.0.0' --source 'https://rubygems.org/'`
succeeds before bundling.
In Gemfile:
  gitlab-mail_room was resolved to 0.0.10, which depends on
    net-imap was resolved to 0.2.1, which depends on
      digest

I'm a bit surprised this Gem is even trying to mess with system files. I can see that breaking all sorts of things. In addition, it seems that rolling back would require you to:

1. Uninstall the Gem
2. Reinstall Ruby, otherwise the header file may not be compatible

[stanhu]

As announced in https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-released/, Ruby v3.0.0 promoted a number of stdlib libraries, including this one, into a default gem. I think the error above happens because if Ruby 2.7 attempts to use these gems, the gem will attempt to overwrite the system /usr/include/ruby/digest.h, which may not be permitted on the system.

Should this gem be disallowed with Ruby v2 versions, or can these header files be installed locally with the gem?

[ahwatts]

I think the problem would also arise with Ruby 3.0.0, though. The reason it isn't right now is because Ruby 3.0.0 has digest 3.0.0 as a default gem. If digest were to, for some reason, release a 3.0.1 version, installing that gem with a system Ruby of 3.0.0 would try to update the digest.h file and run in to this issue. Do default gems not get released out-of-cycle with Ruby versions like that?

Similarly, Ruby updating to 3.0.1 would presumably produce a digest 3.0.1, which Ruby 3.0.0 installations would try to install if one does a gem update, and they would then start running in to this issue.

Finally, if one is using Bundler and included digest 3.0.0 in the bundle, but (hypothetical) digest 3.0.1 had been installed alongside digest 3.0.0 (and didn't run in to this issue), the digest.h that extensions built for that bundle would see would be the one for digest 3.0.1, since Rubygems doesn't manage header files, so far as I know.

Because of the fact that it depends on a header file that others can depend on (in my cursory looking, it appears that only https://rubygems.org/gems/digest-sha3 actually does use it), either it shouldn't be a gem or else Rubygems should offer some facility to manage header files.

[stanhu]

Good point, I realized the same after I wrote my comment. It seems that https://github.com/ruby/digest/blob/master/ext/digest/extconf.rb will always install this header because HDRDIR points to the Ruby interpreter include directory. https://github.com/phusion/digest-sha3-ruby/blob/master/ext/digest/extconf.rb#L3 currently uses the have_header method, which looks in the system ruby directory.

I wonder if Ruby has a good way to handle this. Gems that need to link against a specific digest.h should probably be tied to a specific version of digest. Should gems like digest-sha3 download the source here directly?

[stanhu]

@mrkn What do you think about this issue?

[mrkn]

@stanhu I don't have any idea about this issue right now because I've never touched extconf.rb in this library.

[mrkn]

@stanhu As my understanding, this issue is occurred with the following conditions:

1. Use system ruby
2. Setting a user-writable directory in GEM_PATH
3. Running gem install digest as non-root user

RIght?

[stanhu]

@mrkn Yes, I believe that's correct.

[mrkn]

I guess we need to discuss where should we install a header file when gem install --user.

[mrkn]

I opened the ticket for further discussion for the header install location at https://bugs.ruby-lang.org/issues/17760.

And please run gem update digest by root user before running gem update by non-root user for now.

[stanhu]

It looks like ruby/ruby@5cdf99f (from https://bugs.ruby-lang.org/issues/17761) will fix this issue.

[voxik]

I wonder, if there might be similar case already handled in https://github.com/ruby-gnome/ruby-gnome, which is quite complex system of gems. @kou any insights please?

[kou]

How about the following?

diff --git a/ext/digest/depend b/ext/digest/depend
new file mode 100644
index 0000000..a454e12
--- /dev/null
+++ b/ext/digest/depend
@@ -0,0 +1,3 @@
+install-so: install-headers
+install-headers:
+	$(INSTALL_DATA) $(srcdir)/digest.h $(RUBYARCHDIR)
diff --git a/ext/digest/extconf.rb b/ext/digest/extconf.rb
index a8c6885..ba2c7b6 100644
--- a/ext/digest/extconf.rb
+++ b/ext/digest/extconf.rb
@@ -4,8 +4,4 @@
 
 require "mkmf"
 
-$INSTALLFILES = {
-  "digest.h" => "$(HDRDIR)"
-}
-
 create_makefile("digest")

[knu]

I've just released v3.1.0.pre3 which should fix this issue. (0e1e672)

[nevans]

@knu v3.1.0.pre3 doesn't fix the issue for me in ruby 2.7.4

setting up an unprivileged user:

$ docker run --rm -it ruby:2.7.4 bash
root@67a522cc35ef:/# useradd --create-home --shell /bin/bash --user-group user; su - user
user@67a522cc35ef:~$ export GEM_HOME="$HOME/gems/$RUBY_MAJOR"
user@67a522cc35ef:~$ export GEM_PATH="${GEM_HOME}:${GEM_PATH}"

Proving the user can install other gems and compile their extensions:

user@67a522cc35ef:~$ gem install json
Fetching json-2.6.1.gem
Building native extensions. This could take a while...
Successfully installed json-2.6.1
1 gem installed

Cannot install v3.0.0:

user@67a522cc35ef:~$ gem install digest
Fetching digest-3.0.0.gem
Building native extensions. This could take a while...
ERROR:  Error installing digest:
        ERROR: Failed to build gem native extension.

    current directory: /home/user/gems/gems/digest-3.0.0/ext/digest
/usr/local/bin/ruby -I /usr/local/lib/ruby/2.7.0 -r ./siteconf20211119-101-1sawl1h.rb extconf.rb
creating Makefile

current directory: /home/user/gems/gems/digest-3.0.0/ext/digest
make "DESTDIR=" clean

current directory: /home/user/gems/gems/digest-3.0.0/ext/digest
make "DESTDIR="
compiling digest.c
linking shared-object digest.so

current directory: /home/user/gems/gems/digest-3.0.0/ext/digest
make "DESTDIR=" install
/usr/bin/install -c -m 0755 digest.so ./.gem.20211119-101-faw7ja
installing digest libraries
/usr/bin/install: cannot remove '/usr/local/include/ruby-2.7.0/ruby/digest.h': Permission denied
make: *** [Makefile:204: /usr/local/include/ruby-2.7.0/ruby/digest.h] Error 1

make install failed, exit code 2

Gem files will remain installed in /home/user/gems/gems/digest-3.0.0 for inspection.
Results logged to /home/user/gems/extensions/x86_64-linux/2.7.0/digest-3.0.0/gem_make.out

Cannot install v3.1.0.pre3:

user@67a522cc35ef:~$ gem install digest:3.1.0.pre3
Fetching digest-3.1.0.pre3.gem
Building native extensions. This could take a while...
ERROR:  Error installing digest:
        ERROR: Failed to build gem native extension.

    current directory: /home/user/gems/gems/digest-3.1.0.pre3/ext/digest
/usr/local/bin/ruby -I /usr/local/lib/ruby/2.7.0 -r ./siteconf20211119-137-1v2e6u6.rb extconf.rb
creating Makefile

current directory: /home/user/gems/gems/digest-3.1.0.pre3/ext/digest
make "DESTDIR=" clean

current directory: /home/user/gems/gems/digest-3.1.0.pre3/ext/digest
make "DESTDIR="
compiling digest.c
linking shared-object digest.so

current directory: /home/user/gems/gems/digest-3.1.0.pre3/ext/digest
make "DESTDIR=" install
/usr/bin/install -c -m 0755 digest.so ./.gem.20211119-137-rsw835
installing header files
install -c -m 644 ./digest.h /usr/local/include/ruby-2.7.0/ruby
/usr/local/lib/ruby/2.7.0/fileutils.rb:1415:in `initialize': Permission denied @ rb_sysopen - /usr/local/include/ruby-2.7.0/ruby/digest.h (Errno::EACCES)
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1415:in `open'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1415:in `block in copy_file'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1414:in `open'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1414:in `copy_file'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:511:in `copy_file'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:880:in `block in install'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1589:in `block in fu_each_src_dest'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1603:in `fu_each_src_dest0'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:1587:in `fu_each_src_dest'
        from /usr/local/lib/ruby/2.7.0/fileutils.rb:876:in `install'
        from install_headers.rb:9:in `block in <main>'
        from install_headers.rb:8:in `each'
        from install_headers.rb:8:in `<main>'
make: *** [Makefile:283: do-install-headers] Error 1

make install failed, exit code 2

Gem files will remain installed in /home/user/gems/gems/digest-3.1.0.pre3 for inspection.
Results logged to /home/user/gems/extensions/x86_64-linux/2.7.0/digest-3.1.0.pre3/gem_make.out

What do you think?