Trouble with openssl and RHEL 6.3

[juanitogan]

Openssl does not appear to be compiling correctly when installing any version of Ruby on our RHEL 6.3 system. I have been trying to leave user installs of RVM behind and replace them with root installs via ruby-install. Openssl works fine in our RVM user installs (with the prescribed RVM fix) as well as in the built-in system install of Ruby 1.8.7 in /usr/bin.

Openssl is broken in each Ruby version I have tried with ruby-install: 1.9.3-p392 (our prod version), 1.9 latest, and 2.1.0 current. I have tried every openssl fix/workaround I can find, such as the --with-openssl-dir config pointing to various openssl folders, but nothing works for me. I feel like I'm missing something basic but can't see it not being a Linux guru.

Here are some relevant messages from a few of my many attempts:

[root@dbatcit ~]# ruby-install ruby
>>> Installing ruby 2.1.0 into /opt/rubies/ruby-2.1.0 ...
>>> Installing dependencies for ruby 2.1.0 ...
Loaded plugins: product-id, rhnplugin, security, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Install Process
Package gcc-4.4.7-4.el6.x86_64 already installed and latest version
Package automake-1.11.1-4.el6.noarch already installed and latest version
Package zlib-devel-1.2.3-29.el6.x86_64 already installed and latest version
Package libyaml-devel-0.1.3-1.el6.x86_64 already installed and latest version
Package openssl-devel-1.0.1e-16.el6_5.4.x86_64 already installed and latest version
Package gdbm-devel-1.8.0-36.el6.x86_64 already installed and latest version
Package readline-devel-6.0-4.el6.x86_64 already installed and latest version
Package ncurses-devel-5.7-3.20090208.el6.x86_64 already installed and latest version
Package libffi-devel-3.0.5-3.2.el6.x86_64 already installed and latest version
Nothing to do
.
.
make[2]: Entering directory `/usr/local/src/ruby-2.1.0/ext/openssl'
compiling ossl_pkey.c
compiling ossl_ssl.c
ossl_ssl.c:121: error: âTLSv1_2_methodâ undeclared here (not in a function)
ossl_ssl.c:122: error: âTLSv1_2_server_methodâ undeclared here (not in a function)
ossl_ssl.c:123: error: âTLSv1_2_client_methodâ undeclared here (not in a function)
ossl_ssl.c:127: error: âTLSv1_1_methodâ undeclared here (not in a function)
ossl_ssl.c:128: error: âTLSv1_1_server_methodâ undeclared here (not in a function)
ossl_ssl.c:129: error: âTLSv1_1_client_methodâ undeclared here (not in a function)
make[2]: *** [ossl_ssl.o] Error 1
make[2]: Leaving directory `/usr/local/src/ruby-2.1.0/ext/openssl'
make[1]: *** [ext/openssl/all] Error 2
make[1]: Leaving directory `/usr/local/src/ruby-2.1.0'
make: *** [build-ext] Error 2
!!! Compiling ruby 2.1.0 failed!



[root@dbatcit ~]# ruby-install ruby 1.9
>>> Installing ruby 1.9.3-p484 into /opt/rubies/ruby-1.9.3-p484 ...
>>> Installing dependencies for ruby 1.9.3-p484 ...
Loaded plugins: product-id, rhnplugin, security, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Install Process
Package gcc-4.4.7-4.el6.x86_64 already installed and latest version
Package automake-1.11.1-4.el6.noarch already installed and latest version
Package zlib-devel-1.2.3-29.el6.x86_64 already installed and latest version
Package libyaml-devel-0.1.3-1.el6.x86_64 already installed and latest version
Package openssl-devel-1.0.1e-16.el6_5.4.x86_64 already installed and latest version
Package gdbm-devel-1.8.0-36.el6.x86_64 already installed and latest version
Package readline-devel-6.0-4.el6.x86_64 already installed and latest version
Package ncurses-devel-5.7-3.20090208.el6.x86_64 already installed and latest version
Package libffi-devel-3.0.5-3.2.el6.x86_64 already installed and latest version
Nothing to do
.
.
make[2]: Entering directory `/usr/local/src/ruby-1.9.3-p484/ext/openssl'
compiling ossl_pkey.c
compiling ossl_ssl.c
compiling ossl_pkcs12.c
compiling ossl_bn.c
compiling ossl_hmac.c
ossl_hmac.c: In function âossl_hmac_copyâ:
ossl_hmac.c:90: warning: implicit declaration of function âHMAC_CTX_copyâ
compiling ossl_asn1.c
compiling ossl.c
compiling ossl_bio.c
compiling ossl_pkey_rsa.c
compiling ossl_ocsp.c
ossl_ocsp.c: In function âossl_ocspreq_add_certidâ:
ossl_ocsp.c:180: warning: function called through a non-compatible type
ossl_ocsp.c:180: note: if this code is reached, the program will abort
ossl_ocsp.c: In function âossl_ocspreq_get_certidâ:
ossl_ocsp.c:200: warning: function called through a non-compatible type
ossl_ocsp.c:200: note: if this code is reached, the program will abort
ossl_ocsp.c: In function âossl_ocspbres_get_statusâ:
ossl_ocsp.c:541: warning: function called through a non-compatible type
ossl_ocsp.c:541: note: if this code is reached, the program will abort
compiling ossl_pkey_dh.c
ossl_pkey_dh.c: In function âossl_dh_initializeâ:
ossl_pkey_dh.c:184: warning: function called through a non-compatible type
ossl_pkey_dh.c:184: note: if this code is reached, the program will abort
ossl_pkey_dh.c: In function âossl_dh_to_public_keyâ:
ossl_pkey_dh.c:372: warning: function called through a non-compatible type
ossl_pkey_dh.c:372: note: if this code is reached, the program will abort
compiling ossl_ns_spki.c
compiling ossl_x509attr.c
compiling ossl_x509name.c
ossl_x509name.c: In function âossl_x509name_hash_oldâ:
ossl_x509name.c:342: warning: implicit declaration of function âX509_NAME_hash_oldâ
compiling ossl_pkcs7.c
compiling ossl_pkey_ec.c
ossl_pkey_ec.c: In function âossl_ec_group_initializeâ:
ossl_pkey_ec.c:784: warning: function called through a non-compatible type
ossl_pkey_ec.c:784: note: if this code is reached, the program will abort
ossl_pkey_ec.c: In function âossl_ec_group_to_stringâ:
ossl_pkey_ec.c:1154: warning: function called through a non-compatible type
ossl_pkey_ec.c:1154: note: if this code is reached, the program will abort
compiling ossl_ssl_session.c
ossl_ssl_session.c: In function âossl_ssl_session_initializeâ:
ossl_ssl_session.c:53: warning: function called through a non-compatible type
ossl_ssl_session.c:53: note: if this code is reached, the program will abort
ossl_ssl_session.c:57: warning: function called through a non-compatible type
ossl_ssl_session.c:57: note: if this code is reached, the program will abort
ossl_ssl_session.c: In function âossl_ssl_session_to_pemâ:
ossl_ssl_session.c:251: warning: function called through a non-compatible type
ossl_ssl_session.c:251: note: if this code is reached, the program will abort
compiling openssl_missing.c
compiling ossl_x509.c
compiling ossl_x509cert.c
compiling ossl_digest.c
compiling ossl_pkcs5.c
ossl_pkcs5.c: In function âossl_pkcs5_pbkdf2_hmacâ:
ossl_pkcs5.c:39: warning: implicit declaration of function âPKCS5_PBKDF2_HMACâ
compiling ossl_rand.c
compiling ossl_engine.c
compiling ossl_x509crl.c
compiling ossl_cipher.c
ossl_cipher.c: In function âossl_cipher_copyâ:
ossl_cipher.c:143: warning: implicit declaration of function âEVP_CIPHER_CTX_copyâ
compiling ossl_x509ext.c
compiling ossl_config.c
compiling ossl_x509store.c
compiling ossl_x509revoked.c
compiling ossl_pkey_dsa.c
compiling ossl_x509req.c
linking shared-object openssl.so
installing default openssl libraries
make[2]: Leaving directory `/usr/local/src/ruby-1.9.3-p484/ext/openssl'
.
.
>>> Successfully installed ruby 1.9.3-p484 into /opt/rubies/ruby-1.9.3-p484

Note all the ossl warnings above.

Restart session.
Test system Ruby 1.8.7 openssl: Works.
Test ruby-install Ruby 1.9.3 openssl: Fails.

[root@dbatcit ~]# chruby
   ruby-1.9.3-p484
[root@dbatcit ~]# which ruby
/usr/bin/ruby
[root@dbatcit ~]# ruby -v
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
[root@dbatcit ~]# ruby -ropenssl -e "puts OpenSSL::VERSION"
1.0.0
[root@dbatcit ~]# chruby 1.9
[root@dbatcit ~]# chruby
 * ruby-1.9.3-p484
[root@dbatcit ~]# which ruby
/opt/rubies/ruby-1.9.3-p484/bin/ruby
[root@dbatcit ~]# ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]
[root@dbatcit ~]# ruby -ropenssl -e "puts OpenSSL::VERSION"
/opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/x86_64-linux/openssl.so: undefined symbol: EC_GROUP_new_curve_GF2m - /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/x86_64-linux/openssl.so (LoadError)
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/openssl.rb:17:in `<top (required)>'
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
[root@dbatcit ~]#


[root@dbatcit ~]# which -a openssl
/usr/bin/openssl
/usr/local/bin/openssl
[root@dbatcit ~]# openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
[root@dbatcit ~]# /usr/local/bin/openssl version
OpenSSL 0.9.8d 28 Sep 2006

Test RVM Ruby 1.9.3 openssl in user home: Works.

[mjerniga@dbatcit ~]$ ruby -v
ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]
[mjerniga@dbatcit ~]$ ruby -ropenssl -e "puts OpenSSL::VERSION"
1.1.0
[mjerniga@dbatcit ~]$ which openssl
/usr/local/bin/openssl
[mjerniga@dbatcit ~]$ openssl version
OpenSSL 0.9.8d 28 Sep 2006
[mjerniga@dbatcit ~]$ .rvm/usr/bin/openssl version
OpenSSL 1.0.1c 10 May 2012

[postmodern]

It would appear that ruby 2.1.0 requires a newer version of OpenSSL, than what RHEL 6.3 provides.

[juanitogan]

Perhaps, but that doesn't explain why I can install Ruby 1.9.3 with RVM but not with ruby-install. Ruby 1.9.3 is all I need at the moment. Ruby 2.1.0 was just a curiosity test. Every install of Ruby I have tried with ruby-install either fails or compiles ossl with many warnings that always leads to an openssl error:

[root@dbatcit ~]# ruby -ropenssl -e "puts OpenSSL::VERSION"
/opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/x86_64-linux/openssl.so: undefined symbol: EC_GROUP_new_curve_GF2m - /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/x86_64-linux/openssl.so (LoadError)
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/openssl.rb:17:in `<top (required)>'
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from /opt/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

[postmodern]

@juanitogan what is odd, is that the output of ruby-install shows 1.9.3 successfully compiling openssl; albeit with some warnings.

[juanitogan]

Yes, very odd. Not just warnings though, but also threats: note: if this code is reached, the program will abort... which is exactly what happens.

The RVM solution to openssl was to force a reinstall after initial install, such as:

\curl -L https://get.rvm.io | bash -s stable --ruby=1.9.3
rvm pkg install openssl
rvm reinstall 1.9.3 --with-openssl-dir=$rvm_path/usr

I'll keep digging as to why this was needed but similar options have not worked with ruby-install yet. I might put a similar question to Stack Overflow as well. I don't yet understand what files are needed by openssl and why. Why does RVM install its own version? Why does ruby-install seem to be installing/updating a version in /usr/bin as well as extra stuff in /opt/rubies? Etc.

[juanitogan]

I am also beginning to wonder about the value of yum here. On one hand it is nice that ruby-install uses yum to automatically install required packages. On the other hand, our users currently do not have sudo rights to yum (no idea if this is best practice or not) and therefore cannot execute ruby-install for local installs. Furthermore, it appears yum caused an unwanted upgrade from openssl 1.0.1c to 1.0.1e and I have no idea yet if this is part of the problem.

[postmodern]

If the users need to install additional software, you should either have root do it or give sudo access to them. You can also disable the installation of dependencies with the --no-install-deps option. If your distro does not have the required OpenSSL version in the package manager, you should submit a bug requesting a package update. You can also install ruby against a custom openssl install: ruby-install ruby 2.1.0 -- --with-openssl-dir=/where/openssl/was/installed.

[juanitogan]

Thanks for the help, I'll take a look at --no-install-deps. I tried various paths for --with-openssl-dir before my original post but the failures I was running into there made it apparent I wasn't approaching this correctly. Currently, I'm trying to make sense of the openssl files installed by RVM versus the files installed by ruby-install (as well as looking at mkmf.log) and hope to find some intel there.

I have to switch to another task for a week or so, so I may go quiet on this.

[juanitogan]

I am closing this because I have learned that this is not specific to ruby-install but, rather, applies to all installers. Still, others might need to know what I found so I will briefly describe it here.

After far too many hours of research, and learning far more about Linux than I ever cared to, I have narrowed the problem to basic peculiarities of RHEL and OpenSSL and an incorrect assumption made by Ruby (extconf.rb) during installation. The following sites gave me some good clues as to what to look at but I found nothing that put it all together like what I needed.

http://www.openssl.org/support/faq.html#BUILD8
http://wiki.openssl.org/index.php/Compilation_and_Installation#Fedora_and_Red_Hat
http://sachachua.com/blog/2011/04/setting-up-ruby-on-rails-on-a-redhat-enterprise-linux-rackspace-cloud-server/
https://web.archive.org/web/20130430124941/https://rvm.io/packages/openssl/

...and not much thanks to a whole bunch of red herrings involving Ruby patches and EC2M. Also, RVM needs to correct their optimism that they have accounted for this with autolibs and should reinstate their previous openssl page.

Basic rules

Rule 1

The install of OpenSSL (1.0.1e) created and maintained by yum in /usr/bin cannot be used to compile Ruby's OpenSSL extension correctly -- at least, not on my machine at this time with the latest versions of Ruby (1.9.3-p484, 2.0.0, 2.1.0). RHEL 6.3. I can only surmise that this is due to peculiarities in RedHat's compilation of OpenSSL as hinted at in the OpenSSL FAQ.

Rule 2

I found two old versions (0.9.8) of OpenSSL in /usr/local (in bin + openssl, and ssl/bin) and updating/replacing these got me a bit closer to a solution. For whatever reason, every manual install of OpenSSL 1.0.1f in /usr/local (regardless of bin,openssl,ssl directory arrangement) insisted on putting the libraries in /usr/local/lib64 instead of /usr/local/lib (unless I hacked the Makefile, of course). Ruby's expconf.rb script, however, assumes the OpenSSL libraries will always be in a lib directory. Chasing down this single annoyance (and clash with OpenSSL) was the hardest part of all this. Therefore, to make using an install of OpenSSL in /usr/local work, you must do two things: (1) install Ruby with the --with-openssl-dir switch, and (2) recompile Ruby's OpenSSL extension while also modifying the Makefile to point to lib64 instead of lib. Thus, run something like the following string of commands as root:

ruby-install ruby 1.9.3-p545 -- --with-openssl-dir=/usr/local
cd /usr/local/src/ruby-1.9.3-p545/ext/openssl
ruby extconf.rb

## edit Makefile to replace something like this:
libpath = . $(libdir) /usr/local/lib
LIBPATH =  -L. -L$(libdir) -Wl,-R$(libdir) -L/usr/local/lib -Wl,-R/usr/local/lib
## with something like this:
libpath = . $(libdir) /usr/local/lib64
LIBPATH =  -L. -L$(libdir) -Wl,-R$(libdir) -L/usr/local/lib64 -Wl,-R/usr/local/lib64
## save, and back to commands:

make
make install

The new Ruby install should now work with OpenSSL properly. As a quick check, I restart my sudo session and then (assuming using chruby):

chruby 1.9
ruby -ropenssl -e "puts OpenSSL::VERSION"

Rule 3

Installing OpenSSL anywhere besides /usr puts the libraries in the expected lib instead of lib64. (Don't ask me why... dunno.) This may be the more maintainable solution as it lets you avoid hacking up the Makefile. This is also the solution RVM uses when running rvm pkg install openssl. Thus, to install both OpenSSL and Ruby, you may run commands something like these (I run as sudo bash):

cd /opt/local
wget http://www.openssl.org/source/openssl-1.0.1f.tar.gz
tar -xzf openssl-1.0.1f.tar.gz
cd openssl-1.0.1f
./config --prefix=/opt/local shared no-asm zlib > openssl_config.log
make > openssl_make.log
make install > openssl_install.log

## optional, update openssl certs:
cd /opt/local/ssl
wget http://curl.haxx.se/ca/cacert.pem
mv cacert.pem cert.pem
cd /opt
## back to ruby:

ruby-install ruby 1.9.3-p545 -- --with-openssl-dir=/opt/local

The new Ruby install should now work with OpenSSL properly. As a quick check, I restart my sudo session and then (assuming using chruby):

chruby 1.9
ruby -ropenssl -e "puts OpenSSL::VERSION"

---

Perhaps I'll add more later as I refine our install procedure. This much, thus far, is the result of perhaps a hundred trials of Ruby and OpenSSL installs under various conditions and switches. If anyone has more answers, please share. Time to get back to real life.

[postmodern]

@juanitogan you should consider submitting a bug about openssl's extconf.rb to https://bugs.ruby-lang.org/