Rails: crash on DB access with custom OpenSSL config on Ubuntu 14.04

[ged]

Original report by Andrey Novikov (Bitbucket: Envek, GitHub: Envek).

---

It may be not a pg bug, but for me it occurs only with pg gem.

I've patched ruby with patches from Ruby #9830 and OpenSSL configured for GOST algorithms usage on Ubuntu 14.04 Server x64.

App crashes on DB rake tasks invocation or when first request coming to it, yielding next message and no backtrace:

envek@ubuntu:~/TestOsslFail$ rake db:create
GOST engine already loaded
Auto configuration failed
140060080711488:error:260B606D:engine routines:DYNAMIC_LOAD:init failed:eng_dyn.c:521:
140060080711488:error:260BC066:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:204:section=gost_section, name=dynamic_path, value=/usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so
140060080711488:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:235:module=engines, value=engine_section, retcode=-1

This appears ONLY on fresh Ubuntu (13.10 and 14.04). Everything is OK on Ubuntu 12.04 and Mac OS X with OpenSSL from Homebrew.

How to reproduce

Take a machine with Ubuntu 14.04.

Configure OpenSSL

Insert next line at very beginning of /etc/ssl/openssl.cnf file:

openssl_conf = openssl_def

Insert next lines at very end of this file:

[openssl_def]
engines = engine_section

[engine_section]
gost = gost_section

[gost_section]
engine_id = gost
dynamic_path = /usr/lib/x86_64-linux-gnu/openssl-1.0.0/engines/libgost.so
default_algorithms = ALL
CRYPT_PARAMS = id-Gost28147-89-CryptoPro-A-ParamSet

Check configuration: execute openssl ciphers | tr ":" "\n" | grep GOST, it should return something.

See official manual in OpenSSL sources for details: https://github.com/openssl/openssl/blob/master/engines/ccgost/README.gost

Install Ruby (example with RVM)

Crash occurs only in patched Ruby, because first patch enables Ruby to take openssl.cnf in account.

rvm install ruby-2.1.2-gost --patch https://bugs.ruby-lang.org/attachments/download/4420/respect_system_openssl_settings.patch --patch https://bugs.ruby-lang.org/attachments/download/4415/gost_keys_support_draft.patch

Patch for Rbenv (ruby-build): https://gist.github.com/Envek/82be109c58a0a565d382

Install Rails and create sample app

gem install rails --no-rdoc --no-ri
rails new pg_ossl_boom -d postgresql
cd pg_ossl_boom
echo "gem 'therubyracer',  platforms: :ruby" >> Gemfile
bundle

Setup Rails to connect to DB over HTTP

Add host: localhost to the default section.

Enjoy!

rake db:create

Questions

1. Why pg (or libpq) ever tries to load GOST engine? Can it be disabled?
2. Can SSL be disabled? (I don't need it)
3. Any other workarounds? Where it is better to send this bug report?
4. At least, how to get backtrace?

Additional information

I'm freely can load OpenSSL engine from irb or rails console, like this (it works):

require 'openssl'
OpenSSL::Engine.load
@gost_engine = OpenSSL::Engine.by_id('gost')
@gost_engine.set_default(0xFFFF)
@gost_engine.digest('md_gost94') # => <OpenSSL::Digest>

Also, all GOST-related stuff (digital signing and so on) works well (examples can be found at Ruby #9830).

I've found some issue that may be related to this in OpenSSL here: http://rt.openssl.org/Ticket/Display.html?id=3116&user=guest&pass=guest (may it help)

Tested with both 0.17.1 and 0.18.0.pre20140820094244. PostgreSQL (and all libs) version: 9.3 and 9.4 beta 2.

Thank you very much for attention, please help, I'm really lack of understanding what is going on.

[ged]

Original comment by Andrey Novikov (Bitbucket: Envek, GitHub: Envek).

---

Added info about tested versions and patched rubies.

[ged]

Original comment by Michael Granger (Bitbucket: ged, GitHub: ged).

---

Thanks for the fantastic report. I don't have time to reproduce this now, but I should be able to look at it and the end of next week.

[ged]

Original comment by Andrey Novikov (Bitbucket: Envek, GitHub: Envek).

---

I've found two workarounds for this:

1. Do not use TCP connection, use Unix socket

2. Install custom OpenSSL and build Ruby against it, for example for RVM:

   rvm pkg install openssl         # will install OpenSSL 1.0.1i to ~/.rvm/usr
   vim ~/.rvm/usr/ssl/openssl.cnf # TODO: Apply GOST enabling configurations here
   rvm reinstall ruby-2.1.2-gost --with-openssl-dir=$HOME/.rvm/usr --patch https://bugs.ruby-lang.org/attachments/download/4420/respect_system_openssl_settings.patch --patch https://bugs.ruby-lang.org/attachments/download/4415/gost_keys_support_draft.patch
   

[ged]

Original comment by Brian Stevens (Bitbucket: othatbrian, GitHub: othatbrian).

---

Andrey, I was able to reproduce this issue and I'm [slowly] working my way through it.

The pg gem uses libpq underneath. The documentation on libpq-connect mentions sslmode in the available options. sslmode can be one of

• disable
• allow
• prefer
• require
• verify-ca
• verify-full

prefer is the default. One work around is to set it to disable or allow; that will prevent the connection from attempting to use SSL (unless explicitly told to do so, in the case of allow). That can be done by adding sslmode: 'disable' in your databases.yml.

That answers your question # 2.

As far as question # 1: GOST pg/libpq is loading the GOST engine because, by default (as discussed above) libpq will attempt to use SSL, and your SSL configuration is set to use the GOST engine.

"Can it be disabled?" Well, attempting to connect via SSL can be disabled or the GOST could be disabled by changing your SSL configuration file.

The main remaining questions is: why, if the GOST engine is enabled, does this connection fail? I have not yet figured that part out.

[ged]

Original comment by Andrey Novikov (Bitbucket: Envek, GitHub: Envek).

---

@othatbrian, thank you very much!

As for question #1 I probably meant: Can libpq check in libpq-connect
for OpenSSL engines already loaded and don't try to load them one more
time? Probably I should open a ticket somewhere near to libpq
development :-)

Also, why there is no error if I compile Ruby against separate
OpenSSL? Is that means that Ruby code and pg/libpq configure and use
two different OpenSSLs at the same time?

And the last thing: I forced to use GOST cryptoalgorithms in my app
(for digital signatures and TLS between my app and another system), so
there is no choice for me, I just need it.

Thank you again for taking your time and providing helpful info. Ask
for any assistance I can provide to resolve this.

[ged]

Original comment by Brian Stevens (Bitbucket: othatbrian, GitHub: othatbrian).

---

As I received your reply, I was writing another comment that it appears to me to be a problem with "double initialization" of the SSL libraries, as mentioned here: http://www.postgresql.org/docs/9.3/static/libpq-ssl.html#LIBPQ-SSL-INITIALIZE

I tested with a two simple scripts:

#!ruby
require 'pg'

PG.connect( host: 'localhost', dbname: 'postgres', password: 'foo' )

and

#!ruby
require 'ssl'
require 'pg'

PG.connect( host: 'localhost', dbname: 'postgres', password: 'foo' )

The first one works; the second crashes exactly as you mentioned.

I assume Rails is loading the ssl library before the database connection is attempted, thus creating the problem. I'm not sure if there's a was for PG to query if SSL has already been initialized, or whether PG needs to implement the PQinitOpenSSL() call so that Ruby can call that.

[ged]

Original comment by Brian Stevens (Bitbucket: othatbrian, GitHub: othatbrian).

---

I can't tell whether it's more an error in PG or in libpq.

One option would be to implement libpq's PQinitOpenSSL in PG. I tried for several hours yesterday to do so, but was never successful.

With that functionality, a user of PG could do something like:

#!ruby
PG.initOpenSSL false

if their application has already initiated OpenSSL

or

#!ruby
PG.initOpenSSL defined?(OpenSSL) ? false : true

if they're unsure.

As libpq is written, the default is effectively 'true'.

[ged]

Original comment by Lars Kanis (Bitbucket: larskanis, GitHub: larskanis).

---

The two initialize functions are now in commit: https://bitbucket.org/ged/ruby-pg/commits/45610159a2
Could you please try, whether this addition can help to solve this issue? I just pushed another pre-release to rubygems.org.

[ged]

Original comment by Andrey Novikov (Bitbucket: Envek, GitHub: Envek).

---

Thank you very much!

I've installed version 0.18.0.pre20141117110243 and placed defined?(OpenSSL) ? PG.init_openssl(false, false) : PG.init_openssl(true, true) into initializer and Rakefile. And problem is gone!

Currently tested only in testing VM, have no time for more testing now.

[ged]

Original comment by Lars Kanis (Bitbucket: larskanis, GitHub: larskanis).

---

pg-0.18.0 is released now and it adds the openssl_init() function, which can be used to avoid double initialization.