New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

uninitialized constant OpenSSL::KDF #180

Closed
Jehops opened this Issue Dec 17, 2017 · 27 comments

Comments

Projects
None yet
4 participants
@Jehops

Jehops commented Dec 17, 2017

After upgrading to version 2.1.0, we are seeing the issue described in tootsuite/mastodon#6059. Downgrading back to 2.0.6 fixes the issue.

@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 17, 2017

Member

From tootsuite/mastodon#6059:

Hi,

jrm (FreeBSD packager) reported on IRC that after updating to Mastodon v2.1.0, Sidekiq and Puma raise the following error:

uninitialized constant OpenSSL::KDF
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac'
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:19:in `pbkdf2_hmac_sha1'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:21:in `generate_key'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:36:in `generate_key'
/usr/local/lib/ruby/gems/2.3/gems/globalid-0.4.1/lib/global_id/railtie.rb:26:in `block (2 levels) in <class:Railtie>'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
/usr/local/lib/ruby/2.3/tsort.rb:228:in `block in tsort_each'
/usr/local/lib/ruby/2.3/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.3/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `each'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `call'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:226:in `tsort_each'
/usr/local/lib/ruby/2.3/tsort.rb:205:in `tsort_each'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
/usr/local/www/mastodon/config/environment.rb:5:in `<top (required)>'
/usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:257:in `boot_system'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:54:in `run'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/bin/sidekiq:12:in `<top (required)>'
/usr/local/bin/sidekiq:23:in `load'
/usr/local/bin/sidekiq:23:in `<main>'

I believe this is because OpenSSL::KDF was added in Ruby 2.4.

Correction: it's new in openssl gem v2.1.0 which was released a few days ago.

This is still weird because the extension library (openssl.so), which should be loaded by require "openssl", must have defined OpenSSL::KDF module. Could you check what is inside $LOADED_FEATURES?

puts $LOADED_FEATURES.grep(/openssl/)
Member

rhenium commented Dec 17, 2017

From tootsuite/mastodon#6059:

Hi,

jrm (FreeBSD packager) reported on IRC that after updating to Mastodon v2.1.0, Sidekiq and Puma raise the following error:

uninitialized constant OpenSSL::KDF
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac'
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:19:in `pbkdf2_hmac_sha1'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:21:in `generate_key'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:36:in `generate_key'
/usr/local/lib/ruby/gems/2.3/gems/globalid-0.4.1/lib/global_id/railtie.rb:26:in `block (2 levels) in <class:Railtie>'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
/usr/local/lib/ruby/2.3/tsort.rb:228:in `block in tsort_each'
/usr/local/lib/ruby/2.3/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.3/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `each'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `call'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:226:in `tsort_each'
/usr/local/lib/ruby/2.3/tsort.rb:205:in `tsort_each'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
/usr/local/www/mastodon/config/environment.rb:5:in `<top (required)>'
/usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:257:in `boot_system'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:54:in `run'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/bin/sidekiq:12:in `<top (required)>'
/usr/local/bin/sidekiq:23:in `load'
/usr/local/bin/sidekiq:23:in `<main>'

I believe this is because OpenSSL::KDF was added in Ruby 2.4.

Correction: it's new in openssl gem v2.1.0 which was released a few days ago.

This is still weird because the extension library (openssl.so), which should be loaded by require "openssl", must have defined OpenSSL::KDF module. Could you check what is inside $LOADED_FEATURES?

puts $LOADED_FEATURES.grep(/openssl/)
@Jehops

This comment has been minimized.

Show comment
Hide comment
@Jehops

Jehops Dec 17, 2017

$ irb
irb(main):001:0> puts $LOADED_FEATURES.grep(/openssl/)
=> nil
irb(main):002:0>

Jehops commented Dec 17, 2017

$ irb
irb(main):001:0> puts $LOADED_FEATURES.grep(/openssl/)
=> nil
irb(main):002:0>
@Jehops

This comment has been minimized.

Show comment
Hide comment
@Jehops

Jehops Dec 17, 2017

Sorry, I think/hope you meant this.

$ irb
irb(main):001:0> require "openssl"
=> true
irb(main):002:0> puts $LOADED_FEATURES.grep(/openssl/)
/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
=> nil
irb(main):003:0>

Jehops commented Dec 17, 2017

Sorry, I think/hope you meant this.

$ irb
irb(main):001:0> require "openssl"
=> true
irb(main):002:0> puts $LOADED_FEATURES.grep(/openssl/)
/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
=> nil
irb(main):003:0>

Jehops added a commit to Jehops/freebsd-ports that referenced this issue Dec 17, 2017

Jehops pushed a commit to Jehops/freebsd-ports that referenced this issue Dec 17, 2017

jrm
Copy security/rubygem-openssl to create new port,
security/rubygem-openssl20

net-im/mastodon fails to run with newer versions of rubygem-openssl:
ruby/openssl#180 for details.


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@456565 35697150-7ecd-e111-bb59-0022644237b5

uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Dec 17, 2017

Copy security/rubygem-openssl to create new port,
security/rubygem-openssl20

net-im/mastodon fails to run with newer versions of rubygem-openssl:
ruby/openssl#180 for details.
@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 18, 2017

Member

Sorry that I wasn't clear enough. Please inject that into the code where the problem exists, say, into /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb, just before line 14.

I suspect the openssl.so from Ruby 2.3 and lib/openssl/pkcs5.rb from openssl gem v2.1.0 are loaded at the same time.

Member

rhenium commented Dec 18, 2017

Sorry that I wasn't clear enough. Please inject that into the code where the problem exists, say, into /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb, just before line 14.

I suspect the openssl.so from Ruby 2.3 and lib/openssl/pkcs5.rb from openssl gem v2.1.0 are loaded at the same time.

mat813 pushed a commit to mat813/freebsd-ports that referenced this issue Dec 18, 2017

jrm
Copy security/rubygem-openssl to create new port,
security/rubygem-openssl20

net-im/mastodon fails to run with newer versions of rubygem-openssl:
ruby/openssl#180 for details.


git-svn-id: https://svn.freebsd.org/ports/head@456565 35697150-7ecd-e111-bb59-0022644237b5
@Jehops

This comment has been minimized.

Show comment
Hide comment
@Jehops

Jehops Dec 19, 2017

Here it is the output with the line added.

[71269] * Version 3.11.0 (ruby 2.3.6-p384), codename: Love Song
[71269] * Min threads: 5, max threads: 5
[71269] * Environment: production
[71269] * Process workers: 2
[71269] * Preloading application
/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb
[71269] ! Unable to load application: NameError: uninitialized constant OpenSSL::KDF
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:15:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
        from /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:20:in `pbkdf2_hmac_sha1'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:21:in `generate_key'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:36:in `generate_key'
        from /usr/local/lib/ruby/gems/2.3/gems/globalid-0.4.1/lib/global_id/railtie.rb:26:in `block (2 levels) in <class:Railtie>'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
        from /usr/local/lib/ruby/2.3/tsort.rb:228:in `block in tsort_each'
        from /usr/local/lib/ruby/2.3/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:431:in `each_strongly_connected_component_from'
        from /usr/local/lib/ruby/2.3/tsort.rb:349:in `block in each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `each'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `call'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:226:in `tsort_each'
        from /usr/local/lib/ruby/2.3/tsort.rb:205:in `tsort_each'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
        from /usr/local/www/mastodon/config/environment.rb:5:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
        from config.ru:4:in `block in <main>'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'
        from config.ru:in `new'
        from config.ru:in `<main>'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:49:in `eval'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:49:in `new_from_string'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:40:in `parse_file'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/configuration.rb:318:in `load_rackup'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/configuration.rb:243:in `app'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/runner.rb:138:in `load_and_bind'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/cluster.rb:397:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/launcher.rb:183:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/cli.rb:77:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/bin/puma:10:in `<top (required)>'
        from /usr/local/bin/puma:23:in `load'
        from /usr/local/bin/puma:23:in `<main>'

Jehops commented Dec 19, 2017

Here it is the output with the line added.

[71269] * Version 3.11.0 (ruby 2.3.6-p384), codename: Love Song
[71269] * Min threads: 5, max threads: 5
[71269] * Environment: production
[71269] * Process workers: 2
[71269] * Preloading application
/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb
[71269] ! Unable to load application: NameError: uninitialized constant OpenSSL::KDF
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:15:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
        from /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:20:in `pbkdf2_hmac_sha1'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:21:in `generate_key'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:36:in `generate_key'
        from /usr/local/lib/ruby/gems/2.3/gems/globalid-0.4.1/lib/global_id/railtie.rb:26:in `block (2 levels) in <class:Railtie>'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
        from /usr/local/lib/ruby/2.3/tsort.rb:228:in `block in tsort_each'
        from /usr/local/lib/ruby/2.3/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:431:in `each_strongly_connected_component_from'
        from /usr/local/lib/ruby/2.3/tsort.rb:349:in `block in each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `each'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `call'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:226:in `tsort_each'
        from /usr/local/lib/ruby/2.3/tsort.rb:205:in `tsort_each'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
        from /usr/local/www/mastodon/config/environment.rb:5:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
        from config.ru:4:in `block in <main>'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'
        from config.ru:in `new'
        from config.ru:in `<main>'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:49:in `eval'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:49:in `new_from_string'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:40:in `parse_file'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/configuration.rb:318:in `load_rackup'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/configuration.rb:243:in `app'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/runner.rb:138:in `load_and_bind'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/cluster.rb:397:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/launcher.rb:183:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/cli.rb:77:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/bin/puma:10:in `<top (required)>'
        from /usr/local/bin/puma:23:in `load'
        from /usr/local/bin/puma:23:in `<main>'
@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 20, 2017

Member

/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb

Well, this is strange -- since (one of the dependencies of) mastodon has openssl as its runtime dependency, only files from the gem should have been loaded. This implies something is doing require "openssl" before Bundler activates the gem (with bundle exec or Bundler.setup), but I I wasn't able to reproduce on Travis CI.

https://travis-ci.org/rhenium/mastodon/jobs/318783772

Member

rhenium commented Dec 20, 2017

/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb

Well, this is strange -- since (one of the dependencies of) mastodon has openssl as its runtime dependency, only files from the gem should have been loaded. This implies something is doing require "openssl" before Bundler activates the gem (with bundle exec or Bundler.setup), but I I wasn't able to reproduce on Travis CI.

https://travis-ci.org/rhenium/mastodon/jobs/318783772

trombik pushed a commit to trombik/freebsd-ports that referenced this issue Dec 22, 2017

Copy security/rubygem-openssl to create new port,
security/rubygem-openssl20

net-im/mastodon fails to run with newer versions of rubygem-openssl:
ruby/openssl#180 for details.

trombik pushed a commit to trombik/freebsd-ports that referenced this issue Dec 23, 2017

Copy security/rubygem-openssl to create new port,
security/rubygem-openssl20

net-im/mastodon fails to run with newer versions of rubygem-openssl:
ruby/openssl#180 for details.
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 27, 2017

I have just replaced rubygem-openssl 2.0.6 with rubygem-openssl 2.1.0 in my existing mastodon 2.0.0_3 setup and that was enough to reproduce the issue.

It seems that puma does require 'openssl' and it does not care whether it's this gem or anything else.

Adding gem 'openssl' as the first line there fixes the error, but this is certainly no solution; and also it raises the question why did it work with rubygem-openssl 2.0.6 ? Why does it not complain about undefined OpenSSL::PKCS5 with 2.0.6?

saper commented Dec 27, 2017

I have just replaced rubygem-openssl 2.0.6 with rubygem-openssl 2.1.0 in my existing mastodon 2.0.0_3 setup and that was enough to reproduce the issue.

It seems that puma does require 'openssl' and it does not care whether it's this gem or anything else.

Adding gem 'openssl' as the first line there fixes the error, but this is certainly no solution; and also it raises the question why did it work with rubygem-openssl 2.0.6 ? Why does it not complain about undefined OpenSSL::PKCS5 with 2.0.6?

@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 27, 2017

I just noticed that

OpenSSL::PKCS5.pbkdf2_hmac
OpenSSL::PKCS5.pbkdf2_hmac_sha1

are defined in the /usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so module shipped with ruby. So rubygem-openssl 2.1.0 tries to override them in Ruby but there is no guarantee that openssl.so provided via the gem is loaded.

This is very messy and is a consequence of a namespace clash between the built-in module and this gem.

saper commented Dec 27, 2017

I just noticed that

OpenSSL::PKCS5.pbkdf2_hmac
OpenSSL::PKCS5.pbkdf2_hmac_sha1

are defined in the /usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so module shipped with ruby. So rubygem-openssl 2.1.0 tries to override them in Ruby but there is no guarantee that openssl.so provided via the gem is loaded.

This is very messy and is a consequence of a namespace clash between the built-in module and this gem.

saper added a commit to saper/openssl that referenced this issue Dec 27, 2017

uninitialized constant OpenSSL::KDF if stock openssl.so is loaded
If Ruby's standard 'openssl' module is loaded before this
gem gets activated, strange things will happen. For example,
an attempt to shadow OpenSSL::PKCS5 functions (that would otherwise
work) fails with the "uninitialized constant OpenSSL::KDF" message.

Reproduction case for: ruby#180
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 27, 2017

To reproduce this problem, try my test_kdf_repr branch of this project.

What is interesting, test_pkcs5.rb from v2.0.6 is fully working with a stock openssl.so

@Jehops we are also running into interesting FreeBSD-related problems additionally. Ruby links to the base system OpenSSL and the port will use one from ports if installed:

/usr/local/bin/ruby24 -I./lib -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
OpenSSL 1.0.2n  7 Dec 2017
OpenSSL 1.0.2n  7 Dec 2017
Stock Ruby module:
/usr/local/bin/ruby24 -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
OpenSSL 1.0.1s-freebsd  1 Mar 2016
OpenSSL 1.0.1u-freebsd  22 Sep 2016

saper commented Dec 27, 2017

To reproduce this problem, try my test_kdf_repr branch of this project.

What is interesting, test_pkcs5.rb from v2.0.6 is fully working with a stock openssl.so

@Jehops we are also running into interesting FreeBSD-related problems additionally. Ruby links to the base system OpenSSL and the port will use one from ports if installed:

/usr/local/bin/ruby24 -I./lib -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
OpenSSL 1.0.2n  7 Dec 2017
OpenSSL 1.0.2n  7 Dec 2017
Stock Ruby module:
/usr/local/bin/ruby24 -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
OpenSSL 1.0.1s-freebsd  1 Mar 2016
OpenSSL 1.0.1u-freebsd  22 Sep 2016
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 27, 2017

Running a whole test suite against the stock openssl.so (not the one from the gem) gives interesting results, including crashes:

  • test_pkey_ec.rb: NoMethodError: undefined method 'to_octet_string' for #OpenSSL::PKey::EC::Point:0x00000008038ead18
  • test_pkey_rsa.rb: NoMethodError: undefined method 'sign_pss' for #OpenSSL::PKey::RSA:0x0000000804f089d8
  • test_ssl.rb: NoMethodError: undefined method `add_certificate' for #OpenSSL::SSL::SSLContext:0x0000000804daa9d8
  • test_ssl.rb: NameError: uninitialized constant OpenSSL::SSL::SSL3_VERSION
  • test_ssl.rb and test_ssl_session.rb: TLS 1.2 is not supported (this is technically not true)
  • test_asn1.rb: NoMethodError: undefined method `indefinite_length' for #OpenSSL::ASN1::ASN1Data:0x0000000804ee2a30
  • test_asn1.rb: OpenSSL::ASN1::ASN1Error: Constructive shall only be used with infinite length
  • test_asn1.rb: OpenSSL::ASN1::ASN1Error: universal tag for OpenSSL::ASN1::Primitive not found
  • test_asn1.rb: NoMethodError: undefined method `indefinite_length' for #OpenSSL::ASN1::BitString:0x0000000804c219b8 (also for other ASN1 subtypes)
  • test_bn.rb: NoMethodError: undefined method `negative?' for #OpenSSL::BN:0x0000000804da2e90
  • test_bn.rb: NoMethodError: undefined method `+@' for #OpenSSL::BN:0x00000008033f46d8
  • test_cipher.rb: ruby crashes
  • test_kdf.rb: NameError: uninitialized constant OpenSSL::KDF (this bug, fix: #181)
  • test_ocsp.rb: NoMethodError: undefined method `signed?' for #OpenSSL::OCSP::Request:0x00000008095d8ef8
  • test_pair.rb: ArgumentError: invalid byte sequence in UTF-8
  • test_pair.rb: ArgumentError: wrong number of arguments (given 2, expected 1)
  • test_x509attr.rb: boolean comparison failure
  • test_x509crl.rb: boolean comparison failure
  • test_x509ext.rb: boolean comparison failure
  • test_x509cert.rb: boolean comparison failure
  • test_x509name.rb: TypeError: no implicit conversion of Hash into Integer
  • test_x509name.rb: NoMethodError: undefined method `to_utf8' for #OpenSSL::X509::Name:0x0000000803913f60
  • test_x509req.rb: boolean comparison failure

I am not sure that having two binary modules of the same name is a good idea and two different require "openssl" APIs. Maybe this gem could detect that required binary symbols are not present and quickly back off?

Full log (except for crash): https://gist.github.com/saper/f96b27e3f2fb14ccf4ada382d8d6a6b7 (running saper@ed48775).

saper commented Dec 27, 2017

Running a whole test suite against the stock openssl.so (not the one from the gem) gives interesting results, including crashes:

  • test_pkey_ec.rb: NoMethodError: undefined method 'to_octet_string' for #OpenSSL::PKey::EC::Point:0x00000008038ead18
  • test_pkey_rsa.rb: NoMethodError: undefined method 'sign_pss' for #OpenSSL::PKey::RSA:0x0000000804f089d8
  • test_ssl.rb: NoMethodError: undefined method `add_certificate' for #OpenSSL::SSL::SSLContext:0x0000000804daa9d8
  • test_ssl.rb: NameError: uninitialized constant OpenSSL::SSL::SSL3_VERSION
  • test_ssl.rb and test_ssl_session.rb: TLS 1.2 is not supported (this is technically not true)
  • test_asn1.rb: NoMethodError: undefined method `indefinite_length' for #OpenSSL::ASN1::ASN1Data:0x0000000804ee2a30
  • test_asn1.rb: OpenSSL::ASN1::ASN1Error: Constructive shall only be used with infinite length
  • test_asn1.rb: OpenSSL::ASN1::ASN1Error: universal tag for OpenSSL::ASN1::Primitive not found
  • test_asn1.rb: NoMethodError: undefined method `indefinite_length' for #OpenSSL::ASN1::BitString:0x0000000804c219b8 (also for other ASN1 subtypes)
  • test_bn.rb: NoMethodError: undefined method `negative?' for #OpenSSL::BN:0x0000000804da2e90
  • test_bn.rb: NoMethodError: undefined method `+@' for #OpenSSL::BN:0x00000008033f46d8
  • test_cipher.rb: ruby crashes
  • test_kdf.rb: NameError: uninitialized constant OpenSSL::KDF (this bug, fix: #181)
  • test_ocsp.rb: NoMethodError: undefined method `signed?' for #OpenSSL::OCSP::Request:0x00000008095d8ef8
  • test_pair.rb: ArgumentError: invalid byte sequence in UTF-8
  • test_pair.rb: ArgumentError: wrong number of arguments (given 2, expected 1)
  • test_x509attr.rb: boolean comparison failure
  • test_x509crl.rb: boolean comparison failure
  • test_x509ext.rb: boolean comparison failure
  • test_x509cert.rb: boolean comparison failure
  • test_x509name.rb: TypeError: no implicit conversion of Hash into Integer
  • test_x509name.rb: NoMethodError: undefined method `to_utf8' for #OpenSSL::X509::Name:0x0000000803913f60
  • test_x509req.rb: boolean comparison failure

I am not sure that having two binary modules of the same name is a good idea and two different require "openssl" APIs. Maybe this gem could detect that required binary symbols are not present and quickly back off?

Full log (except for crash): https://gist.github.com/saper/f96b27e3f2fb14ccf4ada382d8d6a6b7 (running saper@ed48775).

saper added a commit to saper/openssl that referenced this issue Dec 27, 2017

Fix #180: Do not overwrite OpenSSL::PKCS5 if KDF is not loaded
Let applications not knowing this gem to work with the openssl.so
API included with Ruby
@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 28, 2017

Member

I have just replaced rubygem-openssl 2.0.6 with rubygem-openssl 2.1.0 in my existing mastodon 2.0.0_3 setup and that was enough to reproduce the issue.

It seems that puma does require 'openssl' and it does not care whether it's this gem or anything else.

Adding gem 'openssl' as the first line there fixes the error, but this is certainly no solution; and also it raises the question why did it work with rubygem-openssl 2.0.6 ? Why does it not complain about undefined OpenSSL::PKCS5 with 2.0.6?

Does adding gem "openssl" to the code work? As mastodon has openssl in its Gemfile, require "openssl" called in Bundler environment should behave as if there were gem "openssl". It has become apparent Bundler has some trouble activating the gem, but I have no idea what's happening -- why don't I reproduce on Travis CI?

I guess you did not see the error with 2.0.6 just because the difference between 2.0.6 and the stdlib one of Ruby 2.4 (= 2.0.x) is so small.

I just noticed that

OpenSSL::PKCS5.pbkdf2_hmac
OpenSSL::PKCS5.pbkdf2_hmac_sha1

are defined in the /usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so module shipped with ruby. So rubygem-openssl 2.1.0 tries to override them in Ruby but there is no guarantee that openssl.so provided via the gem is loaded.

This is very messy and is a consequence of a namespace clash between the built-in module and this gem.

As openssl is a "default gem" in Ruby 2.4 onwards, require "openssl" behaves differently depending on how Bundler is used. But, in any case, files from the gem and from the stdlib one will never be mixed. This is not special to openssl, but applies to any other default gem such as json (in Ruby >= 1.9.3). See also [Feature #5481].

Bundler is not used
The latest version of the gem installed is loaded. (The gemspec is activated.)
Bundle is used, and Gemfile includes openssl
The specified version of the gem is loaded. (The gemspec is activated.)
Bundler is used, but Gemfile does not include openssl
The stdlib one is loaded. (The gemspec is not activated.)

In previous versions of Ruby, say Ruby 2.3, openssl was not a "default gem". So:

Bundler is not used
The stdlib one is loaded.
Bundle is used, and Gemfile includes openssl
The specified version of the gem is loaded. The stdlib one will be overridden and not loaded.
Bundler is used, but Gemfile does not include openssl
The stdlib one is loaded.
Member

rhenium commented Dec 28, 2017

I have just replaced rubygem-openssl 2.0.6 with rubygem-openssl 2.1.0 in my existing mastodon 2.0.0_3 setup and that was enough to reproduce the issue.

It seems that puma does require 'openssl' and it does not care whether it's this gem or anything else.

Adding gem 'openssl' as the first line there fixes the error, but this is certainly no solution; and also it raises the question why did it work with rubygem-openssl 2.0.6 ? Why does it not complain about undefined OpenSSL::PKCS5 with 2.0.6?

Does adding gem "openssl" to the code work? As mastodon has openssl in its Gemfile, require "openssl" called in Bundler environment should behave as if there were gem "openssl". It has become apparent Bundler has some trouble activating the gem, but I have no idea what's happening -- why don't I reproduce on Travis CI?

I guess you did not see the error with 2.0.6 just because the difference between 2.0.6 and the stdlib one of Ruby 2.4 (= 2.0.x) is so small.

I just noticed that

OpenSSL::PKCS5.pbkdf2_hmac
OpenSSL::PKCS5.pbkdf2_hmac_sha1

are defined in the /usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so module shipped with ruby. So rubygem-openssl 2.1.0 tries to override them in Ruby but there is no guarantee that openssl.so provided via the gem is loaded.

This is very messy and is a consequence of a namespace clash between the built-in module and this gem.

As openssl is a "default gem" in Ruby 2.4 onwards, require "openssl" behaves differently depending on how Bundler is used. But, in any case, files from the gem and from the stdlib one will never be mixed. This is not special to openssl, but applies to any other default gem such as json (in Ruby >= 1.9.3). See also [Feature #5481].

Bundler is not used
The latest version of the gem installed is loaded. (The gemspec is activated.)
Bundle is used, and Gemfile includes openssl
The specified version of the gem is loaded. (The gemspec is activated.)
Bundler is used, but Gemfile does not include openssl
The stdlib one is loaded. (The gemspec is not activated.)

In previous versions of Ruby, say Ruby 2.3, openssl was not a "default gem". So:

Bundler is not used
The stdlib one is loaded.
Bundle is used, and Gemfile includes openssl
The specified version of the gem is loaded. The stdlib one will be overridden and not loaded.
Bundler is used, but Gemfile does not include openssl
The stdlib one is loaded.
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

@Jehops do you know why we need openssl dependency to be added explicitly? ostatus2 and rails stuff should run on the stock openssl.so module as well...

saper commented Dec 28, 2017

@Jehops do you know why we need openssl dependency to be added explicitly? ostatus2 and rails stuff should run on the stock openssl.so module as well...

@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

@rhenium

But, in any case, files from the gem and from the stdlib one will never be mixed.

So how would you explain the behaviour we are seeing? Is this true also when the stdlib binary module has been loaded into interpreter's memory?

saper commented Dec 28, 2017

@rhenium

But, in any case, files from the gem and from the stdlib one will never be mixed.

So how would you explain the behaviour we are seeing? Is this true also when the stdlib binary module has been loaded into interpreter's memory?

@Jehops

This comment has been minimized.

Show comment
Hide comment
@Jehops

Jehops Dec 28, 2017

@saper, the FreeBSD ostatus2 port/package matches what ostatus2 gem calls for: https://rubygems.org/gems/ostatus2/versions/2.0.1.

It may be worth noting that bundler is not used with the FreeBSD Mastodon package.

Jehops commented Dec 28, 2017

@saper, the FreeBSD ostatus2 port/package matches what ostatus2 gem calls for: https://rubygems.org/gems/ostatus2/versions/2.0.1.

It may be worth noting that bundler is not used with the FreeBSD Mastodon package.

@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

Thanks, I can see that this was done as a quick fix for openssl.so incompatibility: tootsuite/ostatus2#2

saper commented Dec 28, 2017

Thanks, I can see that this was done as a quick fix for openssl.so incompatibility: tootsuite/ostatus2#2

@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

Here's what I think happens:

  • puma starts and requires openssl, which is loaded from stdlib location /usr/local/lib/ruby/2.3/openssl.rb. (Bundler is not used)
  • puma tries to initialize application. $LOAD_PATH is set to
["/usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib",
 "/usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/puma-3.11.0",
 "/usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib",
 "/usr/local/lib/ruby/site_ruby/2.3",
 "/usr/local/lib/ruby/site_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/site_ruby",
 "/usr/local/lib/ruby/vendor_ruby/2.3",
 "/usr/local/lib/ruby/vendor_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/vendor_ruby",
 "/usr/local/lib/ruby/2.3",
 "/usr/local/lib/ruby/2.3/amd64-freebsd10"]
  • mastodon requires bundler/setup in config/boot.rb - @Jehops this is where Gemfile.lock gets re-generated btw.
  • After this the $LOAD_PATH changes:
["/usr/local/lib/ruby/gems/2.3/gems/bundler-1.16.0/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/webpush-0.3.2/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/webpacker-3.0.2/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/uglifier-3.2.0/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/tzinfo-data-1.2017.3/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/twitter-text-1.14.7/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/strong_migrations-0.1.9/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/simple_form-3.5.0/lib",
 ...
 "/usr/local/lib/ruby/gems/2.3/gems/ostatus2-2.0.1/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib",
 "/usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/openssl-2.1.0",
 ...
 "/usr/local/lib/ruby/gems/2.3/gems/rake-12.3.0/lib",
 "/usr/local/lib/ruby/site_ruby/2.3",
 "/usr/local/lib/ruby/site_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/site_ruby",
 "/usr/local/lib/ruby/vendor_ruby/2.3",
 "/usr/local/lib/ruby/vendor_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/vendor_ruby",
 "/usr/local/lib/ruby/2.3",
 "/usr/local/lib/ruby/2.3/amd64-freebsd10"]

Therefore any subsequent require "openssl" would use /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb, but /usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/openssl-2.1.0/openssl.so will not be used, because openssl.so has already been loaded into the ruby interpreter by puma.

@rhenium Does the above matches your description of how things should work?

saper commented Dec 28, 2017

Here's what I think happens:

  • puma starts and requires openssl, which is loaded from stdlib location /usr/local/lib/ruby/2.3/openssl.rb. (Bundler is not used)
  • puma tries to initialize application. $LOAD_PATH is set to
["/usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib",
 "/usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/puma-3.11.0",
 "/usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib",
 "/usr/local/lib/ruby/site_ruby/2.3",
 "/usr/local/lib/ruby/site_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/site_ruby",
 "/usr/local/lib/ruby/vendor_ruby/2.3",
 "/usr/local/lib/ruby/vendor_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/vendor_ruby",
 "/usr/local/lib/ruby/2.3",
 "/usr/local/lib/ruby/2.3/amd64-freebsd10"]
  • mastodon requires bundler/setup in config/boot.rb - @Jehops this is where Gemfile.lock gets re-generated btw.
  • After this the $LOAD_PATH changes:
["/usr/local/lib/ruby/gems/2.3/gems/bundler-1.16.0/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/webpush-0.3.2/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/webpacker-3.0.2/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/uglifier-3.2.0/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/tzinfo-data-1.2017.3/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/twitter-text-1.14.7/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/strong_migrations-0.1.9/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/simple_form-3.5.0/lib",
 ...
 "/usr/local/lib/ruby/gems/2.3/gems/ostatus2-2.0.1/lib",
 "/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib",
 "/usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/openssl-2.1.0",
 ...
 "/usr/local/lib/ruby/gems/2.3/gems/rake-12.3.0/lib",
 "/usr/local/lib/ruby/site_ruby/2.3",
 "/usr/local/lib/ruby/site_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/site_ruby",
 "/usr/local/lib/ruby/vendor_ruby/2.3",
 "/usr/local/lib/ruby/vendor_ruby/2.3/amd64-freebsd10",
 "/usr/local/lib/ruby/vendor_ruby",
 "/usr/local/lib/ruby/2.3",
 "/usr/local/lib/ruby/2.3/amd64-freebsd10"]

Therefore any subsequent require "openssl" would use /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb, but /usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/openssl-2.1.0/openssl.so will not be used, because openssl.so has already been loaded into the ruby interpreter by puma.

@rhenium Does the above matches your description of how things should work?

@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 28, 2017

Member

@saper

But, in any case, files from the gem and from the stdlib one will never be mixed.

So how would you explain the behaviour we are seeing? Is this true also when the stdlib binary module has been loaded into interpreter's memory?

That's why I said weird...

@Jehops

It may be worth noting that bundler is not used with the FreeBSD Mastodon package.

OK, this explains why files from Ruby 2.3's stdlib are loaded even though mastodon's Gemfile.lock has openssl. gem "openssl" must be done before require "openssl" happens so that the gem is properly used in Ruby 2.3 as ostatus2 requests. (By the way, you may run into similar troubles with other libraries too unless you can use Bundler, since many stdlibs are "default gem"-ified this year.)

As a separate issue, still I have no idea why only lib/openssl.rb and lib/openssl/pkcs5.rb from the gem.

Member

rhenium commented Dec 28, 2017

@saper

But, in any case, files from the gem and from the stdlib one will never be mixed.

So how would you explain the behaviour we are seeing? Is this true also when the stdlib binary module has been loaded into interpreter's memory?

That's why I said weird...

@Jehops

It may be worth noting that bundler is not used with the FreeBSD Mastodon package.

OK, this explains why files from Ruby 2.3's stdlib are loaded even though mastodon's Gemfile.lock has openssl. gem "openssl" must be done before require "openssl" happens so that the gem is properly used in Ruby 2.3 as ostatus2 requests. (By the way, you may run into similar troubles with other libraries too unless you can use Bundler, since many stdlibs are "default gem"-ified this year.)

As a separate issue, still I have no idea why only lib/openssl.rb and lib/openssl/pkcs5.rb from the gem.

@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 28, 2017

Member

@saper Ah, that explains what's going on. Because openssl is not a default gem in Ruby 2.3, RubyGems could not detect the collision. I could minimize that to:

$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/)'
ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/opt/ruby/2.3.6/lib/ruby/2.3.0/x86_64-linux/openssl.so
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/bn.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/pkey.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/cipher.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/config.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/digest.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/x509.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/buffering.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/ssl.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl.rb
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl.rb
Member

rhenium commented Dec 28, 2017

@saper Ah, that explains what's going on. Because openssl is not a default gem in Ruby 2.3, RubyGems could not detect the collision. I could minimize that to:

$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/)'
ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/opt/ruby/2.3.6/lib/ruby/2.3.0/x86_64-linux/openssl.so
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/bn.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/pkey.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/cipher.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/config.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/digest.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/x509.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/buffering.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/ssl.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl.rb
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl.rb
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

Yes!

On ruby 2.3 with mastodon:

# ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.5p376 (2017-09-14 revision 59905) [amd64-freebsd10]
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/2.3/amd64-freebsd10/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
	from -e:1:in `<main>'

On a clean installation of ruby 2.4 with the following packages installed only:

# pkg -j pureruby24 info
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libedit-3.1.20170329_2,1       Command line editor library
libffi-3.2.1_2                 Foreign Function Interface
libunwind-20170113_1           Generic stack unwinding library
libyaml-0.1.6_2                YAML 1.1 parser and emitter written in C
ruby-2.4.3,1                   Object-oriented interpreted scripting language
ruby24-gems-2.6.14             Package management framework for the Ruby language
rubygem-openssl-2.1.0          Ruby gem that wraps around the OpenSSL library

I am getting the same:

root@pureruby24:/ # ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so
/usr/local/lib/ruby/2.4/openssl/bn.rb
/usr/local/lib/ruby/2.4/openssl/pkey.rb
/usr/local/lib/ruby/2.4/openssl/cipher.rb
/usr/local/lib/ruby/2.4/openssl/config.rb
/usr/local/lib/ruby/2.4/openssl/digest.rb
/usr/local/lib/ruby/2.4/openssl/x509.rb
/usr/local/lib/ruby/2.4/openssl/buffering.rb
/usr/local/lib/ruby/2.4/openssl/ssl.rb
/usr/local/lib/ruby/2.4/openssl.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
	from -e:1:in `<main>'

It looks to me that ruby 2.3 and 2.4 behave similarly here....

What is special here: OpenSSL::PKCS5.pbkdf2_hmac is defined as a C function and the openssl gem 2.1.0 overrides it with a ruby method instead.

saper commented Dec 28, 2017

Yes!

On ruby 2.3 with mastodon:

# ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.5p376 (2017-09-14 revision 59905) [amd64-freebsd10]
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/2.3/amd64-freebsd10/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
	from -e:1:in `<main>'

On a clean installation of ruby 2.4 with the following packages installed only:

# pkg -j pureruby24 info
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libedit-3.1.20170329_2,1       Command line editor library
libffi-3.2.1_2                 Foreign Function Interface
libunwind-20170113_1           Generic stack unwinding library
libyaml-0.1.6_2                YAML 1.1 parser and emitter written in C
ruby-2.4.3,1                   Object-oriented interpreted scripting language
ruby24-gems-2.6.14             Package management framework for the Ruby language
rubygem-openssl-2.1.0          Ruby gem that wraps around the OpenSSL library

I am getting the same:

root@pureruby24:/ # ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so
/usr/local/lib/ruby/2.4/openssl/bn.rb
/usr/local/lib/ruby/2.4/openssl/pkey.rb
/usr/local/lib/ruby/2.4/openssl/cipher.rb
/usr/local/lib/ruby/2.4/openssl/config.rb
/usr/local/lib/ruby/2.4/openssl/digest.rb
/usr/local/lib/ruby/2.4/openssl/x509.rb
/usr/local/lib/ruby/2.4/openssl/buffering.rb
/usr/local/lib/ruby/2.4/openssl/ssl.rb
/usr/local/lib/ruby/2.4/openssl.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
	from -e:1:in `<main>'

It looks to me that ruby 2.3 and 2.4 behave similarly here....

What is special here: OpenSSL::PKCS5.pbkdf2_hmac is defined as a C function and the openssl gem 2.1.0 overrides it with a ruby method instead.

@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper commented Dec 28, 2017

Possibly related: rubygems/rubygems#1507

@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 28, 2017

Member
  • mastodon requires bundler/setup in config/boot.rb - @Jehops this is where Gemfile.lock gets re-generated btw.

So Bundler is already a runtime dependency of the mastodon port. The quickest and solid fix for the port would be to let puma run in the Bundler environment.

Let me close this issue, as this is not an issue of openssl anymore.

Member

rhenium commented Dec 28, 2017

  • mastodon requires bundler/setup in config/boot.rb - @Jehops this is where Gemfile.lock gets re-generated btw.

So Bundler is already a runtime dependency of the mastodon port. The quickest and solid fix for the port would be to let puma run in the Bundler environment.

Let me close this issue, as this is not an issue of openssl anymore.

@rhenium rhenium closed this Dec 28, 2017

@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

@rhenium using Bundler is not a fix for the require problem we have in #180 (comment) ...

saper commented Dec 28, 2017

@rhenium using Bundler is not a fix for the require problem we have in #180 (comment) ...

@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 28, 2017

Member

@saper

It looks to me that ruby 2.3 and 2.4 behave similarly here....

Apparently Ruby from FreeBSD ports have disabled the "default gem" mechanism for some reason I don't know.

https://github.com/freebsd/freebsd-ports/blob/6de157d78b39cbe7986736613b3cd25958934f11/lang/ruby24/files/patch-tool_rbinstall.rb

Possibly related: rubygems/rubygems#1507

It's irrelevant. It's about the filename extension of extension libraries.

@rhenium using Bundler is not a fix for the require problem we have in #180 (comment) ...

There might be room for improvement in RubyGems (e.g., detect the collision and make second require fail with an exception, though I don't think it can be implemented easily), but nothing openssl can do here...

Member

rhenium commented Dec 28, 2017

@saper

It looks to me that ruby 2.3 and 2.4 behave similarly here....

Apparently Ruby from FreeBSD ports have disabled the "default gem" mechanism for some reason I don't know.

https://github.com/freebsd/freebsd-ports/blob/6de157d78b39cbe7986736613b3cd25958934f11/lang/ruby24/files/patch-tool_rbinstall.rb

Possibly related: rubygems/rubygems#1507

It's irrelevant. It's about the filename extension of extension libraries.

@rhenium using Bundler is not a fix for the require problem we have in #180 (comment) ...

There might be room for improvement in RubyGems (e.g., detect the collision and make second require fail with an exception, though I don't think it can be implemented easily), but nothing openssl can do here...

saper added a commit to saper/ostatus2 that referenced this issue Dec 28, 2017

Runtime check for RSA.set_key
Technically the set_key can be made
available by installing openssl gem 2.0.0 or later.

Remove openssl 2.0 dependency (to avoid
ruby/openssl#180)

Test on old rubies.
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

I think this is not a FreeBSD-only problem.

I have tried this now with Fedora 25, with openssl-2.1.0 installed in $HOME/.gem via gem install openssl.

[saper@fedora0 ~]$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/lib64/ruby/openssl.so
/usr/share/ruby/openssl/bn.rb
/usr/share/ruby/openssl/pkey.rb
/usr/share/ruby/openssl/cipher.rb
/usr/share/ruby/openssl/config.rb
/usr/share/ruby/openssl/digest.rb
/usr/share/ruby/openssl/x509.rb
/usr/share/ruby/openssl/buffering.rb
/usr/share/ruby/openssl/ssl.rb
/usr/share/ruby/openssl.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
	from -e:1:in `<main>'
[saper@fedora0 ~]$ cat /etc/redhat-release 
Fedora release 25 (Twenty Five)
[saper@fedora0 ~]$ rpm -qa | grep ruby
ruby-devel-2.3.4-64.fc25.x86_64
rubygems-2.5.2.1-64.fc25.noarch
ruby-2.3.4-64.fc25.x86_64
rubypick-1.1.1-5.fc24.noarch
rubygem-io-console-0.4.5-64.fc25.x86_64
rubygem-rdoc-4.2.2-3.fc25.noarch
ruby-libs-2.3.4-64.fc25.x86_64
rubygem-bigdecimal-1.2.8-64.fc25.x86_64
rubygem-json-1.8.3.1-64.fc25.x86_64
ruby-irb-2.3.4-64.fc25.noarch
rubygem-psych-2.1.0-64.fc25.x86_64
rubygem-did_you_mean-1.0.0-64.fc25.x86_64

ruby-libs package contains an older version of openssl:

[saper@fedora0 ~]$ rpm -ql ruby-libs | grep ssl
/usr/lib64/ruby/openssl.so
/usr/share/ruby/drb/ssl.rb
/usr/share/ruby/openssl
/usr/share/ruby/openssl.rb
/usr/share/ruby/openssl/bn.rb
/usr/share/ruby/openssl/buffering.rb
/usr/share/ruby/openssl/cipher.rb
/usr/share/ruby/openssl/config.rb
/usr/share/ruby/openssl/digest.rb
/usr/share/ruby/openssl/pkey.rb
/usr/share/ruby/openssl/ssl.rb
/usr/share/ruby/openssl/x509.rb
/usr/share/ruby/webrick/accesslog.rb
/usr/share/ruby/webrick/ssl.rb
[saper@fedora0 ~]$ objdump -T /usr/lib64/ruby/openssl.so |grep pkcs5
0000000000036cd0 g    DF .text	000000000000007b  Base        Init_ossl_pkcs5
[saper@fedora0 ~]$ objdump -T ~/.gem/ruby/extensions/x86_64-linux/2.3.0/openssl-2.1.0/openssl.so  | grep pkcs5 ; echo $?
1

saper commented Dec 28, 2017

I think this is not a FreeBSD-only problem.

I have tried this now with Fedora 25, with openssl-2.1.0 installed in $HOME/.gem via gem install openssl.

[saper@fedora0 ~]$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/lib64/ruby/openssl.so
/usr/share/ruby/openssl/bn.rb
/usr/share/ruby/openssl/pkey.rb
/usr/share/ruby/openssl/cipher.rb
/usr/share/ruby/openssl/config.rb
/usr/share/ruby/openssl/digest.rb
/usr/share/ruby/openssl/x509.rb
/usr/share/ruby/openssl/buffering.rb
/usr/share/ruby/openssl/ssl.rb
/usr/share/ruby/openssl.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
	from -e:1:in `<main>'
[saper@fedora0 ~]$ cat /etc/redhat-release 
Fedora release 25 (Twenty Five)
[saper@fedora0 ~]$ rpm -qa | grep ruby
ruby-devel-2.3.4-64.fc25.x86_64
rubygems-2.5.2.1-64.fc25.noarch
ruby-2.3.4-64.fc25.x86_64
rubypick-1.1.1-5.fc24.noarch
rubygem-io-console-0.4.5-64.fc25.x86_64
rubygem-rdoc-4.2.2-3.fc25.noarch
ruby-libs-2.3.4-64.fc25.x86_64
rubygem-bigdecimal-1.2.8-64.fc25.x86_64
rubygem-json-1.8.3.1-64.fc25.x86_64
ruby-irb-2.3.4-64.fc25.noarch
rubygem-psych-2.1.0-64.fc25.x86_64
rubygem-did_you_mean-1.0.0-64.fc25.x86_64

ruby-libs package contains an older version of openssl:

[saper@fedora0 ~]$ rpm -ql ruby-libs | grep ssl
/usr/lib64/ruby/openssl.so
/usr/share/ruby/drb/ssl.rb
/usr/share/ruby/openssl
/usr/share/ruby/openssl.rb
/usr/share/ruby/openssl/bn.rb
/usr/share/ruby/openssl/buffering.rb
/usr/share/ruby/openssl/cipher.rb
/usr/share/ruby/openssl/config.rb
/usr/share/ruby/openssl/digest.rb
/usr/share/ruby/openssl/pkey.rb
/usr/share/ruby/openssl/ssl.rb
/usr/share/ruby/openssl/x509.rb
/usr/share/ruby/webrick/accesslog.rb
/usr/share/ruby/webrick/ssl.rb
[saper@fedora0 ~]$ objdump -T /usr/lib64/ruby/openssl.so |grep pkcs5
0000000000036cd0 g    DF .text	000000000000007b  Base        Init_ossl_pkcs5
[saper@fedora0 ~]$ objdump -T ~/.gem/ruby/extensions/x86_64-linux/2.3.0/openssl-2.1.0/openssl.so  | grep pkcs5 ; echo $?
1
@saper

This comment has been minimized.

Show comment
Hide comment
@saper

saper Dec 28, 2017

I think the problem with openssl 2.1 is that it breaks C ABI by removing OpenSSL::PKCS5 stuff from the C binding file (Init_ossl_pkcs5 is no longer there) and the backward-compatibility works only on the Ruby level

@rhenium Would you be open for a change that restores OpenSSL::PKCS5 in the C binding and drops the pkcs5.rb workaround? I'd propose to remove OpenSSL::PKCS5 altogether in 3.0.

saper commented Dec 28, 2017

I think the problem with openssl 2.1 is that it breaks C ABI by removing OpenSSL::PKCS5 stuff from the C binding file (Init_ossl_pkcs5 is no longer there) and the backward-compatibility works only on the Ruby level

@rhenium Would you be open for a change that restores OpenSSL::PKCS5 in the C binding and drops the pkcs5.rb workaround? I'd propose to remove OpenSSL::PKCS5 altogether in 3.0.

@rhenium

This comment has been minimized.

Show comment
Hide comment
@rhenium

rhenium Dec 29, 2017

Member

@saper

[saper@fedora0 ~]$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]

I only wanted to show how the mastodon port was going wrongly by a simplified code. In Ruby 2.3 (and also in FreeBSD's Ruby 2.4 port), where openssl is not a "default gem", the first require loads the stdlib one, and then gem activates the gem, and then the second require loads the gem one. This is wrong since it is not possible to load two different versions of a library in Ruby.

I think the problem with openssl 2.1 is that it breaks C ABI by removing OpenSSL::PKCS5 stuff from the C binding file (Init_ossl_pkcs5 is no longer there) and the backward-compatibility works only on the Ruby level

@rhenium Would you be open for a change that restores OpenSSL::PKCS5 in the C binding and drops the pkcs5.rb workaround? I'd propose to remove OpenSSL::PKCS5 altogether in 3.0.

There is no such thing as C level ABI. Our only public interface is the Ruby level one.

As for the "uninitialized constant OpenSSL::KDF" symptom, its cause is that files from two different versions of the openssl library are mixed -- it is a user error. Indeed it is breaking in a weird way, so this:

There might be room for improvement in RubyGems (e.g., detect the collision and make second require fail with an exception, though I don't think it can be implemented easily), but nothing openssl can do here...

Member

rhenium commented Dec 29, 2017

@saper

[saper@fedora0 ~]$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]

I only wanted to show how the mastodon port was going wrongly by a simplified code. In Ruby 2.3 (and also in FreeBSD's Ruby 2.4 port), where openssl is not a "default gem", the first require loads the stdlib one, and then gem activates the gem, and then the second require loads the gem one. This is wrong since it is not possible to load two different versions of a library in Ruby.

I think the problem with openssl 2.1 is that it breaks C ABI by removing OpenSSL::PKCS5 stuff from the C binding file (Init_ossl_pkcs5 is no longer there) and the backward-compatibility works only on the Ruby level

@rhenium Would you be open for a change that restores OpenSSL::PKCS5 in the C binding and drops the pkcs5.rb workaround? I'd propose to remove OpenSSL::PKCS5 altogether in 3.0.

There is no such thing as C level ABI. Our only public interface is the Ruby level one.

As for the "uninitialized constant OpenSSL::KDF" symptom, its cause is that files from two different versions of the openssl library are mixed -- it is a user error. Indeed it is breaking in a weird way, so this:

There might be room for improvement in RubyGems (e.g., detect the collision and make second require fail with an exception, though I don't think it can be implemented easily), but nothing openssl can do here...

@RafaelCamarda

This comment has been minimized.

Show comment
Hide comment
@RafaelCamarda

RafaelCamarda Feb 5, 2018

I had the same issue here:

RafaelCamarda commented Feb 5, 2018

I had the same issue here:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment