java.security.InvalidKeyException when upgrading to Logstash 6.3.2

[MikeKemmerer]

Since upgrading from Logstash 6.3.0 to 6.3.2 and from version 5.2.1 to version 5.2.2 of this plugin, I began experiencing this error when outputting data via the http output plugin. I'm running RedHat 7 and java is: java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64

The error persisted after downgrading to version 5.2.1 of the plugin. After downgrading from 6.3.2 down to 6.3.0, the issue went away without changing anything else.

Thanks.

[2018-08-23T22:06:48,376][ERROR][logstash.outputs.http ] Error in http output loop {:class=>"Java::JavaSecuritySpec::InvalidKeySpecException", :message=>"java.security.InvalidKeyException: IOException : algid parse error, not a sequence", :backtrace=>["sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(sun/security/rsa/RSAKeyFactory.java:217)", "java.security.KeyFactory.generatePrivate(java/security/KeyFactory.java:372)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)", "RUBY.block in setup_key_store(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:681)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.setup_key_store(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:676)", "RUBY.ssl_socket_factory_from_options(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:622)", "RUBY.pool_builder(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:397)", "RUBY.pool(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:405)", "RUBY.initialize(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:209)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1001)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.make_client(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-http_client-6.0.1/lib/logstash/plugin_mixins/http_client.rb:180)", "RUBY.client(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-http_client-6.0.1/lib/logstash/plugin_mixins/http_client.rb:185)", "RUBY.send_event(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.1/lib/logstash/outputs/http.rb:259)", "RUBY.send_events(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.1/lib/logstash/outputs/http.rb:194)", "RUBY.multi_receive(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.1/lib/logstash/outputs/http.rb:124)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:908)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.doOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:219)", "org.logstash.config.ir.compiler.OutputStrategyExt$SharedOutputStrategyExt.output(org/logstash/config/ir/compiler/OutputStrategyExt.java:247)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:109)", "org.logstash.config.ir.compiler.OutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/OutputDelegatorExt.java:156)", "org.logstash.config.ir.compiler.OutputDelegatorExt$INVOKER$i$1$0$multiReceive.call(org/logstash/config/ir/compiler/OutputDelegatorExt$INVOKER$i$1$0$multiReceive.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther5:multi_receive(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:475)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.block in output_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:475)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1362)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1359)", "org.jruby.RubyHash.visitLimited(org/jruby/RubyHash.java:662)", "org.jruby.RubyHash.visitAll(org/jruby/RubyHash.java:647)", "org.jruby.RubyHash.iteratorVisitAll(org/jruby/RubyHash.java:1319)", "org.jruby.RubyHash.each_pairCommon(org/jruby/RubyHash.java:1354)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1343)", "org.jruby.RubyHash$INVOKER$i$0$0$each.call(org/jruby/RubyHash$INVOKER$i$0$0$each.gen)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.invokeOther11:each(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:474)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.output_batch(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:474)", "RUBY.worker_loop(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426)", "RUBY.block in start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}

[robbavey]

Hi @MikeKemmerer Can you share your configuration (removing any sensitive information)? From the stack trace, it looks like you might be using a client cert - if so, can you also share how the client cert/key were created

[MikeKemmerer]

Here's the configuration:

http {
        client_cert => "/etc/logstash/HOSTNAME.pem"
        client_key => "/etc/logstash/HOSTNAME.key"
        content_type => "application/json"
        http_method => "post"
        format => "message"
        codec => "json"
        message => "%{message_v5}"
        truststore => "/etc/pki/tls/certs/client.truststore.jks"
        truststore_password => REDACTED
        url => "https://REDACTED"
        socket_timeout => 90
    }

Here's How I'm creating my key and CSR:

• openssl genrsa -out /etc/pki/tls/private/$(hostname -s)_$(date +%Y-%m-%d).key 4096
• openssl req -new -out /etc/pki/tls/private/$(hostname -s)_$(date +%Y-%m-%d).csr -key /etc/pki/tls/private/$(hostname -s)_$(date +%Y-%m-%d).key

My org is creating the certificate using an internal CA

[imintowin]

I am getting the same error with Logstash 6.6.0 when I use an RSA-encoded private key.
Converting it to PKCS8 format with the openssl command mentioned in elastic/logstash#9897 solves the problem, but we currently use the RSA format in our application and it works fine with Logstash 6.2.4 (exact same http output setup).
Did the newer version of Logstash drop support of RSA private keys or is this a bug?

My output configuration:

output {
        http {
            http_method => "post"
            url => "https://..."
            cacert => "/tmp/certs/ca.pem"
            client_cert => "/tmp/certs/client.pem"
            client_key => "/tmp/certs/key.pem"
            content_type => "application/json"
            format => "json_batch"
            automatic_retries => 1
            connect_timeout => 10
            cookies => true
            follow_redirects => true
            http_compression => true
            keepalive => true
            pool_max => 50
            pool_max_per_route => 25
            request_timeout => 60
            retry_failed => true
            retry_non_idempotent => false
            socket_timeout => 10
        }
}

Error:

:message=>"java.security.InvalidKeyException: IOException : algid parse error, not a sequence", :class=>"Java::JavaSecuritySpec::InvalidKeySpecException", :backtrace=>["sun.security.rsa.RSAKeyFactory.engineGeneratePrivate(sun/security/rsa/RSAKeyFactory.java:217)"
, "java.security.KeyFactory.generatePrivate(java/security/KeyFactory.java:372)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:290)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.block in setup_key_store(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:681)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.setup_key_store(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:676)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$setup_key_store$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.ssl_socket_factory_from_options(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:622)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$ssl_socket_factory_from_options$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.pool_builder(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:397)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.RUBY$method$pool_builder$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.pool(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:405)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.initialize(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:209)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_mixin_minus_http_client_minus_6_dot_0_dot_1.lib.logstash.plugin_mixins.http_client.make_client(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-http_client-6.0.1/lib/logstash/plugin_mixins/http_client.rb:180)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_mixin_minus_http_client_minus_6_dot_0_dot_1.lib.logstash.plugin_mixins.http_client.RUBY$method$make_client$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_mixin_minus_http_client_minus_6_dot_0_dot_1/lib/logstash/plugin_mixins//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-http_client-6.0.1/lib/logstash/plugin_mixins/http_client.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_mixin_minus_http_client_minus_6_dot_0_dot_1.lib.logstash.plugin_mixins.http_client.client(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-http_client-6.0.1/lib/logstash/plugin_mixins/http_client.rb:185)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_mixin_minus_http_client_minus_6_dot_0_dot_1.lib.logstash.plugin_mixins.http_client.RUBY$method$client$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_mixin_minus_http_client_minus_6_dot_0_dot_1/lib/logstash/plugin_mixins//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-http_client-6.0.1/lib/logstash/plugin_mixins/http_client.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_output_minus_http_minus_5_dot_2_dot_3.lib.logstash.outputs.http.send_event(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.3/lib/logstash/outputs/http.rb:239)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_output_minus_http_minus_5_dot_2_dot_3.lib.logstash.outputs.http.RUBY$method$send_event$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_output_minus_http_minus_5_dot_2_dot_3/lib/logstash/outputs//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.3/lib/logstash/outputs/http.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_output_minus_http_minus_5_dot_2_dot_3.lib.logstash.outputs.http.send_events(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.3/lib/logstash/outputs/http.rb:175)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_output_minus_http_minus_5_dot_2_dot_3.lib.logstash.outputs.http.RUBY$method$send_events$0$__VARARGS__(users/bssnd/tools/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_output_minus_http_minus_5_dot_2_dot_3/lib/logstash/outputs//users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.3/lib/logstash/outputs/http.rb)", "Users.boris_sandler.tools.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_output_minus_http_minus_5_dot_2_dot_3.lib.logstash.outputs.http.multi_receive(/users/bssnd/tools/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-http-5.2.3/lib/logstash/outputs/http.rb:124)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.invokeOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:124)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.doOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:242)", "org.logstash.config.ir.compiler.OutputStrategyExt$SharedOutputStrategyExt.output(org/logstash/config/ir/compiler/OutputStrategyExt.java:271)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:114)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doOutput(org/logstash/config/ir/compiler/OutputDelegatorExt.java:78)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$1$0$multiReceive.call(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt$INVOKER$i$1$0$multiReceive.gen)", "Users.boris_sandler.tools.logstash.logstash_minus_core.lib.logstash.pipeline.block in output_batch(/users/bssnd/tools/logstash/logstash-core/lib/logstash/pipeline.rb:373)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1362)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1359)", "org.jruby.RubyHash.visitLimited(org/jruby/RubyHash.java:662)", "org.jruby.RubyHash.visitAll(org/jruby/RubyHash.java:647)", "org.jruby.RubyHash.iteratorVisitAll(org/jruby/RubyHash.java:1319)", "org.jruby.RubyHash.each_pairCommon(org/jruby/RubyHash.java:1354)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1343)", "org.jruby.RubyHash$INVOKER$i$0$0$each.call(org/jruby/RubyHash$INVOKER$i$0$0$each.gen)", "Users.boris_sandler.tools.logstash.logstash_minus_core.lib.logstash.pipeline.output_batch(/users/bssnd/tools/logstash/logstash-core/lib/logstash/pipeline.rb:372)", "RUBY.worker_loop(/users/bssnd/tools/logstash/logstash-core/lib/logstash/pipeline.rb:324)", "Users.boris_sandler.tools.logstash.logstash_minus_core.lib.logstash.pipeline.block in start_workers(/users/bssnd/tools/logstash/logstash-core/lib/logstash/pipeline.rb:287)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :will_retry=>false}

[borissnd]

This is still an issue with more recent Logstash versions and plugins. Tested with Logstash version 8.4.2.
Getting: :method=>:post, :message=>\"java.security.InvalidKeyException: IOException : null when the private key has a leading/trailing whitespace in one of the lines that apparently gets trimmed, happens once in every 50-100 generated certs.

[kosch]

@borissnd Can you be more specific? I checked the affected private keys, but I can't the mentioned whitespaces. When I exactly know where to look at, I can check the private keys when they are created and recreate if needed.