I encountered an error when running JRuby 1.6.5/1.6.4 with Capistrano: "(OpenSSL::PKey::DSAError: bad sig size)" I found this ticket filed with net-ssh http://net-ssh.lighthouseapp.com/projects/36253-net-ssh/tickets/18-bad-sig-size-opensslpkeydsaerror-with-jruby-151 that looked like a similar issue. I did some digging and saw the difference between JRuby and MRI was that the OpenSSL::ASN1::Integer#value method returns Bignum under JRuby versus the OpenSSL::BN in MRI. It looks like this is a problem in either jruby-openssl or possibly bouncycastle. I posted a workaround to the net-ssh ticket until this is fixed.
Thank you. Confirmed the problem as follows.
% ruby -ropenssl -e 'p OpenSSL::ASN1.decode(OpenSSL::ASN1::Integer.new(1).to_der).value.class'
% jruby -ropenssl -e 'p OpenSSL::ASN1.decode(OpenSSL::ASN1::Integer.new(1).to_der).value.class'
Filed the ticket at https://jira.codehaus.org/browse/JRUBY-6260
I posted a patch to the ticket.