openssl 'BC' (security provider) leak #1543

Closed
wants to merge 84 commits into from

3 participants

@kares
JRuby Team member

there's an issue around https://github.com/jruby/jruby/blob/d4ed739885758c482a8a7147f482cf934eeebb67/ext/openssl/src/main/java/org/jruby/ext/openssl/OpenSSLReal.java#L90 ... it might be causing (unpredictable) leaks esp. under environments where multiple Ruby runtimes load their own BC classes and execute addProvider.

the only solution is to review all security "getInstance" factory calls and avoid everything that depends on 'BC' being registered (e.g. NetscapeCertRequest), which is what happened here with some additional cleanup since I was unable to read some parts of the code (in a shallow attempt to understand what's it doing) ... let me know what do you guys think. I also have this ready for jruby-1_7 since that is what I'm actually be putting onto production :)

I kept the commits (well I did some squashing-re-arranging of mine) as they are so that anyone following the log better understands why things are the way they are.

kares added some commits Feb 28, 2014
@kares kares cleanup and re-arrange some imports to be more readable f35f34d
@kares kares implement BC specific (factory) helpers using BC APIs "directly"
we're trying to avoid registering the java.security provider since it's problematic in environments with multiple runtimes started

+ start implementing our own [security].getInstance factory methods
cc3b088
@kares kares start using our factory methods instead of the java.security stuff di…
…rectly
4ce51c1
@kares kares replace Signature.getInstance usage with a BC first than rest of prov…
…iders one
888d8f3
@kares kares cleanup PKeyDSA to use our own factories - added `getKeyPairGenerator` da145e2
@kares kares setup our own getMac factory and use it + some minor HMAC cleanup
- mostly do not catch and hide argument to_s conversion errors raised
8a15ff9
@kares kares building strings ( the Java "way" :) e9f5ced
@kares kares deprecate doWithBCProvider and avoid it in code base (NetscapeSPKI)
generixize getWithBCProvider's Callable lambda interface
d1fe0bd
@kares kares NetscapeSPKI now loadable even without the BC API (fails on invocation) 22c3981
@kares kares introducing our own NetscapeCertRequest with the get "BC" provider av…
…oided
db5ade6
@kares kares finally, we seem to be able to deprecate getWithBCProvider - not used…
… at all
2ac74c5
@kares kares setup tests for our factories and tune them out + initialize BC when …
…ext loads

unfinalize constants (for tests) and only call BC-factories if provider != null
89775c3
@kares kares cleanup Ruby Cipher impl - use our getCipher factory instead of the J…
…ava one
6803e3c
@kares kares Cipher.getInstance gets tricky due JCE internals - but seems fine the…
… API way

it depends on which provider's classes are loaded with the SunJCE's one it won't work since the internal `JceSecurityManager.INSTANCE.isCallerTrusted()` fails
- but Sun's getCipher(transformation, provider) works without any assumptions being made for the provider to be registered among the Security providers ...
c6291c4
@kares kares move factories into a new SecurityHelper - allows to change/register …
…provider

- for slightly extreme cases where users want to force using a given provider (and they of course understand the drawbacks)
- also if registering a provider is desired there's a public method to be called
39110c9
@kares kares cleanup the `OpenSSL::X509::CRL` impl a bit - as it seemed hard to read f639e02
@kares kares avoid Jave Security get_instance factories and registering BC in pkcs…
…12.rb
9fa27c2
@kares
JRuby Team member

OK, this still isn't enough - the 1.47 PEMReader does https://travis-ci.org/jruby/jruby/jobs/20046780#L881

@mkristian
JRuby Team member

from https://travis-ci.org/jruby/jruby/builds/20241315 you see if you run
$mvn -Pmain -Dinvoker.skip=false
that there are side effects if you run jruby with an old jruby-openssl gem installed. I will have a look at that if I can find something.

@kares
JRuby Team member

@mkristian yep I noticed ... just hadn't had time to review those (probably a "refactoring" regression)
... it's definitely on my list, but I appreciate any reviews or feedback I get as it is already. so thanks!

kares added some commits Apr 9, 2014
@kares kares avoid direct getInstance and use our own + more ("jruby-style") cleanup f2e8624
@kares kares implement (and use) getKeyGenerator and getSecureRandom factory e1f810d
@kares kares use getSecureRandom + re-invent RandomHolder & cleanup JRuby methods da1493e
@kares kares setup getSSLContext (for completeness) and cleanup SSL impl classes some ecb2cdd
@kares kares allow to set the OpenSSL.debug with a system property + isDebug helpe…
…r(s)
a4c1119
@kares kares deprecate the PEMHandler (seems no longer used)
- we do not want to use PEMReader (deprecated in BC + uses getInstance)
e7c0d80
@kares kares provider PEMUtils that are JS getInstance call free + all the noise a…
…round

we're reinventing some of the new org.bouncycastle.openssl.PEMxxx classes since PEMReader (depending on Security factorier) is deprecated anyways in favor of PEMParser ... this will be only needed until we upgrate to BC 1.48/49
4aa6ef7
@kares kares PEMReader (BC API depending on provider being registered) avoided 4ba25a1
@kares kares use Ruby runtime's STDOUT instead of Java System.err 2c73de5
@kares kares BC's `X509CRLObject` depends on security provider being installed (av…
…oid `crl.verify`)
6efc3ca
@kares kares more internal "JRuby" cleanup + renamed `X509StoreCtx` in a binary co…
…mpatible way
6021e2e
@kares kares generate some serialVersionUIDs since they've been (on purpose) remov…
…ed previously
0abfdd0
@kares kares genericsize our Function-al interfaces 9488c4d
@kares kares use an unify function iterface generics + start hiding private/packag…
…e state in x509store
6bcac12
@kares kares allow sharing of our custom `java.security.X509CRL.verify` logic (wit…
…h BC not installed)
9cefe6d
@kares kares avoid `X509CRL.verify` call and use our helper built to do that (with…
…out BC installed)

+ some minor cleanup - only SecurityException should be catched for `System.getenv`
318096b
@mkristian
JRuby Team member

I merged it into the branch test-openssl-bc-leak so I can look into the error on jruby/maven/main

which I did not see locally :(

@kares
JRuby Team member

@mkristian thanks, it's probably done (all green again) - I wanted to try rebasing actually ... which I'll do now

@mkristian
JRuby Team member

just realized it is a bit difficult to get the new openssl with this 'detached' ext module now. look into it tomorrow.

@kares
JRuby Team member

not sure what you meant ... but rebasing went smooth (no conflicts) and on JDK 7 all seems green

@kares
JRuby Team member

updated jruby-1_7 and also tried rebasing - JDK 6/7 went green but Java 8 did not ... at first look not sure why's there this OpenSSL related failure. before diving in more I'd like to ask if you guys consider "backporting" this one and releasing it with JRuby 1.7.x ? I know it seems as a lot of changes but almost all of it is actually internal. I'd also like to look at an OpenSSL issue or two and try updating incrementally BC's version and I would prefer doing that work on top of this.

@mkristian
JRuby Team member
@mkristian
JRuby Team member

still I do have no clue why this shows up on travis:
https://travis-ci.org/jruby/jruby/jobs/22791812#L3565

locally it just works :(

it happens when gems get installed, i.e. before the actual tests

@kares
JRuby Team member

I see, thanks for clarifying things ... master vs. jruby-1_7 was confusing for me since I thought you're doing jruby-openssl releases from the (jruby-1_7) stable branch (esp. since master is not build with Java 6).

that bouncy-castle-java gem dependency seems unfortunate but seems to me at that doing next gem releases such as 1.5.0.148 (and on newer jruby-openssl explicitly declaring what is tested e.g. using add_dependency(%q<bouncy-castle-java>, ["~> 1.5.0.147", "< 1.5.0.150"])) would elegantly solve the issue (without the need to embed those) ... while likely avoiding the headaches with embedding ?

p.s. not sure what to do about that broken build ... might be a regression but no reproducing either ;(

@mkristian
JRuby Team member

if you want to change the bouncy-castle-version please do it in here
https://github.com/jruby/jruby/blob/master/ext/openssl/lib/jopenssl/version.rb
I just went ahead with that recently. of course we can go back - but my point was less future releases of jruby-openssl but the old ones break with future releases of bouncy-castle-java.

about the master vs. jruby-1_7 branch - jruby-openssl gets compiled against jruby-1.7.11 (even on the master branch ;)

good point though to set java6 compatibility during the build.

@mkristian
JRuby Team member

after switching to oracle-jdk8 I can reproduce the error from travis now :)

@kares
JRuby Team member

thanks for the hints, I'll do the compatibility (if possible with multiple BC versions) after you guys can confirm that this is fine to merge.

I understand the desire for less future jruby-openssl releases, however with the embedding you're likely achieving the opposite ... BC is a library that as it gets released might contain important security fixes, by embedding it we can not provide those without a new jruby-openssl release.

any chance you would consider getting "partially" back to a similar way of loading openssl (not immediately but when and if the compatibility with newer versions is resolved) as it was done previously: https://github.com/jruby/jruby/blob/jruby-1_7/lib/ruby/shared/jopenssl/load.rb ... there would be at least 2 important benefits besides being able to update (bouncy-castle separately) :

  • when a new BC gem versions is released and it seems to not work as expected due users reporting issues we can advice them to degrade to an older version (with BC embedded there's the only way of yanking the jruby-openssl release)
  • current versions of the gem might get BC updates (assuming they work of course) ... what is esp. crucial that now seems impossible (and I do think this scenario is a "must" to work towards), if a user installs the BC security provider (globally) we shall NOT attempt to load any version of BC (feel like this was possible previously)

of course this is all assuming that we work-around the old versioning scheme issue ([">= 1.5.0147"]) which I think would be doable by changing the release numbering or introducing a new gem name.
there were quite some "internal" refactoring done between 147 - 150 versions of BC but it feels like most of it is done and next releases will probably not break for us that much.

@mkristian
JRuby Team member
@kares
JRuby Team member

thanks, great to know that were on the same page here - was afraid I've missed something obvious.

a new gem is an option but you probably did not notice that the above "proposed" versioning of the gem would work as well: add_dependency(%q<bouncy-castle-java>, ["~> 1.5.0.147", "< 1.5.0.150"]) since 1.5.0.147 as well as 1.5.0.150 are not matched by [">= 1.5.0147"] :) or maybe it's too clever of a trick (for some) ... leave it up to you to decide but I'm fine with it.

yeah, the 'future' BC compatibility might turn out hard but we should at least try managing it somehow sanely. I'll look into these but the current blocker to merge seems to be the failure on oracle-8 right?
I should probably look into that one or are you working on it?

@mkristian
JRuby Team member

yes, I did not see the "extra" 0 in the version scheme - thanx for pointing that out. and 0147 will be treated as the number 147.

I do have some time today to look into that oracle-8 thingy.

@mkristian
JRuby Team member

just a follow up on the oracle-8 problem:

java -cp core/target/jruby-core-9000.dev-SNAPSHOT-complete.jar:maven/jruby-stdlib/target/jruby-stdlib-9000.dev-SNAPSHOT.jar org.jruby.Main -ropenssl -e "OpenSSL::Cipher.new 'AES-256-CBC'"

succeeds :)
but

java -cp core/target/jruby-core-9000.dev-SNAPSHOT-complete.jar:maven/jruby-stdlib/target/jruby-stdlib-9000.dev-SNAPSHOT.jar org.jruby.Main -e "require 'openssl';OpenSSL::Cipher.new 'AES-256-CBC'"

fails :(

there are actually NO ciphers at all registered in the internal list CIPHERS

@kares
JRuby Team member

woow, that seems really weird ... did you test that code without the changes presented here if it behaves the same ? so that we actually know if it's not a separate issue not affected by these changes ...

on first look seems like it's not loading the "real" openssl thing ... which I'm not sure how it's used (but exists historically). could you maybe for the record note on which internal CIPHERS you've meant ...

@mkristian
JRuby Team member
@kares
JRuby Team member

did some updates but mostly only cleanup ... tried Java 7/8 all seems working locally at my end :

kares@sputnik:~/workspace/oss/jruby$ bin/jruby -v
jruby 9000.dev (2.1.0.dev) 2014-04-15 0cfbe0b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_51-b13 [linux-amd64]
kares@sputnik:~/workspace/oss/jruby$ bin/jruby -e "require 'openssl'; puts OpenSSL::Cipher.new 'AES-256-CBC'"
#<OpenSSL::Cipher:0x3ff970c2>
kares@sputnik:~/workspace/oss/jruby$ bin/jruby -ropenssl -e "puts OpenSSL::Cipher.new 'AES-256-CBC'"
#<OpenSSL::Cipher:0x3173399c>
kares@sputnik:~/workspace/oss/jruby$ 
kares@sputnik:~/workspace/oss/jruby$ export JAVA_HOME=/opt/java/jdk-8
kares@sputnik:~/workspace/oss/jruby$ 
kares@sputnik:~/workspace/oss/jruby$ bin/jruby -v
jruby 9000.dev (2.1.0.dev) 2014-04-15 0cfbe0b on Java HotSpot(TM) 64-Bit Server VM 1.8.0-b132 +indy [linux-amd64]
kares@sputnik:~/workspace/oss/jruby$ bin/jruby -e "require 'openssl'; puts OpenSSL::Cipher.new 'AES-256-CBC'"
#<OpenSSL::Cipher:0x88a8218>
kares@sputnik:~/workspace/oss/jruby$ bin/jruby -ropenssl -e "puts OpenSSL::Cipher.new 'AES-256-CBC'"
#<OpenSSL::Cipher:0x3c2772d1>

I would have tried java -cp as well but I'm not sure why I do not have the stdlib jar built - will investigate ...

@mkristian
JRuby Team member
@mkristian
JRuby Team member

that is the cause of the problem
Caused by: java.util.jar.JarException: Class is on the bootclasspath
at javax.crypto.JarVerifier.verify(JarVerifier.java:247)
at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:160)
at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:186)
at javax.crypto.Cipher.getInstance(Cipher.java:648)

which is the same as this person claim about JCE https://www.java.net/node/677015#comment-717325

but
Cipher.getInstance(transformation)
without provider does work on my system. I will come in minute with a possible "fix" or "workaround"

@mkristian mkristian and 1 other commented on an outdated diff Apr 15, 2014
...c/main/java/org/jruby/ext/openssl/SecurityHelper.java
+ return service.getAlgorithm();
+ }
+ }
+ return null;
+ }
+
+ /**
+ * @note code calling this should not assume BC provider internals !
+ */
+ public static Cipher getCipher(final String transformation)
+ throws NoSuchAlgorithmException, NoSuchPaddingException {
+ try {
+ final Provider provider = getSecurityProvider();
+ if ( securityProvider != null ) return getCipher(transformation, provider);
+ }
+ catch (NoSuchAlgorithmException e) { }
@mkristian
JRuby Team member

first thing I thought that is might be a NoSuchPaddingException but here the JarException was thrown. see #1543 (comment)

I would suggest that instead of only looking for NoSuchAlgorithmException we catch all Exception and try to load the cipher without provider.

was there a reason to not handle NoSuchPaddingException the same as NoSuchAlgorithmException ?

@kares
JRuby Team member
kares added a note Apr 15, 2014

probably not - will change that ... good catch here, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mkristian
JRuby Team member

the full stacktrace from above
java.lang.SecurityException: JCE cannot authenticate the provider BC
at javax.crypto.Cipher.getInstance(Cipher.java:652)
at org.jruby.ext.openssl.SecurityHelper.getCipher(SecurityHelper.java:322)
at org.jruby.ext.openssl.SecurityHelper.getCipher(SecurityHelper.java:309)
at org.jruby.ext.openssl.Cipher.getCipher(Cipher.java:318)
at org.jruby.ext.openssl.Cipher.getCipher(Cipher.java:599)
at org.jruby.ext.openssl.Cipher.updateCipher(Cipher.java:594)
at org.jruby.ext.openssl.Cipher.initialize(Cipher.java:383)
at org.jruby.ext.openssl.Cipher$INVOKER$i$1$0$initialize.call(Cipher$INVOKER$i$1$0$initialize.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrN.call(JavaMethod.java:769)
at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
at org.jruby.RubyClass.newInstance(RubyClass.java:795)
at org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)
at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneOrNBlock.call(JavaMethod.java:297)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:636)
at org.jruby.runtime.invokedynamic.InvocationLinker.invocationFallback(InvocationLinker.java:157)
at ruby.dash_e.file(-e:1)
at ruby.dash_e.load(-e)
at org.jruby.Ruby.runScript(Ruby.java:833)
at org.jruby.Ruby.runScript(Ruby.java:826)
at org.jruby.Ruby.runNormally(Ruby.java:724)
at org.jruby.Ruby.runFromMain(Ruby.java:563)
at org.jruby.Main.doRunFromMain(Main.java:397)
at org.jruby.Main.internalRun(Main.java:292)
at org.jruby.Main.run(Main.java:219)
at org.jruby.Main.main(Main.java:199)
Caused by: java.util.jar.JarException: Class is on the bootclasspath
at javax.crypto.JarVerifier.verify(JarVerifier.java:247)
at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:160)
at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:186)
at javax.crypto.Cipher.getInstance(Cipher.java:648)
at org.jruby.ext.openssl.SecurityHelper.getCipher(SecurityHelper.java:322)
at org.jruby.ext.openssl.SecurityHelper.getCipher(SecurityHelper.java:309)
at org.jruby.ext.openssl.Cipher.getCipher(Cipher.java:318)
at org.jruby.ext.openssl.Cipher.tryCipher(Cipher.java:308)
at org.jruby.ext.openssl.Cipher.access$000(Cipher.java:60)
at org.jruby.ext.openssl.Cipher$CipherModule.initializeCiphers(Cipher.java:115)
at org.jruby.ext.openssl.Cipher$CipherModule.isSupportedCipher(Cipher.java:97)
at org.jruby.ext.openssl.Cipher.initialize(Cipher.java:377)
... 20 more

@kares
JRuby Team member

I see thanks ... I'll probably need to re-thing this one than esp. if there's such a verification going on ...

@mkristian
JRuby Team member

what about using the lightweight api (at 4.0): http://www.bouncycastle.org/specifications.html

that would avoid the JCE completely but it looks like bigger change.

@mkristian
JRuby Team member

one more thought: it looks to me (though I did not look into the old code now) that bouncy castle provider was not used and this is the reason you needed to have the "activate" https://github.com/jruby/jruby/wiki/UnlimitedStrengthCrypto

maybe I am wrong - just random thoughts going through my head . . .

kares added some commits Apr 15, 2014
@kares kares allow to force provider registration with **-Djruby.openssl.provider.…
…register=true**
871b3de
@kares kares no need to close StringReader + rescue IOException since PEMIO should…
… only fail that
202ce00
@kares kares some mangling trying to handle getCipher in a more smart way "around …
…JCE limits" :

- if the jar is on the boot class-path we're likely get a *java.lang.SecurityException: JCE cannot authenticate the provider BC* (`java -cp ... -ropenssl`  works differently than `jruby -S`)

- fooling around with reflection might not help us either if JCE considers our "caller" not trusted ... ends up with *java.lang.NullPointerException at javax.crypto.Cipher.<init>*

we make sure cases above are catched and "quickly" fall to `Cipher.getInstance(String)`
973f96e
@kares
JRuby Team member

BC was sure used and a lot of the OpenSSL API depend on it "heavily" - before also we depended on it being registered as a provider ... you probably mean this in the context of Cipher only ... for that case I'm not 100% sure but it might be needed as well.

the Cipher issue is not related to not registering the provider (added a property for controlling that and failed the same), turns out this is a "security-feature" of the JVM - will not allow to instantiate a Cipher class from an "unverified" source. thus with java -cp this will never work (unless BC is properly installed) - previously the exception was simply swallowed ... with kares@973f96e the behaviour is restored and there's also "another" way of trying (although it will usually fail as well with Oracle JDK 7+ using the deafult JCE) and a fallback to using the builtin providers.

the light-weight BC is great I know it exists but that is even more work, since currently some of the OpenSSL parts (such as the Cipher issue we've run into) might have worked due the built-in providers.

let me know if you run into anything else ...

@kares
JRuby Team member

for these I'm only seeing the Java 8 "not-supported" truffle failure: https://travis-ci.org/kares/jruby/jobs/23070622#L990 ... I'm thinking running with java -cp hitting the jar verification failure might be worth a warning (that not all parts of OpenSSL will work unless the BC provider is properly setup), although people might then start to care about things they might not need to :)

@kares kares changed the title from openssl 'BC' leak to openssl 'BC' (security provider) leak Apr 16, 2014
@mkristian
JRuby Team member

FYI java -jar jruby-complete-9000.dev.jar faces the same problem which is a much more common use-case then the java -cp case

and the actual failure on the integration test on travis was in th gem-maven-plugin which uses the jruby-complete.jar. maven builds a separate classloader for each plugin and jruby-complete will be added to that classloader.

currently it seems to work - hard to find 'situations' which do not work. I guess we have to wait for real world problems ;)

@kares
JRuby Team member

@mkristian yep that's obvious - all java will "fail" unless BC moved elsewhere (in terms of class-loader) ... but it's now behaving as before we just actually know what's happening underneath :)

p.s. there's a few more cleanup (e.g. jruby-openssl can not be compiled against 1.7 as it is currently) commits and a bug fix to be added here (waiting for ci)

@mkristian
JRuby Team member
@kares
JRuby Team member

OK, thanks a lot - just for the record that is NOT something I broke it is actually broken as is on master :)

@mkristian
JRuby Team member
@mkristian
JRuby Team member

so now I have two more issues:

in this thread here
https://www.ruby-forum.com/topic/4423238
I tried to help someone where jruby-complete does not work with bouncy-castle gem or the build in stuff, but works with BC added to -cp
I never was able to reproduce it but the last error this person dug up for me was that DESede was not available which very much sounds like the extra security for the classloading we saw the other day. I gave up on that issue since I never was able to reproduce it and remote debugging is not possible to that extend. one major issue is that exception were swallowed and there is not way to see them.

now back to my issue: it would be great if we could enable verbose debugging on openssl to help such in cases to find the cause of the problem.

next issue: here I mentioned the List of possible cipher names #1543 (comment)
this List means that an instance of each possible cipher will instantiated even if I only want a "AES-256-CBC" cipher for example.

I am sure that this can be made much cheaper in terms of object instantiation. and this refactoring would be the right moment to address this issue ?!?

@kares
JRuby Team member

Hey Kristian ... :) well yeah JRuby-OpenSSL really needs some more maintenance we might do those 2 here but maybe on another PR would be better. I know there's a lot of changes it's actually a result of a simple commitment to cleanup code as I have rolled through figuring out the 'BC' leak internals ...

  • quite an issue - hard to tell without diving deep ... PEM had issues when trying to update pass BC 147 (haven't looked into that yet) ... it's definitely something to look into later when trying to upgrade BC ...

I've tuned the "debug" helpers available already and refactored some places where things might get interesting e.g. kares@973f96e there's a OpenSSLReal.isDebug() followed by printing the stack ... more commonly isDebug(runtime) is used and stacks get printed into STDOUT (where possible). if you look at OpenSSLReal you'll understand how it works ... it can now be turned using -Djruby.openssl.debug=true

  • definitely a waste to instantiate ~ 70 ciphers, but at least it does not happen while loading JRuby-OpenSSL

I'm not sure there's another "good" portable way - we can use the JCE internal classes but that is dangerous (or needs to be implemented with a fall-back) since those are definitely different between JVM vendors + if a custom JCE is installed might not work either. But it should be possible to speed that part up.

For now I need to keep at why this PR exists in the first place. I've identified another possible leak source and managed to reproduce one failure with Java 8 I've seen with travis (was likely in before - but seems to not always happen). So there's still stuff to work on for me and could take a while, but I would appreciate if this got into master to have a base point for other issues to work on (if you feel it's OK) esp. since currently I'm loosing a lot of focus tracking CI / rebasing onto test- branches (I can not find a good base point for a green master build e.g. https://travis-ci.org/kares/jruby/jobs/23199443#L128)

Before merging - there's some conflicts (master included some code that does not compile against 1.7) ... let me know if you would want me to rebase somewhere on another branch or something.

@mkristian
JRuby Team member
@kares
JRuby Team member

not sure I understood all ... to recap: the openssl updates from here won't get to 1_7 branch.
also for the record last build https://travis-ci.org/kares/jruby/builds/23199492 went fine (it's the same code but on test-openssl-bc-leak branch of mine with a slightly more recent master probably).

PR code should currently compile all the way down to 1.6.8 (master won't do even 1.7.11 currently).
if anything else needs to be done, let me know and thank you!

@mkristian
JRuby Team member
kares added some commits Apr 17, 2014
@kares kares unify error location and naming convention as well as OpenSSL::X cons…
…tant resolution +

- when looping through Ruby arrays there's no need to do the toJava conversions
- ASN1 got a new decodeImpl that should be used by other in the package
- ASN1Data "internal" printing now happens on STDOUT instead of System.out
- X509::CRL's extension now behaves more Ruby-sh - set array is kept/returned
- dried-up and made some of the X509::Extension implementation more dev friendly
71c8fd2
@kares kares improved HMAC internals & less string/bytes garbage on digests + re-i…
…nvented hex-ing
2b74ca5
@kares kares review ASN1.decode usage in Attribute#value= there was a "double" to_…
…der_if_possible
48cf947
@kares kares less of our own deprecations + cleanup (some left over 1.7 incompatib…
…le imports in BN)
0889827
@kares kares more X509Name cleanup - added a debug for suspicious exception + revi…
…ewed list fields
b1cd971
@kares
JRuby Team member

@mkristian no hurries if you're busy it can wait a few ... mostly wanted to avoid you waiting for "more" fixes.

in the mean time I added some more battle tested commits ... thanks!

kares added some commits Apr 18, 2014
@kares kares more stack trace debugging - esp. due "strange" occasional failures R…
…SA failures
ff108c7
@kares kares decrade GeneralSecurityException signature + allow to pass key factor…
…y + fix "DSA" typo
88a88a2
@kares kares tune debugging and align DSA/RSA PKey impls - share some common initi…
…alize pieces
04b85ea
@kares kares some less traces from RSA/DSA while running with `-J-Djruby.openssl.d…
…ebug=true`
395976f
@kares kares sharing `Utils.toHex` implementation caused a HMAC regression fd5d4dc
@kares kares PKCS7 internal cleanup 45fb7a2
@kares kares remove unused imports a60d0c2
@kares kares handle HMAC's key/data arguments using `to_s` instead of `to_str` 0a647e0
@kares kares improve BN's (to_i) conversion to happen without the "middle-man" 95c0124
@kares kares reformat SSLSocket and add debug-ing around exceptions + fix some met…
…hod names
8ddd849
@kares kares these seems to prefer `to_s` over `to_str` conversion d96c72a
@kares kares avoid getClassFromPath since `X509.wrap` is called from a loop d04b218
@kares kares toASN1: use `to_s` & "optimize" for the BN case - avoid the need to r…
…ebuild a big int
d28d3b9
@kares kares internalize (ASN1) default objects initialization c58b30b
@kares kares avoid initializing the ASN1Registry class from ASN1 (for now) + dry-u…
…p some
a8c285e
@kares kares ASN1Registry review - optimize OBJ_ constants used - do not look them…
… up in the map
7d133e3
@kares kares (internal) code cleanup db1bb65
@kares kares ASNRegistry minor optimization - no need to be calling to add these w…
…ith `oid == null`
dcaa686
@kares kares at last - avoid ASN1's static maps leaking as multiple runtimes start…
…up and shutdown
0c43646
@kares kares backported MiscPEMGenerator class from bcpkix latest (master) f9bf7fe
@kares kares MiscPEMGenerator tuned to work on BC 1.47 as well as 1.48+ fb4b230
@kares kares support for compiling source under BC 1.48 (1.47 is still working as …
…well)
3280c3a
@kares kares PKey (internal) cleanup - make sure we do not concatenate strings unl…
…ess isDebug
850d87e
@kares kares add a spec for `OpenSSL::PKey::RSA.new` regression - working fine as …
…is (fixes #357)

not tested on Mac (only Linux with Java 1.7.0_51) but we're now not simply doing `Cipher.getInstance(String)` ... we're first attempt to consult (BC) provider directly
c7d7d8a
@kares kares cleanup our internal PEMUtilities copy-pasted from BC to only the par…
…ts we really need
8b1af11
@kares kares good idea to synchronize while updating the maps programatically (fro…
…m user code)
4b8e3a6
@kares kares one more hidden place where we're not using SecurityHelper but securi…
…ty API directly
a389571
@kares kares de-ja-vu BC 1.49 (binary) compatibility while still maintaining 1.47 …
…compatibility

- a better 941ab04 (part) which got reverted
- tested #1238 regression - fine on 1.47 as well as 1.49 with this change
cbd7e4a
@kares kares only use ASN1Integer(BigInteger) constructor for better compatibility…
… with BC versions
eb81669
@kares kares (still) compile JRuby-OpenSSL as Java 1.6 compatible 3541583
@kares
JRuby Team member

... and at last JRuby-OpenSSL is now BC 1.47-1.49 compatible. in can easily be 1.50 compatible as well (and probably is just it does not compile against BC 1.50) it requires removing a deprecated class, but just in case I would first do a 0.10.0 release before going further. as noted before it's JRuby 1.6.x / 1.7.x compatible and I've changed the pom.xml to build using javac --target 1.6 ... the static map leak in ASN1 has also been handled (using a WeakHashMap) ... let me know if there's anything more to be done here.

@mkristian
JRuby Team member

merged to test-openssl-bc-leak and once that is green (enough) I will merge to master.

please also have a look at bd519e7 since that was a conflict which I am not sure in which way to resolve it. with this commit it follows master.

@mkristian mkristian closed this Apr 23, 2014
@kares
JRuby Team member

thanks! ... I see that BN revert is actually something we do not want (not 1.6.8 nor 1.7.x compatible) ...
just take the latest of what's in on my side kares@0e69ad0 (it's been reviewed and I just did look again)

@mkristian
JRuby Team member
@kares
JRuby Team member

ClassIndex is not available all the way down to JRuby 1.6 ... had issues with it with non-recent 1.7 as well.
it's pretty straightforward to re-implement without the ClassIndex ... not worth the hustle of keeping it (you should have seen a failure already esp. if you're compiling against 1.6.8's API) ... the rebased versions gets a lot of failures but here's how it went for me with an older base commit https://travis-ci.org/kares/jruby/builds/23575841

@mkristian
JRuby Team member
@kares
JRuby Team member

let me know how I can help - all should be fine, however with the new base I can not test the old way either - due to mvn now complaining (have not been looking into that as I started writing some repo local tests to fix the issue and it's much simpler to test that way ... should have not modified openssl imported tests anyways - no worries there I'll submit another PR for this as this one gets handled).

@mkristian
JRuby Team member

do you mean the mvn test does not run the junit tests ? hmm - that is my fault. I will fix this. I also reverted that the commit of mine back to how it was from the PR

@kares
JRuby Team member

@mkristian no that's not that important I meant mvn -P main -rf ... from #1543 (comment) seemed no longer usable (but as I mentioned I do not need it and travis clearly has some other issues as is)

@enebo
JRuby Team member

@kares There are a few things that should be addressed in this fix and most of them are actually the same thing. I am not sure how many of these public Java methods are used beyond their @JRubyMethod bindings but removing ThreadContext broke backwards compatibility (You also renamed some initialize methods like _initialize -> bm_initialize in BM.java).

The other reason for keeping ThreadContext on these methods is that context passed through is much faster at retrieving runtime than calling getRuntime(). In fact we started phasing out uses of IRubyObject.getRuntime() from our codebase (although there are still a ton of them).

I know this is a big item but as a followup could you restore all signatures (which seem to mostly revolve around initialize) to their previous values (as a followup PR). Sorry, I wish I had gotten into this review process to save you the pain...

@mkristian This note will be useful for you to read in merging 1.7 branch stuff (and usefulness of ThreadContext).

@kares
JRuby Team member

@enebo thanks, actually I tried doing the thing you mention - pass in context (it was almost never passed) esp. with methods that use it extensively and previously did a getRuntime().getCurrentContext() ... compatibility seemed fine - tried tests against 1.6.8 (did compile all the way down to 1.6.4 - did not try further).

who else could be using the JRuby-OpenSSL @JRubyMethod Java API (if there's anything concrete you had in mind) ?

those BN changes were a bit of an accident but by the time I noticed I left it as is ... I'll be happy to restore those if it's that important. for others such as _initialize I mostly added a ThreadContext param, so those will need to get back as well (or all of them need to get back to the state where context is not passed - if so let pls do let us know why or what compatibility we need to test with these) ?

thanks again for looking into this - much appreciated, I'll be off a week or so but would be happy to get it ironed out once I'm back.

also @mkristian told me this will not need to get "back-merged" to 1_7 (or anywhere else - I actually did the merge there were no "huge" conflicts) - as releases will be done from master and the pom actually compiles against JRuby 1.6.8 + Java 6 ... maybe you guys need to talk figuring this out :)

@mkristian
JRuby Team member
@enebo
JRuby Team member

@kares I actually read the merge backwards. ignore most of my backwards compat issues. I thought this was landed on jruby-1_7 and not on master. :) The merge was just so large and I changes to ThreadContext from public signatures. Perhaps I just reversed everything in my head?

So don't bother to change anything now. Once I have completed a merge between jruby-1_7 and master I will change anything which I notice. As @mkristian said, we only release one gem from either branch (and it probably should be moved to its own repo). So long as this will load from jruby-1_7 we should probably continue to use master repo.

@enebo
JRuby Team member

There is a potential wrinkle that some APIs for native extensions might actually change in JRuby 9000...Hmmm, perhaps any new APIs we can backport to 1.7.x and then major rev jossl and it will force people to update to later 1.7 version to get ossl fixes past a particular point?

@enebo
JRuby Team member

@kares To answer your question I am unsure if anyone will directly hit the Java APIs in jossl which was my original fear that if someone does then a jossl release will break them. I guess if it seems extremely unlikely the handful of initialize methods can change name. If not we should change them back.

@kares
JRuby Team member

OK, thanks for the replies ... if anything let me know what needs more (reverting) work or if you guys need help. I tried to clean upcode as I was into it trying to understand what goes on ... and since a quite some parts depended on security APIs directly alot got changed (hopefully for the better :). There's also another PR on top of this that fixes an issue and improves ext.value performance.

@kares kares referenced this pull request in trinidad/trinidad_init_services May 14, 2014
Closed

Memory usage grows over -J-Xmx? #42

@enebo enebo added this to the JRuby 1.7.13 milestone Jun 24, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment