Keywhiz 0.10.2 having error "java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider" ...

[FlavioImbertDomingos]

I was able to compile the latest version of Keywhiz 0.10.2 successfully.

But when I try to run it, I got an error: "java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider"

Do you know why that happen and how to fix it?

$ java -javaagent:/opt/jolokia-jvm-1.3.6-agent.jar -jar /opt/keywhiz-server-0.10.2-SNAPSHOT-shaded.jar server /opt/keywhiz-master/server/target/classes/keywhiz-stg.yml
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL http://127.0.0.1:8778/jolokia/
java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider
at keywhiz.ServiceModule.configure(ServiceModule.java:63)
at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:344)
at com.google.inject.spi.Elements.getElements(Elements.java:103)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:137)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.google.inject.Guice.createInjector(Guice.java:69)
at com.google.inject.Guice.createInjector(Guice.java:59)
at keywhiz.KeywhizService.run(KeywhizService.java:114)
at keywhiz.KeywhizService.run(KeywhizService.java:71)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:93)
at keywhiz.KeywhizService.main(KeywhizService.java:78)
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 16 more
[keywhiz@keywhiz-233048331-1-258246470 ~]$

$ java --version
openjdk 11.0.3 2019-04-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.3+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7-LTS, mixed mode, sharing)
[keywhiz@keywhiz-233048331-1-258246470 ~]$

Any idea?

Thank you in advance,

Flavio.

[mbyczkowski]

@FlavioImbertDomingos Looks like you're missing the BouncyCastle jar? Has anything changed in your Keywhiz set-up since 0.8? It might be helpful if you could show your YAML config.

@alokmenghrajani I'm not super familiar with BC use in Keywhiz, do you know if we have always excluded BC from the shaded jar?

[alokmenghrajani]

BouncyCastle (BC) is excluded from the shaded jar because people with HSMs (or cloud keys or whatever specific setup you might have) will want to provide their own provider (the exclusion happens here: https://github.com/square/keywhiz/blob/master/pom.xml#L481).

In addition, with older versions of Java, we had to sign providers and including signed jars in a shaded jar would break the signature. With newer versions of Java, we can include BC and figure out a way to provide an override when needed.

I'll have to investigate why NoClassDefFoundError is being triggered here. @FlavioImbertDomingos are you running the code on the same machine you built it?

[FlavioImbertDomingos]

Hi Guys!
Thanks so much for jumping on this. I got so busy with a lot of stuff and could not check in before. So, let me answer the question:

@FlavioImbertDomingos are you running the code on the same machine you built it?

No, I initially tried to build in the same machine I am running Keywhiz 0.8 in STG running the same machine/version/code in prod. But I had so many issues trying to install Java 11 there and Maven that I decided to build a new fresh machine. So I built a new CentOS machine with JDK Java 11 installed without issues and Maven as well. So, I built the Keywhiz binary there, different machine, with vanilla Java 11 installed.

I am I understanding, I may be missing (have to download) some Java libraries (those restrict ones)?

[FlavioImbertDomingos]

Hi @alokmenghrajani

I know you are busy man :) Anyways, is there anything I can do to help solve this issue?

Thank you so much!

[FlavioImbertDomingos]

I guess I understand now. I will remove the line that exclude BouncyCastle and compile it version again.

Thank you for the information.

[mbyczkowski]

@FlavioImbertDomingos you can also just download a jar from https://www.bouncycastle.org/latest_releases.html and make sure you add it to the list of jars that you use to launch keywhiz with

[FlavioImbertDomingos]

@mbyczkowski
It make sense :) I am going to build a new one sometime this week.
Thank you again Mat!

[alokmenghrajani]

Sorry for the delay. @mbyczkowski's answer should work 🤞

We could add instructions to our docs re building + then running the server on another machine.

[FlavioImbertDomingos]

no problem man! You guys ROCK!

Anyways, I don't think I will have time to focus on it this week. Maybe next week I will focus on Keywhiz upgrade project ;)

Have an amazing week. Flavio.

[FlavioImbertDomingos]

I was able to compile with the missing BouncyCastle. I guess my issue now is just update my HSM client to be compatible with Java 11... I will ask the team that manage the HSM to help with that. Hopefully it fix the issue bellow ;)

$ java -javaagent:/opt/jolokia-jvm-1.3.6-agent.jar -jar ./keywhiz-server-0.10.2-SNAPSHOT-shaded.jar server ./keywhiz-stg.yml
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL http://127.0.0.1:8778/jolokia/
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/keywhiz-server-0.10.2-SNAPSHOT-shaded.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1. Error in custom provider, java.lang.RuntimeException: java.lang.ClassNotFoundException: com.safenetinc.luna.provider.LunaProvider
   at keywhiz.service.crypto.CryptoModule.derivationProvider(CryptoModule.java:92) (via modules: keywhiz.ServiceModule -> keywhiz.service.crypto.CryptoModule)
   while locating java.security.Provider annotated with @keywhiz.service.crypto.CryptoModule$Derivation()
   for the 1st parameter of keywhiz.service.crypto.CryptoModule.baseDerivationKey(CryptoModule.java:60)
   at keywhiz.service.crypto.CryptoModule.baseDerivationKey(CryptoModule.java:60) (via modules: keywhiz.ServiceModule -> keywhiz.service.crypto.CryptoModule)
   while locating javax.crypto.SecretKey annotated with @keywhiz.service.crypto.CryptoModule$Derivation()
   for the 1st parameter of keywhiz.service.crypto.ContentCryptographer.(ContentCryptographer.java:80)
   while locating keywhiz.service.crypto.ContentCryptographer
   for the 1st parameter of keywhiz.service.crypto.RowHmacGenerator.(RowHmacGenerator.java:32)
   at keywhiz.service.crypto.RowHmacGenerator.class(RowHmacGenerator.java:32)
   while locating keywhiz.service.crypto.RowHmacGenerator
   for the 4th parameter of keywhiz.service.daos.ClientDAO$ClientDAOFactory.(ClientDAO.java:161)
   while locating keywhiz.service.daos.ClientDAO$ClientDAOFactory
   for the 1st parameter of keywhiz.service.providers.ClientAuthFactory.(ClientAuthFactory.java:50)
   while locating keywhiz.service.providers.ClientAuthFactory

1 error
at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:226)
at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1053)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1086)
at keywhiz.KeywhizService.run(KeywhizService.java:129)
at keywhiz.KeywhizService.run(KeywhizService.java:71)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:93)
at keywhiz.KeywhizService.main(KeywhizService.java:78)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.safenetinc.luna.provider.LunaProvider
at com.google.common.base.Throwables.propagate(Throwables.java:241)
at keywhiz.service.crypto.CryptoModule.derivationProvider(CryptoModule.java:100)
at keywhiz.service.crypto.CryptoModule$$FastClassByGuice$$c00ba834.invoke()
at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:173)
at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:173)
at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
... 8 more
Caused by: java.lang.ClassNotFoundException: com.safenetinc.luna.provider.LunaProvider
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at keywhiz.service.crypto.CryptoModule.derivationProvider(CryptoModule.java:92)
... 48 more

$ java --version
openjdk 11.0.4 2019-07-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing)