front50 failing to start following upgrade to 1.31.0 on debian

[pembertonrw]

Issue Summary:

I have upgraded my Spinnaker instance from 1.30.2 to 1.31.0 and front50 is failing to start due to Caused by: java.lang.NoClassDefFoundError: com/google/api/services/storage/model/HmacKeyMetadata

It might be similar to this issue spinnaker/front50#1103 that was fixed previously

Environment:

Spinnaker running on Debian
We have Spinnaker running on a debian instance and reading files from GCS.

Steps to Reproduce:

Setup instance running on Debian and using GCS to store files

Additional Details:

`Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-08-04 16:49:50.122 ERROR 15704 --- [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'v2PipelineTemplateController': Unsatisfied dependency expressed through field 'pipelineTemplateDAO'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pipelineTemplateDAO' defined in class path resource [com/netflix/spinnaker/front50/config/CommonStorageServiceDAOConfig.class]: Unsatisfied dependency expressed through method 'pipelineTemplateDAO' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultGoogleCloudStorageService' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Unsatisfied dependency expressed through method 'defaultGoogleCloudStorageService' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.20.jar:5.3.20]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.20.jar:5.3.20]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.14.jar:2.5.14]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:780) ~[spring-boot-2.5.14.jar:2.5.14]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:453) ~[spring-boot-2.5.14.jar:2.5.14]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:343) ~[spring-boot-2.5.14.jar:2.5.14]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:143) ~[spring-boot-2.5.14.jar:2.5.14]
at com.netflix.spinnaker.front50.Main.main(Main.java:50) ~[front50-web-2.29.0.jar:2.29.0]
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pipelineTemplateDAO' defined in class path resource [com/netflix/spinnaker/front50/config/CommonStorageServiceDAOConfig.class]: Unsatisfied dependency expressed through method 'pipelineTemplateDAO' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultGoogleCloudStorageService' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Unsatisfied dependency expressed through method 'defaultGoogleCloudStorageService' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:656) ~[spring-beans-5.3.20.jar:5.3.20]
... 19 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultGoogleCloudStorageService' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Unsatisfied dependency expressed through method 'defaultGoogleCloudStorageService' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.20.jar:5.3.20]
... 32 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:597) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.20.jar:5.3.20]
... 46 common frames omitted
Caused by: java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:485) ~[spring-core-5.3.20.jar:5.3.20]
at org.springframework.util.ReflectionUtils.doWithLocalMethods(ReflectionUtils.java:321) ~[spring-core-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.buildLifecycleMetadata(InitDestroyAnnotationBeanPostProcessor.java:232) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.findLifecycleMetadata(InitDestroyAnnotationBeanPostProcessor.java:210) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(InitDestroyAnnotationBeanPostProcessor.java:149) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(CommonAnnotationBeanPostProcessor.java:305) ~[spring-context-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyMergedBeanDefinitionPostProcessors(AbstractAutowireCapableBeanFactory.java:1116) ~[spring-beans-5.3.20.jar:5.3.20]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.3.20.jar:5.3.20]
... 56 common frames omitted
Caused by: java.lang.NoClassDefFoundError: com/google/api/services/storage/model/HmacKeyMetadata
at java.base/java.lang.Class.getDeclaredMethods0(Native Method) ~[na:na]
at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166) ~[na:na]
at java.base/java.lang.Class.getDeclaredMethods(Class.java:2309) ~[na:na]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:467) ~[spring-core-5.3.20.jar:5.3.20]
... 63 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.google.api.services.storage.model.HmacKeyMetadata
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
... 67 common frames omitted`

[mattgogerly]

@j-sandy mind having a look?

[frodood]

@pembertonrw did you find a solution? I'm also facing this issue after upgrade

[pembertonrw]

@pembertonrw did you find a solution? I'm also facing this issue after upgrade

No I didn't sorry, as it's something directly controlled during the spinnaker build process on what dependancies it pulls in (I think). I did ask in the Spinnaker slack channel and one workaround suggested was porting it to use MySql - https://spinnaker.io/docs/setup/productionize/persistence/front50-sql/ (https://spinnakerteam.slack.com/archives/C091CCWRJ/p1691169058553649)

[caongocthai]

+1 Same issue on both 1.31.0 and 1.31.1

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pluginInfoController' defined in URL [jar:file:/opt/front50/lib/front50-web-2.29.1.jar!/com/netflix/spinnaker/front50/controllers/PluginInfoController.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pluginInfoService' defined in URL [jar:file:/opt/front50/lib/front50-core-2.29.1.jar!/com/netflix/spinnaker/front50/model/plugins/PluginInfoService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pluginInfoRepository' defined in class path resource [com/netflix/spinnaker/front50/config/CommonStorageServiceDAOConfig.class]: Unsatisfied dependency expressed through method 'pluginInfoRepository' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultGoogleCloudStorageService' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Unsatisfied dependency expressed through method 'defaultGoogleCloudStorageService' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@67424e82]

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@67424e82]

[mattgogerly]

Hm, I'm not sure what's going on here.

./gradlew front50-gcs:dI --dependency google-api-services-storage --configuration runtimeClasspath yields com.google.apis:google-api-services-storage:v1-rev20200326-1.30.9, which definitely has that class.

[DmitrySolodovnyk]

Same issue 1.31.1:
Caused by: java.lang.ClassNotFoundException: com.google.api.services.storage.model.HmacKeyMetadata
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[na:na]
... 81 common frames omitted

[mattgogerly]

Found the issue. spinnaker/front50#1257 removed the force on the version of google-api-services-storage and replaced it with resolutionStrategy.force for Gradle 7. For some reason the force isn't being applied. If I move it to the front50-web.gradle configuration it does apply.

@j-sandy @dbyron-sf any ideas? I'm not too versed in Gradle dependency management.

[j-sandy]

I think it may not about force isn't taking effect. As, I checked that the resolutionStrategy.force is working and the version for google-api-service-storage is same as expected and forced v1-rev20200326-1.30.9 in all configurations.

$ ./gradlew front50-gcs:dI --dependency google-api-services-storage
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :front50-gcs:dependencyInsight
com.google.apis:google-api-services-storage:v1-rev20200326-1.30.9
   variant "compile" [
      org.gradle.status                  = release (not requested)
      org.gradle.usage                   = java-api
      org.gradle.libraryelements         = jar (compatible with: classes+resources)
      org.gradle.category                = library

      Requested attributes not found in the selected variant:
         org.gradle.dependency.bundling     = external
         org.jetbrains.kotlin.platform.type = jvm
         org.gradle.jvm.version             = 11
   ]
   Selection reasons:
      - Forced
      - By constraint

com.google.apis:google-api-services-storage:v1-rev20200326-1.30.9
\--- com.google.cloud:google-cloud-storage:1.108.0
     +--- compileClasspath (requested com.google.cloud:google-cloud-storage)
     \--- io.spinnaker.kork:kork-bom:7.175.1 (requested com.google.cloud:google-cloud-storage:2.6.1)
          \--- compileClasspath

com.google.apis:google-api-services-storage:v1-rev141-1.25.0 -> v1-rev20200326-1.30.9
\--- io.spinnaker.kork:kork-bom:7.175.1
     \--- compileClasspath

com.google.apis:google-api-services-storage -> v1-rev20200326-1.30.9
\--- compileClasspath

(*) - dependencies omitted (listed previously)

[mattgogerly]

@j-sandy if you run it on front50-web it seems to be taking it from kork-artifacts, not taking in that force though. Not sure why.

[j-sandy]

IMO error logs are suggesting the issue is more localized to front50-gcs module:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'googleCloudStorage' defined in class path resource [com/netflix/spinnaker/front50/config/GcsConfig.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [com.google.cloud.storage.StorageImpl] from ClassLoader [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27]

And HmacKeyMetadata is used in front50-gcs module:

$ git grep HmacKeyMetadata
front50-gcs/src/test/kotlin/com/netflix/spinnaker/front50/model/FakeStorageRpc.kt:import com.google.api.services.storage.model.HmacKeyMetadata
front50-gcs/src/test/kotlin/com/netflix/spinnaker/front50/model/FakeStorageRpc.kt:  override fun getHmacKey(accessId: String?, options: MutableMap<Option, *>?): HmacKeyMetadata {
front50-gcs/src/test/kotlin/com/netflix/spinnaker/front50/model/FakeStorageRpc.kt:  override fun updateHmacKey(hmacKeyMetadata: HmacKeyMetadata?, options: MutableMap<Option, *>?): HmacKeyMetadata {
front50-gcs/src/test/kotlin/com/netflix/spinnaker/front50/model/FakeStorageRpc.kt:  override fun deleteHmacKey(hmacKeyMetadata: HmacKeyMetadata?, options: MutableMap<Option, *>?) {
front50-gcs/src/test/kotlin/com/netflix/spinnaker/front50/model/FakeStorageRpc.kt:  override fun listHmacKeys(options: MutableMap<Option, *>?): Tuple<String, MutableIterable<HmacKeyMetadata>> {

[j-sandy]

@mattgogerly It seems the dependency version is leaking from kork rather than coming from front50-gcs as you suggested. I am also not sure of the reason, and checked that google-api-services-storage:v1-rev141-1.25.0 doesn't have HmacKeyMetadata class :

$ ./gradlew front50-web:dI --dependency google-api-services-storage --configuration runtimeClasspath

> Task :front50-web:dependencyInsight
com.google.apis:google-api-services-storage:v1-rev141-1.25.0
   variant "runtime" [
      org.gradle.status              = release (not requested)
      org.gradle.usage               = java-runtime
      org.gradle.libraryelements     = jar
      org.gradle.category            = library

      Requested attributes not found in the selected variant:
         org.gradle.dependency.bundling = external
         org.gradle.jvm.version         = 11
   ]
   Selection reasons:
      - By constraint
      - Forced

com.google.apis:google-api-services-storage:v1-rev141-1.25.0
+--- io.spinnaker.kork:kork-bom:7.175.1
|    +--- runtimeClasspath
|    +--- project :front50-migrations
|    |    \--- runtimeClasspath
|    +--- project :front50-azure
|    |    \--- runtimeClasspath
|    +--- project :front50-gcs
|    |    \--- runtimeClasspath
|    +--- project :front50-oracle
|    |    \--- runtimeClasspath
|    +--- project :front50-redis
|    |    \--- runtimeClasspath
|    +--- project :front50-s3
|    |    \--- runtimeClasspath
|    +--- project :front50-swift
|    |    \--- runtimeClasspath
|    +--- project :front50-sql
|    |    \--- runtimeClasspath
|    +--- project :front50-core
|    |    +--- runtimeClasspath
|    |    +--- project :front50-migrations (*)
|    |    +--- project :front50-azure (*)
|    |    +--- project :front50-gcs (*)
|    |    +--- project :front50-oracle (*)
|    |    +--- project :front50-redis (*)
|    |    +--- project :front50-s3 (*)
|    |    +--- project :front50-swift (*)
|    |    \--- project :front50-sql (*)
|    +--- project :front50-api
|    |    +--- runtimeClasspath
|    |    +--- project :front50-migrations (*)
|    |    +--- project :front50-azure (*)
|    |    +--- project :front50-gcs (*)
|    |    +--- project :front50-oracle (*)
|    |    +--- project :front50-redis (*)
|    |    +--- project :front50-s3 (*)
|    |    +--- project :front50-swift (*)
|    |    +--- project :front50-sql (*)
|    |    \--- project :front50-core (*)
|    +--- project :front50-sql-mysql
|    |    \--- project :front50-sql (*)
|    \--- project :front50-sql-postgres
|         \--- project :front50-sql (*)
\--- io.spinnaker.kork:kork-secrets-gcp:7.175.1
     +--- io.spinnaker.kork:kork-bom:7.175.1 (*)
     \--- io.spinnaker.kork:kork-runtime:7.175.1
          +--- runtimeClasspath (requested io.spinnaker.kork:kork-runtime)
          \--- io.spinnaker.kork:kork-bom:7.175.1 (*)

com.google.apis:google-api-services-storage:v1-rev20220401-1.32.1 -> v1-rev141-1.25.0
\--- com.google.cloud:google-cloud-storage:2.6.1
     +--- project :front50-gcs (requested com.google.cloud:google-cloud-storage)
     |    \--- runtimeClasspath
     \--- io.spinnaker.kork:kork-bom:7.175.1
          +--- runtimeClasspath
          +--- project :front50-migrations
          |    \--- runtimeClasspath
          +--- project :front50-azure
          |    \--- runtimeClasspath
          +--- project :front50-gcs (*)
          +--- project :front50-oracle
          |    \--- runtimeClasspath
          +--- project :front50-redis
          |    \--- runtimeClasspath
          +--- project :front50-s3
          |    \--- runtimeClasspath
          +--- project :front50-swift
          |    \--- runtimeClasspath
          +--- project :front50-sql
          |    \--- runtimeClasspath
          +--- project :front50-core
          |    +--- runtimeClasspath
          |    +--- project :front50-migrations (*)
          |    +--- project :front50-azure (*)
          |    +--- project :front50-gcs (*)
          |    +--- project :front50-oracle (*)
          |    +--- project :front50-redis (*)
          |    +--- project :front50-s3 (*)
          |    +--- project :front50-swift (*)
          |    \--- project :front50-sql (*)
          +--- project :front50-api
          |    +--- runtimeClasspath
          |    +--- project :front50-migrations (*)
          |    +--- project :front50-azure (*)
          |    +--- project :front50-gcs (*)
          |    +--- project :front50-oracle (*)
          |    +--- project :front50-redis (*)
          |    +--- project :front50-s3 (*)
          |    +--- project :front50-swift (*)
          |    +--- project :front50-sql (*)
          |    \--- project :front50-core (*)
          +--- project :front50-sql-mysql
          |    \--- project :front50-sql (*)
          \--- project :front50-sql-postgres
               \--- project :front50-sql (*)

com.google.apis:google-api-services-storage -> v1-rev141-1.25.0
\--- project :front50-gcs
     \--- runtimeClasspath

I try to fix it first and then will find the RCA.

[DmitrySolodovnyk]

@j-sandy , if you are touching kork, can you please add one line :
tempFile.setReadable(true, true)
here -https://github.com/spinnaker/kork/blob/master/kork-secrets/src/main/java/com/netflix/spinnaker/kork/secrets/SecretManager.java#L96
This would make it possible to use encrypted ssh keys from secrets. At the moment permissions are too broad, ssh tools don't accept keys with 644.

[j-sandy]

The version v1-rev141-1.25.0 is leaking from kork-secret-gcp module to front50-web. So, in order to fix this issue we have to apply the resolutionStrategy.force at build.gradle.

[pembertonrw]

I have confirmed that 1.32.0 and 1.31.2 fixed this issue for me. Thanks for the fix @mattgogerly and @j-sandy it's much appreciated.