InvalidTypeIdException thrown after kork version bump

[lorin]

Starting with kork version 7.35.0, keel regularly throws the following exception if run against the prestaging database:

com.fasterxml.jackson.databind.exc.InvalidTypeIdException:
Could not resolve type id 'depends-on' as a subtype of 
`com.netflix.spinnaker.keel.api.Constraint`: known type ids = [] at 
[Source: (String)"[
  {"type": "depends-on", "environment": "pretest"},
  {"type": "manual-judgement", "timeout": "PT168H"}
 ]"; line: 1, column: 11]
(through reference chain: java.util.HashSet[0])

Complete stack trace

org.jooq.exception.DataAccessException: Rollback caused
	at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$0(DefaultDSLContext.java:578)
	at org.jooq.impl.Tools$12$1.block(Tools.java:4843)
	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
	at org.jooq.impl.Tools$12.get(Tools.java:4840)
	at org.jooq.impl.DefaultDSLContext.transactionResult0(DefaultDSLContext.java:588)
	at org.jooq.impl.DefaultDSLContext.transactionResult(DefaultDSLContext.java:505)
	at com.netflix.spinnaker.keel.sql.JooqKt.inTransaction(Jooq.kt:10)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$itemsDueForCheck$1.invoke(SqlDeliveryConfigRepository.kt:913)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$itemsDueForCheck$1.invoke(SqlDeliveryConfigRepository.kt:52)
	at com.netflix.spinnaker.keel.sql.SqlRetry$sam$java_util_function_Supplier$0.get(SqlRetry.kt)
	at io.github.resilience4j.retry.Retry.lambda$decorateSupplier$2(Retry.java:177)
	at io.vavr.control.Try.of(Try.java:75)
	at io.vavr.control.Try.ofSupplier(Try.java:92)
	at com.netflix.spinnaker.keel.sql.SqlRetry.withRetry(SqlRetry.kt:24)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository.itemsDueForCheck(SqlDeliveryConfigRepository.kt:912)
	at com.netflix.spinnaker.keel.persistence.CombinedRepository.deliveryConfigsDueForCheck(CombinedRepository.kt:296)
	at com.netflix.spinnaker.keel.persistence.CombinedRepository$$FastClassBySpringCGLIB$$9cfcaa57.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:685)
	at com.netflix.spinnaker.keel.persistence.CombinedRepository$$EnhancerBySpringCGLIB$$df998215.deliveryConfigsDueForCheck(<generated>)
	at com.netflix.spinnaker.keel.actuation.CheckScheduler$checkEnvironments$job$1$1.invokeSuspend(CheckScheduler.kt:110)
	at com.netflix.spinnaker.keel.actuation.CheckScheduler$checkEnvironments$job$1$1.invoke(CheckScheduler.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
	at kotlinx.coroutines.SupervisorKt.supervisorScope(Supervisor.kt:53)
	at com.netflix.spinnaker.keel.actuation.CheckScheduler$checkEnvironments$job$1.invokeSuspend(CheckScheduler.kt:108)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'depends-on' as a subtype of `com.netflix.spinnaker.keel.api.Constraint`: known type ids = []
 at [Source: (String)"[{"type": "depends-on", "environment": "pretest"}, {"type": "manual-judgement", "timeout": "PT168H"}]"; line: 1, column: 11] (through reference chain: java.util.HashSet[0])
	at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43)
	at com.fasterxml.jackson.databind.DeserializationContext.invalidTypeIdException(DeserializationContext.java:1761)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1268)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:290)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:162)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:288)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3214)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3197)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$attachDependents$$inlined$let$lambda$1$1.map(SqlDeliveryConfigRepository.kt:995)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$attachDependents$$inlined$let$lambda$1$1.map(SqlDeliveryConfigRepository.kt:52)
	at org.jooq.impl.ResultImpl.map(ResultImpl.java:1317)
	at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:1564)
	at org.jooq.impl.SelectImpl.fetch(SelectImpl.java:3764)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$attachDependents$$inlined$let$lambda$1.invoke(SqlDeliveryConfigRepository.kt:371)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$attachDependents$$inlined$let$lambda$1.invoke(SqlDeliveryConfigRepository.kt:52)
	at com.netflix.spinnaker.keel.sql.SqlRetry$sam$java_util_function_Supplier$0.get(SqlRetry.kt)
	at io.github.resilience4j.retry.Retry.lambda$decorateSupplier$2(Retry.java:177)
	at io.vavr.control.Try.of(Try.java:75)
	at io.vavr.control.Try.ofSupplier(Try.java:92)
	at com.netflix.spinnaker.keel.sql.SqlRetry.withRetry(SqlRetry.kt:35)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository.attachDependents(SqlDeliveryConfigRepository.kt:366)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository.get(SqlDeliveryConfigRepository.kt:349)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$itemsDueForCheck$1$1$2.map(SqlDeliveryConfigRepository.kt:933)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$itemsDueForCheck$1$1$2.map(SqlDeliveryConfigRepository.kt:52)
	at org.jooq.impl.ResultImpl.map(ResultImpl.java:1317)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$itemsDueForCheck$1$1.invoke(SqlDeliveryConfigRepository.kt:932)
	at com.netflix.spinnaker.keel.sql.SqlDeliveryConfigRepository$itemsDueForCheck$1$1.invoke(SqlDeliveryConfigRepository.kt:52)
	at com.netflix.spinnaker.keel.sql.JooqKt$inTransaction$1.run(Jooq.kt:11)
	at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$0(DefaultDSLContext.java:536)
	... 30 common frames omitted

Diagnosis: interaction with SpinnakerPluginManager

The SpinnakerPluginManager is instantiating the KeelApplication bean too early in the life cycle, leading to the @PostConstuct annotated methods of KeelApplication to not be invoked.

Recommended fix by @robzienert is to move all extraneous logic away from the KeelApplication class and onto a different class that is not the main entrypoint.

[lorin]

@jonsie Do you know what was introduced in kork 7.35.0 that could affect deserialization behavior in keel?

[luispollo]

I suspected a change in the version of the jackson libraries. I looked at that kork PR and didn't see anything obvious that might cause our version to change, but when looking at our gradle scripts we're not pinning the version of those libs anywhere, which means they could simply have changed because a new version was published and no one would notice. But... we're on 2.10.3, which was released on March 2, so it's been there for a while.

FWIW, the versions of jackson from all the dependencies we pull are all over the place, so I still think it might be worth digging a little more. Also, the fact that we're not pinning the version, while ensuring that we stay up-to-date, is bound to cause us problems even if the library authors are very good at sticking with semantic versioning.

Example:

> Task :keel-web:dependencyInsight
com.fasterxml.jackson.core:jackson-databind:2.10.3
   variant "runtime" [
      org.gradle.status                   = release (not requested)
      org.gradle.usage                    = java-runtime (not requested)
      org.gradle.libraryelements          = jar (not requested)
      org.gradle.category                 = library (not requested)
      Requested attributes not found in the selected variant:
         org.jetbrains.kotlin.platform.type  = jvm
         org.jetbrains.kotlin.localToProject = local to :keel-web
   ]
   Selection reasons:
      - By constraint
      - By conflict resolution : between versions 2.10.3, 2.8.9, 2.4.3, 2.10.1, 2.10.2 and 2.9.4

[lorin]

This should get fixed by #1017.

[lorin]

Not fixed by #1017. Still debugging.

[lorin]

It looks like the keelApplication bean is now being instantiated by the SpinnakerPluginManager, where it wasn't before. See the following stack traces

[lorin]

The issue has been diagnosed as an interaction between Keel and the Kork plugin manager. @luispollo is working on a fix.