Don't keep re-trying the same bake #997

robfletcher · 2020-04-10T01:24:06Z

Closes #856

robfletcher · 2020-04-10T01:24:35Z

keel-bakery-history-sql/keel-bakery-history-sql.gradle.kts

+plugins {
+  `java-library`
+  id("kotlin-spring")
+}
+
+dependencies {
+  implementation(project(":keel-bakery-plugin"))
+  implementation(project(":keel-sql"))
+
+  testImplementation("com.netflix.spinnaker.kork:kork-sql-test")
+  testImplementation("io.strikt:strikt-core")
+  testImplementation(project(":keel-spring-test-support"))
+  testImplementation(project(":keel-test"))
+  testImplementation(project(":keel-core-test"))
+
+  testImplementation(project(":keel-web")) {
+    // avoid circular dependency which breaks Liquibase
+    exclude(module = "keel-bakery-history-sql")
+  }
+}


New module because I didn't want keel-sql to depend on keel-bakery-plugin

Wondering if I should instead have keel-bakery-plugin depend on keel-sql. Not sure. I'm not entirely happy with adding this new module.

Is it reasonable to keep all sql in one module? or does that present a lot of problems?

...-sql/src/test/kotlin/com/netflix/spinnaker/keel/bakehistory/sql/spring/SpringStartupTests.kt

robfletcher · 2020-04-10T01:25:46Z

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/BakeHistory.kt

+interface BakeHistory {
+  fun contains(appVersion: String, baseAmiVersion: String): Boolean
+  fun add(appVersion: String, baseAmiVersion: String, regions: Collection<String>, taskId: String)
+}


Simple interface to record and query whether we've baked a particular appversion / base_ami_version combination

robfletcher · 2020-04-10T01:26:01Z

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt

-          log.info("Versions or regions for {} differ, launching bake", artifact.name)
-          launchBake(artifact, latestVersion)
+          if (bakeHistory.contains(latestArtifactVersion, latestBaseAmiVersion)) {
+            log.warn("Artifact version {} and base AMI version {} were baked previously", latestArtifactVersion, latestBaseAmiVersion)
+            publisher.publishEvent(RecurrentBakeDetected(latestArtifactVersion, latestBaseAmiVersion))
+          } else {
+            log.info("Versions or regions for {} differ, launching bake", artifact.name)
+            launchBake(artifact, latestArtifactVersion)
+              .also { tasks ->
+                bakeHistory.add(latestArtifactVersion, latestBaseAmiVersion, artifact.vmOptions.regions, tasks.first().id)
+              }
+          }


Don't bake if we've done it before

luispollo

I understand what you're trying to do here, but don't we have that information in orca already? We even have the orca task IDs stored in the task_tracking table (maybe missing some columns for cross-referencing). My feeling is that we could just ask orca about the bake tasks for a particular artifact version. It could also very well be that I'm missing the point entirely. 😄

robfletcher · 2020-04-10T15:51:55Z

I understand what you're trying to do here, but don't we have that information in orca already? We even have the orca task IDs stored in the task_tracking table (maybe missing some columns for cross-referencing). My feeling is that we could just ask orca about the bake tasks for a particular artifact version. It could also very well be that I'm missing the point entirely. 😄

Can we rely on Orca not having cleaned up those tasks? I'm kind of looking at this as an interim fix anyway. It seems like the reason we keep re-baking is because we don't really have a way to handle missing regions on an AMI. I'm going to do some digging in to that but for now this should stop us spamming bake tasks.

luispollo · 2020-04-10T15:56:43Z

Can we rely on Orca not having cleaned up those tasks?

I think we only care about the short term, since we'll only be baking the latest approved version, no? In any case, our cleanup config looks like this right now (i.e. orchestration tasks live forever, apparently):

pollers:
  oldPipelineCleanup:
    enabled: false
    thresholdDays: 90
  pipelineCleanup:
    enabled: false
  orchestrationCleanup:
    enabled: false

luispollo · 2020-04-10T15:57:43Z

P.S. Not trying to discourage you, just wanted to understand if we had that information available elsewhere already.

robfletcher · 2020-04-10T16:34:45Z

P.S. Not trying to discourage you, just wanted to understand if we had that information available elsewhere already.

I don't think it's accessible in any very convenient way. I think the change to Orca to enable us to access it would maybe be bigger than this change is.

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt

emjburns

I like this approach. Hopefully this will help us figure out why we keep rebaking.

robfletcher · 2020-04-10T17:36:31Z

...lugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/telemetry/BakeryTelemetryListener.kt

+  @EventListener(RecurrentBakeDetected::class)
+  fun onRecurrentBakeDetected(event: RecurrentBakeDetected) {
+    spectator.counter(
+      RECURRENT_BAKE_DETECTED_ID,
+      listOf(
+        BasicTag("versions", "${event.appVersion}+${event.baseAmiVersion}")
+      )
+    )
+  }


@emjburns new telemetry

luispollo · 2020-04-10T20:54:28Z

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt

+            publisher.publishEvent(RecurrentBakeDetected(latestArtifactVersion, latestBaseAmiVersion))
+          } else {
+            launchBake(artifact, latestArtifactVersion)
+            diffFingerprintRepository.store(artifact.name, diff)


Great idea!

Do these ever get cleaned up from the database?

There's only ever one per resource but no

emjburns · 2020-04-10T20:54:52Z

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt

+            log.info("Image for {} delta: {}", artifact.name, diff.toDebug())
+          }
+
+          if (diffFingerprintRepository.seen(artifact.name, diff)) {


You should hash this diff, since that's what the diff fingerprint repository stores

It does that internally in the method, same as store does

LOL right. duh.

luispollo · 2020-04-10T20:58:24Z

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt

-        if (image != null && regionsDiffer) {
-          publisher.publishEvent(ImageRegionMismatchDetected(image, artifact.vmOptions.regions))
+        if (current != null && diff.affectedRootPropertyNames == setOf("regions")) {
+          publisher.publishEvent(ImageRegionMismatchDetected(current, artifact.vmOptions.regions))


Great instrumentation. We'd been missing data on the frequency of this forever. 👏

luispollo

Looks great! I like this approach much better than the original.

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlDiffFingerprintRepository.kt

keel-orca/src/main/kotlin/com/netflix/spinnaker/keel/constraints/CanaryConstraintEvaluator.kt

lorin

LGTM

robfletcher · 2020-04-10T21:57:15Z

keel-sql/src/main/resources/db/changelog/20200410-rename-diff-fingerprint-column.yml

+databaseChangeLog:
+- changeSet:
+    id: rename-diff-fingerprint-column
+    author: fletch
+    changes:
+    - renameColumn:
+        tableName: diff_fingerprint
+        oldColumnName: resource_id
+        newColumnName: entity_id
+        columnDataType: varchar(255)


@emjburns did the column name change

…w thing

emjburns

robfletcher requested a review from emjburns April 10, 2020 01:24

robfletcher commented Apr 10, 2020

View reviewed changes

...-sql/src/test/kotlin/com/netflix/spinnaker/keel/bakehistory/sql/spring/SpringStartupTests.kt Outdated Show resolved Hide resolved

robfletcher commented Apr 10, 2020

View reviewed changes

robfletcher requested review from luispollo, gal-yardeni and lorin April 10, 2020 15:17

luispollo reviewed Apr 10, 2020

View reviewed changes

robfletcher force-pushed the prevent-rebaking branch from 3f8d2c6 to a03f6ef Compare April 10, 2020 16:49

emjburns reviewed Apr 10, 2020

View reviewed changes

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt Outdated Show resolved Hide resolved

emjburns reviewed Apr 10, 2020

View reviewed changes

keel-bakery-plugin/src/main/kotlin/com/netflix/spinnaker/keel/bakery/artifact/ImageHandler.kt Show resolved Hide resolved

emjburns reviewed Apr 10, 2020

View reviewed changes

robfletcher force-pushed the prevent-rebaking branch from a03f6ef to 8c2af5d Compare April 10, 2020 17:35

robfletcher commented Apr 10, 2020

View reviewed changes

robfletcher force-pushed the prevent-rebaking branch 2 times, most recently from 0d42de7 to feeb869 Compare April 10, 2020 19:40

luispollo reviewed Apr 10, 2020

View reviewed changes

emjburns reviewed Apr 10, 2020

View reviewed changes

luispollo reviewed Apr 10, 2020

View reviewed changes

luispollo approved these changes Apr 10, 2020

View reviewed changes

lorin reviewed Apr 10, 2020

View reviewed changes

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlDiffFingerprintRepository.kt Outdated Show resolved Hide resolved

lorin reviewed Apr 10, 2020

View reviewed changes

keel-orca/src/main/kotlin/com/netflix/spinnaker/keel/constraints/CanaryConstraintEvaluator.kt Outdated Show resolved Hide resolved

lorin approved these changes Apr 10, 2020

View reviewed changes

robfletcher added 2 commits April 10, 2020 14:55

fix(baking): Don't keep re-trying the same bake

8070c0f

fix(baking): re-work ImageHandler to use a diff

6447b1a

robfletcher force-pushed the prevent-rebaking branch from feeb869 to f7dcc11 Compare April 10, 2020 21:55

robfletcher commented Apr 10, 2020

View reviewed changes

fix(baking): re-work to use DiffFingerprintRepository instead of a ne…

d2fa59c

…w thing

robfletcher force-pushed the prevent-rebaking branch from f7dcc11 to d2fa59c Compare April 10, 2020 22:04

emjburns approved these changes Apr 10, 2020

View reviewed changes

robfletcher added the ready to merge Approved and ready for merge label Apr 10, 2020

mergify bot merged commit 2ccd656 into spinnaker:master Apr 10, 2020

mergify bot added the auto merged label Apr 10, 2020

robfletcher deleted the prevent-rebaking branch April 10, 2020 22:18

lorin mentioned this pull request Apr 11, 2020

REQUEST: New Reviewer status for lorin spinnaker/governance#117

Closed

3 tasks

lorin mentioned this pull request Apr 22, 2020

REQUEST: New Approver status for lorin spinnaker/governance#121

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't keep re-trying the same bake #997

Don't keep re-trying the same bake #997

robfletcher commented Apr 10, 2020 •

edited by luispollo

robfletcher Apr 10, 2020

robfletcher Apr 10, 2020

emjburns Apr 10, 2020

robfletcher Apr 10, 2020

robfletcher Apr 10, 2020

luispollo left a comment

robfletcher commented Apr 10, 2020

luispollo commented Apr 10, 2020

luispollo commented Apr 10, 2020

robfletcher commented Apr 10, 2020

emjburns left a comment

robfletcher Apr 10, 2020

luispollo Apr 10, 2020

luispollo Apr 10, 2020

robfletcher Apr 10, 2020

emjburns Apr 10, 2020

robfletcher Apr 10, 2020

emjburns Apr 10, 2020

luispollo Apr 10, 2020

luispollo left a comment

lorin left a comment

robfletcher Apr 10, 2020

emjburns Apr 10, 2020

emjburns left a comment

Don't keep re-trying the same bake #997

Don't keep re-trying the same bake #997

Conversation

robfletcher commented Apr 10, 2020 • edited by luispollo

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luispollo left a comment

Choose a reason for hiding this comment

robfletcher commented Apr 10, 2020

luispollo commented Apr 10, 2020

luispollo commented Apr 10, 2020

robfletcher commented Apr 10, 2020

emjburns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luispollo left a comment

Choose a reason for hiding this comment

lorin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emjburns left a comment

Choose a reason for hiding this comment

robfletcher commented Apr 10, 2020 •

edited by luispollo