fix(events): Fix paused events for resources due to pausing application #842

luispollo · 2020-02-29T02:29:51Z

Introduces the following changes:

Generalizes the event model so it can be used for other "scopes" beyond resources
Modifies the database structure to repurpose the existing resource_event table into a generic event table as per the above
Introduces a new abstract ApplicationEvent base class for application-level events
Introduces new ApplicationActuationPaused and ApplicationActuationResumed events
Modifies the logic in the EventController to add a matching and "immutable" ResourceActuationPaused event for every ApplicationActuationPaused in the event history in the right position of the timestamp-ordered list

Sample table records:

scope        uid                          type                           timestamp
RESOURCE     01E26DWCV4STWP39WQ06RPA8AJ   "ResourceValid"                2020-02-29 02:24:57
RESOURCE     01E26DWCV4STWP39WQ06RPA8AJ   "ResourceActuationResumed"     2020-02-29 02:24:40
APPLICATION  serverlablpollo              "ApplicationActuationResumed"  2020-02-29 02:24:37
APPLICATION  serverlablpollo              "ApplicationActuationPaused"   2020-02-29 02:23:57
RESOURCE     01E26DWCV4STWP39WQ06RPA8AJ   "ResourceValid"                2020-02-29 01:48:00
RESOURCE     01E26DWCV4STWP39WQ06RPA8AJ   "ResourceActuationResumed"     2020-02-29 01:47:52
APPLICATION  serverlablpollo              "ApplicationActuationResumed"  2020-02-29 01:47:48
APPLICATION  serverlablpollo              "ApplicationActuationPaused"   2020-02-29 01:39:04
RESOURCE     01E26DWCV4STWP39WQ06RPA8AJ   "ResourceValid"                2020-02-28 17:35:42
RESOURCE     01E26DWCV4STWP39WQ06RPA8AJ   "ResourceCreated"              2020-02-28 17:35:37

Matching API response:

---
- type: "ResourceValid"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  timestamp: "2020-02-29T02:24:56.684Z"
  scope: "RESOURCE"
- type: "ResourceActuationResumed"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  timestamp: "2020-02-29T02:24:39.906Z"
  scope: "RESOURCE"
- type: "ResourceActuationPaused"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  reason: "Resource actuation paused at the application level"
  timestamp: "2020-02-29T02:23:57.265Z"
  scope: "RESOURCE"
- type: "ResourceValid"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  timestamp: "2020-02-29T01:47:59.870Z"
  scope: "RESOURCE"
- type: "ResourceActuationResumed"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  timestamp: "2020-02-29T01:47:51.579Z"
  scope: "RESOURCE"
- type: "ResourceActuationPaused"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  reason: "Resource actuation paused at the application level"
  timestamp: "2020-02-29T01:39:03.582Z"
  scope: "RESOURCE"
- type: "ResourceValid"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  timestamp: "2020-02-28T17:35:42.055Z"
  scope: "RESOURCE"
- type: "ResourceCreated"
  apiVersion: "titus.spinnaker.netflix.com/v1"
  kind: "cluster"
  id: "titus:cluster:titustestvpc:serverlablpollo"
  application: "serverlablpollo"
  timestamp: "2020-02-28T17:35:37.426Z"
  scope: "RESOURCE"

Closes #837
Closes #634

keel-sql/src/main/resources/db/changelog/20200228-multi-scope-event-table.yml

luispollo · 2020-03-03T01:23:11Z

@lorin If you want to review... (I think someone with admin permissions needs to add you to the org -- @robzienert @ajordens?)

luispollo · 2020-03-03T01:24:16Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/PersistentEvent.kt

+import java.time.Clock
+import java.time.Instant
+
+abstract class PersistentEvent {


This is the new base class for all persistent events. It defines the common properties I expect to see in all such events regardless of the sub-class.

Again I find the name to be a bit confusing... maybe it's just me :) but can't it be just event? curios to hear other opinions.

luispollo · 2020-03-03T01:24:58Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/ResourceEvent.kt

    resource.apiVersion,
    resource.kind,
    resource.id,
    resource.application,
    reason,
    clock.instant()
  )
+
+  constructor(resource: Resource<*>, reason: String? = null, timestamp: Instant) : this(


Added to allow the synthetic events of this type to be added with a specific timestamp in EventController.

If I understand my kotlin correctly, I think you can avoid defining a second constructor by using default arguments.

constructor(resource: Resource<*>, reason: String? = null, timestamp: Instant = clock.instant()) : this(

(I haven't figured out how to suggest a change as a diff yet).

luispollo · 2020-03-03T01:25:31Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/pause/ActuationPauser.kt

@@ -31,7 +34,7 @@ import org.springframework.context.ApplicationEventPublisher
 import org.springframework.stereotype.Component

 @Component
-class ResourcePauser(
+class ActuationPauser(


Renamed this since it handles pausing of things other than resources.

luispollo · 2020-03-03T01:26:34Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/persistence/ResourceRepository.kt

+   * @param application the name of the application.
+   * @param limit the maximum number of events to return.
+   */
+  fun applicationEventHistory(application: String, limit: Int = DEFAULT_MAX_EVENTS): List<ApplicationEvent>


We don't have an ApplicationRepository since we don't store application metadata, so I decided to add these here for now. Let me know if there's a better place.

luispollo · 2020-03-03T01:32:02Z

keel-sql/keel-sql.gradle.kts

@@ -4,6 +4,7 @@ plugins {
  `java-library`
  id("kotlin-spring")
  id("ch.ayedo.jooqmodelator") version "3.6.0"
+  id("org.liquibase.gradle") version "2.0.2"


I added this so we can run liquibase migration stuff from the command line.

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlResourceRepository.kt

keel-web/src/main/kotlin/com/netflix/spinnaker/keel/rest/EventController.kt

luispollo · 2020-03-03T16:44:17Z

...-core-test/src/main/kotlin/com/netflix/spinnaker/keel/persistence/ResourceRepositoryTests.kt

@@ -201,28 +202,27 @@ abstract class ResourceRepositoryTests<T : ResourceRepository> : JUnit5Minutests
          context("updating the state again") {
            before {
              tick()
-              // TODO: ensure persisting a map with actual data
-              subject.appendHistory(ResourceDeltaDetected(resource, emptyMap(), clock))
+              subject.appendHistory(ResourceValid(resource, clock))


This was unrelated, but ResourceDeltaDetected has ignoreRepeatedInHistory set to false, so I changed this to an event that had it set to true.

robfletcher

I think this looks good. Couple of minor comments/questions

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/ApplicationEvent.kt

robfletcher · 2020-03-03T22:14:29Z

.../src/main/kotlin/com/netflix/spinnaker/keel/persistence/memory/InMemoryResourceRepository.kt

-        events.remove(it)
+        resourceEvents.remove(it)


Should we remove application level events too?

I thought about that, but wasn't 100% sure we weren't using application references elsewhere (meaning, those events might still be relevant even if all the resources were removed). Also, the only place this method is called is from deleteByApplication, which in turn is only called by CombinedRepository.deleteResourcesByApplication, which, in turn... is not called from anywhere.

I think this may have been a side-effect of the change Emily made from delete-by-application to be delete-by-delivery-config. We might be able to remove this method altogether...

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlResourceRepository.kt

keel-sql/src/main/resources/db/changelog/20200228-multi-scope-event-table.yml

keel-web/src/main/kotlin/com/netflix/spinnaker/keel/rest/EventController.kt

lorin

I had a few comments and questions, but nothing that would block a merge.

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/PersistentEvent.kt

lorin · 2020-03-03T22:20:31Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/ResourceEvent.kt

    resource.apiVersion,
    resource.kind,
    resource.id,
    resource.application,
    reason,
    clock.instant()
  )
+
+  constructor(resource: Resource<*>, reason: String? = null, timestamp: Instant) : this(


If I understand my kotlin correctly, I think you can avoid defining a second constructor by using default arguments.

constructor(resource: Resource<*>, reason: String? = null, timestamp: Instant = clock.instant()) : this(

(I haven't figured out how to suggest a change as a diff yet).

lorin · 2020-03-03T23:05:43Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/persistence/ResourceRepository.kt

+   * @param application the name of the application.
+   * @param downTo the time of the oldest event to return.
+   */
+  fun applicationEventHistory(application: String, downTo: Instant): List<ApplicationEvent>


My instinct is to add a limit parameter to all API calls that return a list of objects, just like there is for the previous call.

I decided not to add this now because the whole point of this variant of the method was not to be bound by an arbitrary limit and instead make sure we look into the application event history in the exact same time window as the resource events we're returning. Hopefully, since those are subject to a limit, the number of application events returned will not be a problem.

lorin · 2020-03-03T23:07:32Z

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/persistence/ResourceRepository.kt

+   * Records an event associated with an application.
+   * TODO: adding this here as there's no ApplicationRepository or EventRepository, but might want to move it.
+   */
+  fun appendHistory(event: ApplicationEvent)


Is there an advantage to having separate appendHistory calls for the different subclasses of PersistentEvent versus just having one method for the base class:

fun appendHistory(event: PersistentEvent)

I actually tried to merge them, but the issue I ran into is that we are dynamically determining the value to insert in the uid column for ResourceEvents by doing a select into the resources table, which prevents adding events for resources that don't exist in the database (which is a good thing since we don't have FKs anywhere). We could add the uid to insert as a parameter, but I think this would complicate life unnecessarily for callers.

Maybe worth a refactor in a separate PR?

.../src/main/kotlin/com/netflix/spinnaker/keel/persistence/memory/InMemoryResourceRepository.kt

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlResourceRepository.kt

luispollo · 2020-03-04T01:56:06Z

.../src/main/kotlin/com/netflix/spinnaker/keel/persistence/memory/InMemoryResourceRepository.kt

  private val lastCheckTimes = mutableMapOf<String, Instant>()

-  override fun deleteByApplication(application: String): Int {


This was not used anywhere...

luispollo · 2020-03-04T01:56:36Z

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlResourceRepository.kt

@@ -35,37 +40,6 @@ open class SqlResourceRepository(
  private val sqlRetry: SqlRetry
 ) : ResourceRepository {

-  override fun deleteByApplication(application: String): Int {


This was not used anywhere...

luispollo · 2020-03-04T02:04:28Z

@robfletcher Hopefully this is in decent shape now, if you want to take a final look.

gal-yardeni · 2020-03-04T17:10:28Z

btw, here's the issue for this: #634

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/KeelApplicationEvent.kt

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/KeelApplicationEventListener.kt

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/persistence/CombinedRepository.kt

gal-yardeni · 2020-03-04T17:25:48Z

Look good to me and makes total sense. Small comments about naming :)

…lication events

gal-yardeni

Thanks for addressing my feedback!

luispollo requested review from asher, robfletcher and erikmunson February 29, 2020 02:29

erikmunson mentioned this pull request Feb 29, 2020

feat(core/managed): poll for new history events in modal spinnaker/deck#7985

Merged

luispollo force-pushed the fix-paused-events branch from 2c4ec99 to 3d45d73 Compare March 2, 2020 03:41

luispollo commented Mar 2, 2020

View reviewed changes

keel-sql/src/main/resources/db/changelog/20200228-multi-scope-event-table.yml Show resolved Hide resolved

luispollo force-pushed the fix-paused-events branch from 184a000 to a491c54 Compare March 3, 2020 01:21

luispollo marked this pull request as ready for review March 3, 2020 01:21

luispollo commented Mar 3, 2020

View reviewed changes

keel-sql/src/main/kotlin/com/netflix/spinnaker/keel/sql/SqlResourceRepository.kt Outdated Show resolved Hide resolved

luispollo commented Mar 3, 2020

View reviewed changes

keel-web/src/main/kotlin/com/netflix/spinnaker/keel/rest/EventController.kt Outdated Show resolved Hide resolved

luispollo requested a review from gal-yardeni March 3, 2020 01:36

luispollo self-assigned this Mar 3, 2020

luispollo commented Mar 3, 2020

View reviewed changes

robfletcher reviewed Mar 3, 2020

View reviewed changes

lorin approved these changes Mar 3, 2020

View reviewed changes

luispollo force-pushed the fix-paused-events branch 2 times, most recently from 2471d58 to 9e05ea0 Compare March 4, 2020 01:25

luispollo commented Mar 4, 2020

View reviewed changes

gal-yardeni reviewed Mar 4, 2020

View reviewed changes

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/KeelApplicationEvent.kt Outdated Show resolved Hide resolved

gal-yardeni reviewed Mar 4, 2020

View reviewed changes

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/events/KeelApplicationEventListener.kt Outdated Show resolved Hide resolved

gal-yardeni reviewed Mar 4, 2020

View reviewed changes

keel-core/src/main/kotlin/com/netflix/spinnaker/keel/persistence/CombinedRepository.kt Outdated Show resolved Hide resolved

luispollo added 14 commits March 4, 2020 09:51

feat(events): Generalize event model/table

8fd2cac

chore(dev): Add liquibase gradle plugin

8c6d8ed

feat(events): First cut at refactoring resource paused events via app…

2c8cc17

…lication events

fix(pr): No need for exception when app event history is empty

67360e8

fix(pr): Retrieve app paused history based on oldest resource event

1b76c9b

fix(pr): Ignore event scope in JSON

5fd148f

test(pr): Add tests and required fixes in InMemoryResourceRepository

531f6e6

fix(pr): Fix InMemoryResourceRepository + tests

d65d4eb

fix(pr): Remove unused method deleteByApplication

f84525e

Rename ApplicationEvent to KeelApplicationEvent

19b2367

Simplify event constructors

c68fbf5

Address review feedback

c66898c

Add transactions around event inserts

8d8f037

Move logic out of controller

24bd007

luispollo force-pushed the fix-paused-events branch from 3f5e9e8 to 24bd007 Compare March 4, 2020 17:51

luispollo added 2 commits March 4, 2020 10:37

Rename downTo to until

8e1cae4

Undo renaming of ApplicationEvent

29d836e

gal-yardeni approved these changes Mar 4, 2020

View reviewed changes

luispollo added the ready to merge Approved and ready for merge label Mar 4, 2020

mergify bot merged commit f099afa into spinnaker:master Mar 4, 2020

mergify bot added the auto merged label Mar 4, 2020

luispollo deleted the fix-paused-events branch March 4, 2020 20:26

This was referenced Mar 6, 2020

feat(api): API improvements for environments/artifacts UI #848

Merged

fix(events): Fix issue with synthetic resource paused events in API #853

Closed

fix(events): Revisit reflecting application paused/resumed events on resources #865

Merged

lorin mentioned this pull request Apr 11, 2020

REQUEST: New Reviewer status for lorin spinnaker/governance#117

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(events): Fix paused events for resources due to pausing application #842

fix(events): Fix paused events for resources due to pausing application #842

luispollo commented Feb 29, 2020 •

edited

luispollo commented Mar 3, 2020

luispollo Mar 3, 2020

gal-yardeni Mar 4, 2020

luispollo Mar 3, 2020

lorin Mar 3, 2020

luispollo Mar 3, 2020

luispollo Mar 3, 2020

luispollo Mar 3, 2020

luispollo Mar 3, 2020

robfletcher left a comment

robfletcher Mar 3, 2020

luispollo Mar 3, 2020

lorin left a comment •

edited

lorin Mar 3, 2020

lorin Mar 3, 2020

luispollo Mar 4, 2020

lorin Mar 3, 2020

luispollo Mar 3, 2020 •

edited

luispollo Mar 4, 2020

luispollo Mar 4, 2020

luispollo commented Mar 4, 2020

gal-yardeni commented Mar 4, 2020

gal-yardeni commented Mar 4, 2020

gal-yardeni left a comment

		private val lastCheckTimes = mutableMapOf<String, Instant>()

		override fun deleteByApplication(application: String): Int {

fix(events): Fix paused events for resources due to pausing application #842

fix(events): Fix paused events for resources due to pausing application #842

Conversation

luispollo commented Feb 29, 2020 • edited

luispollo commented Mar 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robfletcher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorin left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luispollo Mar 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luispollo commented Mar 4, 2020

gal-yardeni commented Mar 4, 2020

gal-yardeni commented Mar 4, 2020

gal-yardeni left a comment

Choose a reason for hiding this comment

luispollo commented Feb 29, 2020 •

edited

lorin left a comment •

edited

luispollo Mar 3, 2020 •

edited