Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(queue): Add hydrate queue admin command #2315

Merged
merged 5 commits into from
Jul 19, 2018

Conversation

robzienert
Copy link
Member

Allows the ability to hydrate the work queue from the execution repository state. This is primarily intended for being able to recover a Redis failure while using the (upcoming) SQL backend, where losing the Redis will mean in-flight work is orphaned, but its state is still held in SQL and thus is recoverable.

A secondary use case can be made for "kicking" a stuck execution where a message was dropped and the execution is no longer progressing. Running the command against a single execution could potentially get things going again.

Example output from the API:

{
  "dryRun": true,
  "executions": {
    "01CJ3DVSNE4BG85HNNJKDT0P05": {
      "startTime": 1531271047636,
      "actions": [
        {
          "description": "Task is running and is retryable",
          "message": {
            "kind": "runTask",
            "executionType": "PIPELINE",
            "executionId": "01CJ3DVSNE4BG85HNNJKDT0P05",
            "application": "spintest",
            "stageId": "01CJ3DVSNFYA753Y7RAXEHJD72",
            "taskId": "1",
            "taskType": "com.netflix.spinnaker.orca.pipeline.tasks.WaitTask",
            "attributes": [],
            "ackTimeoutMs": 600000
          },
          "context": {
            "stageId": "01CJ3DVSNFYA753Y7RAXEHJD72",
            "stageType": "wait",
            "stageStartTime": 1531271051731,
            "taskId": "1",
            "taskType": "wait",
            "taskStartTime": 1531271051839
          }
        }
      ],
      "canApply": true
    }
  }
}

Example log output:

18:07:07.792  INFO 23198 --- [nio-8083-exec-5] c.n.s.o.q.a.HydrateQueueCommand          : [] Hydrating queue from execution repository state (dryRun: true)
18:07:07.792  INFO 23198 --- [nio-8083-exec-5] c.n.s.o.q.a.HydrateQueueCommand          : [] Hydrating execution 01CJ3DVSNE4BG85HNNJKDT0P05
18:07:07.792  INFO 23198 --- [nio-8083-exec-5] c.n.s.o.q.a.HydrateQueueCommand          : [] Hydrating execution 01CJ3DVSNE4BG85HNNJKDT0P05 with RunTask(executionType=pipeline, executionId=01CJ3DVSNE4BG85HNNJKDT0P05, application=spintest, stageId=01CJ3DVSNFYA753Y7RAXEHJD72, taskId=1, taskType=class com.netflix.spinnaker.orca.pipeline.tasks.WaitTask): Task is running and is retryable (stageId: 01CJ3DVSNFYA753Y7RAXEHJD72, stageStartTime: 1531271051731, stageType: wait, taskId: 1, taskStartTime: 1531271051839, taskType: wait)

I probably need more tests, what other use cases should I be covering?

@robzienert robzienert force-pushed the hydrate-queue branch 3 times, most recently from 6ed9e97 to 403f3bf Compare July 11, 2018 19:56
@robfletcher
Copy link
Contributor

I presume that message section is just a message that can be dropped right on the Keiko queue?

@robzienert
Copy link
Member Author

@robfletcher That's correct. We collect all of the messages with their descriptions and then just push(message[, ZERO]) them.

it("adds messages to the queue") {
argumentCaptor<Message>().let {
verify(queue, times(1)).push(it.capture())
assertType<CompleteStage>(it.firstValue) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you not just use argumentCaptor<CompleteStage> in the first place?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I can. Victimized by copy/paste.

@robzienert robzienert force-pushed the hydrate-queue branch 4 times, most recently from a9d3df5 to 5cecbcb Compare July 12, 2018 22:36
@robzienert robzienert merged commit d2adc24 into spinnaker:master Jul 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants