Deploy stage failing with "NOT_STARTED" status on synthetic before stage

[louisjimenez]

Issue Summary:

Post merge of #3333 pipelines with a deploy stage are failing.

It appears that the synthetic stage that is intended to run prior to the deploy stage fails to start and is left with the status NOT_STARTED. After a bit of debugging, it seems this is tied to the requisiteStageRefIds array containing the id of an earlier stage in the pipeline. I tested this against 1.17 and found that the same pipeline generates an empty requisiteStageRefIds array for the synthetic stage. In that case, only the parent deploy stage lists the prior stage in its requisiteStageRefIds and the pipeline executes correctly.

Here is the sample JSON for what the synthetic stage looks like on master:

stages:
[
  {
    id: "01DWFT60K9N5000JR1BMCVB79A",
    refId: "DEPLOY<1",
    type: "createServerGroup",
    name: "Deploy in us-east1",
    status: "NOT_STARTED",
    context: {},
    outputs: { },
    tasks: [ ],
    syntheticStageOwner: "STAGE_BEFORE",
    parentStageId: "01DWFT5V5FSZ1A06RNAQW8194V",
    requisiteStageRefIds: ["1"]
  }
]

And for how it appears on 1.17

stages: [
  {},
  {
    id: "01DWFNKQ5X0557J8WH2SDEAV5X",
    refId: "DEPLOY<1",
    type: "createServerGroup",
    name: "Deploy in us-east1",
    startTime: 1576778980594,
    endTime: 1576779057754,
    status: "SUCCEEDED",
    context: {},
    outputs: { },
    tasks: [],
    syntheticStageOwner: "STAGE_BEFORE",
    parentStageId: "01DWFNKPFBJ0MKY3PQX2ZZ1K1Q",
    requisiteStageRefIds: [ ]
  },
]

The NOT_STARTED status for the synthetic stage causes the parent deploy stage to be marked as TERMINAL and the execution to fail. The orca log reflects this with:

ERROR 247858 --- [ handlers-19] c.n.s.o.q.handler.CompleteStageHandler : [] Unhandled condition for stage 01DWFPSBYCR01SK1YZPY6JN3B6 of com.netflix.spinnaker.orca.pipeline.model.Execution@af536b3a.id, marking as TERMINAL. syntheticStatuses=[NOT_STARTED], taskStatuses=[SUCCEEDED], planningStatus=[], afterStageStatuses=[]

Cloud Provider(s):

GCE, EC2

Environment:

Master

Steps to Reproduce:

This can be reproduced by creating a pipeline with a deploy stage that targets gce or ec2. A simple pipeline for reproducing:

{
  "appConfig": {},
  "lastModifiedBy": "anonymous",
  "limitConcurrent": true,
  "parallel": true,
  "stageCounter": 2,
  "stages": [
    {
      "clusters": [
        {
          "account": "my-account",
          "application": "my-app",
          "associatePublicIpAddress": true,
          "authScopes": [
            "cloud.useraccounts.readonly",
            "devstorage.read_only",
            "logging.write",
            "monitoring.write"
          ],
          "automaticRestart": true,
          "availabilityZones": {
            "us-east1": [
              "us-east1-b"
            ]
          },
          "backendServiceMetadata": [],
          "canIpForward": false,
          "capacity": {
            "desired": 1,
            "max": 1,
            "min": 1
          },
          "cloudProvider": "gce",
          "disableTraffic": false,
          "disks": [
            {
              "sizeGb": 10,
              "type": "pd-ssd"
            }
          ],
          "distributionPolicy": {
            "zones": []
          },
          "freeFormDetails": "",
          "imageSource": "priorStage",
          "instanceMetadata": {
            "load-balancer-names": "my-load-balancer",
            "startup-script": "sudo apt-get update && sudo apt-get install apache2 -y"
          },
          "instanceType": "f1-micro",
          "labels": {},
          "loadBalancers": [
            "my-load-balancer"
          ],
          "minCpuPlatform": "",
          "network": "default",
          "onHostMaintenance": "MIGRATE",
          "preemptible": false,
          "provider": "gce",
          "region": "us-east1",
          "regional": false,
          "selectZones": false,
          "serviceAccountEmail": "default",
          "stack": "my-stack",
          "strategy": "",
          "subnet": "",
          "tags": [],
          "targetSize": 1,
          "zone": "us-east1-b"
        }
      ],
      "name": "Deploy",
      "refId": "DEPLOY",
      "requisiteStageRefIds": [
        "1"
      ],
      "type": "deploy"
    },
    {
      "cloudProvider": "gce",
      "cloudProviderType": "gce",
      "cluster": "gce-test",
      "credentials": "my-account",
      "moniker": {
        "app": "my-app",
        "cluster": "gce-test",
        "stack": "test"
      },
      "name": "Find Image from Cluster",
      "onlyEnabled": true,
      "refId": "1",
      "regions": [
        "us-east1"
      ],
      "requisiteStageRefIds": [],
      "selectionStrategy": "LARGEST",
      "type": "findImage"
    }
  ],
  "triggers": [],
  "updateTs": "1576778769493"
}

[pdelagrave]

@spinnakerbot assign-issue @pdelagrave

[spinnakerbot]

User @pdelagrave cannot be assigned in this repo (spinnaker/spinnaker). Are they in the correct org/team?