Core: Lighthouse-service should fail evaluation if sh.keptn.event.get-sli.finished event has result=fail

[arthurpitman]

While investigating issue keptn-contrib/dynatrace-service#603, it emerged that the lighthouse-service will pass an evaluation if the SLO.yaml allows, i.e. only contains objectives that pass, regardless of the state of the result field in the sh.keptn.event.get-sli.finished event it receives from the sli-provider.

Environment

• Keptn Version (keptn version): 0.10.0

Affected Component

• Approval
• Bridge
• CLI
• Datastore
• Docs
• Distributor / Event Broker
• Helm
• Istio
• Jmeter
• Kubernetes Integration
• Openshift Integration
• REST API
• Webhook

Describe the bug
In addition to the description above, the bug is demonstrated in the following events:

• The dynatrace-service produces the following sh.keptn.event.get-sli.finished event:

  {
    "data": {
      "get-sli": {
        "end": "2021-11-23T08:23:50.319Z",
        "indicatorValues": [
          {
            "message": "Data Explorer tile could not be converted to a metric query: unknown space aggregation: ",
            "metric": "response_time",
            "success": false,
            "value": 0
          },
          {
            "metric": "srt",
            "success": true,
            "value": 28.729770608271618
          }
        ],
        "start": "2021-11-23T08:18:50.319Z"
      },
      "labels": {},
      "message": "Data Explorer tile could not be converted to a metric query: unknown space aggregation: ",
      "project": "project1",
      "result": "fail",
      "service": "service1",
      "stage": "quality-gate",
      "status": "succeeded"
    },
    "id": "c4b0a7bc-cff1-4e72-89db-e547cd6c7704",
    "source": "dynatrace-service",
    "specversion": "1.0",
    "time": "2021-11-23T08:24:21.195Z",
    "type": "sh.keptn.event.get-sli.finished",
    "shkeptncontext": "f91f4eec-a93a-447a-b1e8-db52b948853a",
    "shkeptnspecversion": "0.2.3",
    "triggeredid": "245e7066-d431-49fc-ba18-d09f6b00b806"
  }
  

• This is accompanied by the following SLO file:

  spec_version: ""
  filter: {}
  comparison:
    compare_with: single_result
    include_result_with_score: pass
    number_of_comparison_results: 1
    aggregate_function: avg
  objectives:
  - null
  - sli: srt
    displayName: ""
    pass: []
    warning: []
    weight: 1
    key_sli: false
  total_score:
    pass: 90%
    warning: 75%
  

• The lighthouse then produces the following sh.keptn.event.evaluation.finished event:

  {
    "data": {
      "evaluation": {
        "comparedEvents": [
          "f53abb62-507d-4f9e-a20f-26776452324d"
        ],
        "gitCommit": "",
        "indicatorResults": [
          {
            "displayName": "",
            "keySli": false,
            "passTargets": null,
            "score": 0,
            "status": "info",
            "value": {
              "metric": "srt",
              "success": true,
              "value": 28.729770608271618
            },
            "warningTargets": null
          }
        ],
        "result": "pass",
        "score": 100,
        "sloFileContent": "c3BlY192ZXJzaW9uOiAiIgpmaWx0ZXI6IHt9CmNvbXBhcmlzb246CiAgY29tcGFyZV93aXRoOiBzaW5nbGVfcmVzdWx0CiAgaW5jbHVkZV9yZXN1bHRfd2l0aF9zY29yZTogcGFzcwogIG51bWJlcl9vZl9jb21wYXJpc29uX3Jlc3VsdHM6IDEKICBhZ2dyZWdhdGVfZnVuY3Rpb246IGF2ZwpvYmplY3RpdmVzOgotIG51bGwKLSBzbGk6IHNydAogIGRpc3BsYXlOYW1lOiAiIgogIHBhc3M6IFtdCiAgd2FybmluZzogW10KICB3ZWlnaHQ6IDEKICBrZXlfc2xpOiBmYWxzZQp0b3RhbF9zY29yZToKICBwYXNzOiA5MCUKICB3YXJuaW5nOiA3NSUK",
        "timeEnd": "2021-11-23T08:23:50.319Z",
        "timeStart": "2021-11-23T08:18:50.319Z"
      },
      "labels": {},
      "project": "project1",
      "result": "pass",
      "service": "service1",
      "stage": "quality-gate",
      "status": "succeeded"
    },
    "id": "d3d283fb-0ab0-449f-a78b-ecf3d9e10820",
    "source": "lighthouse-service",
    "specversion": "1.0",
    "time": "2021-11-23T08:24:23.345Z",
    "type": "sh.keptn.event.evaluation.finished",
    "shkeptncontext": "f91f4eec-a93a-447a-b1e8-db52b948853a",
    "shkeptnspecversion": "0.2.3",
    "triggeredid": "0735b387-22cf-4b79-9d97-01c8f8b15129"
  }
  

Expected behavior
The lighthouse-service should fail the evaluation if within the sh.keptn.event.get-sli.finished event result=fail

Current behavior
The lighthouse-service erroneously passes the evaluation

[agrimmer]

Please let's discuss if we fix this bug by solving our technical depth in the lighthouse-service (i.e. the lighthouse-service sends out the sli.triggered event instead of the shipyard-controller). More details can be found here keptn/enhancement-proposals#56