feat(o11y): makes the alarm less noisy

[arein]

Description

Our downstream services are not expected to have 100% uptime which this alarm expected. So this ups the limit a bit to reduce noise.

inb4: it's an OR operator so if any of the sources are > 5 it fires an alarm.

How Has This Been Tested?

Not tested.

Due Diligence

• Breaking change
• Requires a documentation update
• Requires a e2e/integration test update

[github-actions]

Show Plan

[command]/home/runner/work/_temp/378fbb4c-e8c7-4760-99ad-87cc8cc360a7/terraform-bin -chdir=terraform show -no-color /tmp/plan.tfplan

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.monitoring.grafana_dashboard.at_a_glance will be updated in-place
  ~ resource "grafana_dashboard" "at_a_glance" {
      ~ config_json  = jsonencode(
          ~ {
              ~ panels               = [
                    # (6 unchanged elements hidden)
                    {
                        datasource  = {
                            type = "cloudwatch"
                            uid  = "XnMFKnQVk"
                        }
                        fieldConfig = {
                            defaults  = {
                                color      = {
                                    mode = "palette-classic"
                                }
                                custom     = {
                                    axisLabel         = ""
                                    axisPlacement     = "auto"
                                    barAlignment      = 0
                                    drawStyle         = "line"
                                    fillOpacity       = 0
                                    gradientMode      = "none"
                                    hideFrom          = {
                                        legend  = false
                                        tooltip = false
                                        viz     = false
                                    }
                                    lineInterpolation = "linear"
                                    lineWidth         = 1
                                    pointSize         = 5
                                    scaleDistribution = {
                                        type = "linear"
                                    }
                                    showPoints        = "auto"
                                    spanNulls         = false
                                    stacking          = {
                                        group = "A"
                                        mode  = "none"
                                    }
                                    thresholdsStyle   = {
                                        mode = "off"
                                    }
                                }
                                mappings   = []
                                thresholds = {
                                    mode  = "absolute"
                                    steps = [
                                        {
                                            color = "green"
                                            value = null
                                        },
                                        {
                                            color = "red"
                                            value = 80
                                        },
                                    ]
                                }
                            }
                            overrides = []
                        }
                        gridPos     = {
                            h = 9
                            w = 7
                            x = 0
                            y = 18
                        }
                        options     = {
                            legend  = {
                                calcs       = []
                                displayMode = "list"
                                placement   = "bottom"
                            }
                            tooltip = {
                                mode = "single"
                                sort = "none"
                            }
                        }
                        targets     = [
                            {
                                alias            = ""
                                datasource       = {
                                    type = "cloudwatch"
                                    uid  = "XnMFKnQVk"
                                }
                                dimensions       = {
                                    LoadBalancer = "app/staging-push-load-balancer/aea5ef9d0a34453a"
                                }
                                expression       = ""
                                id               = ""
                                matchExact       = true
                                metricEditorMode = 0
                                metricName       = "RequestCount"
                                metricQueryType  = 0
                                namespace        = "AWS/ApplicationELB"
                                period           = ""
                                queryMode        = "Metrics"
                                refId            = "A"
                                region           = "default"
                                sqlExpression    = ""
                                statistic        = "Sum"
                            },
                        ]
                        title       = "Requests"
                        type        = "timeseries"
                    },
                  ~ {
                      ~ alert       = {
                          ~ conditions          = [
                              ~ {
                                  ~ evaluator = {
                                      ~ params = [
                                          - 0,
                                          + 5,
                                        ]
                                        # (1 unchanged attribute hidden)
                                    }
                                    # (4 unchanged attributes hidden)
                                },
                              ~ {
                                  ~ evaluator = {
                                      ~ params = [
                                          - 0,
                                          + 5,
                                        ]
                                        # (1 unchanged attribute hidden)
                                    }
                                    # (4 unchanged attributes hidden)
                                },
                            ]
                            name                = "staging Echo Server 5XX alert"
                            # (8 unchanged attributes hidden)
                        }
                        # (8 unchanged attributes hidden)
                    },
                    {
                        datasource  = {
                            type = "cloudwatch"
                            uid  = "XnMFKnQVk"
                        }
                        fieldConfig = {
                            defaults  = {
                                color      = {
                                    mode = "palette-classic"
                                }
                                custom     = {
                                    axisLabel         = ""
                                    axisPlacement     = "auto"
                                    barAlignment      = 0
                                    drawStyle         = "line"
                                    fillOpacity       = 0
                                    gradientMode      = "none"
                                    hideFrom          = {
                                        legend  = false
                                        tooltip = false
                                        viz     = false
                                    }
                                    lineInterpolation = "linear"
                                    lineWidth         = 1
                                    pointSize         = 5
                                    scaleDistribution = {
                                        type = "linear"
                                    }
                                    showPoints        = "auto"
                                    spanNulls         = false
                                    stacking          = {
                                        group = "A"
                                        mode  = "none"
                                    }
                                    thresholdsStyle   = {
                                        mode = "off"
                                    }
                                }
                                mappings   = []
                                thresholds = {
                                    mode  = "absolute"
                                    steps = [
                                        {
                                            color = "green"
                                            value = null
                                        },
                                        {
                                            color = "red"
                                            value = 80
                                        },
                                    ]
                                }
                            }
                            overrides = []
                        }
                        gridPos     = {
                            h = 9
                            w = 7
                            x = 14
                            y = 18
                        }
                        options     = {
                            legend  = {
                                calcs       = []
                                displayMode = "list"
                                placement   = "bottom"
                            }
                            tooltip = {
                                mode = "single"
                                sort = "none"
                            }
                        }
                        targets     = [
                            {
                                alias            = ""
                                datasource       = {
                                    type = "cloudwatch"
                                    uid  = "XnMFKnQVk"
                                }
                                dimensions       = {
                                    LoadBalancer = "app/staging-push-load-balancer/aea5ef9d0a34453a"
                                }
                                expression       = ""
                                id               = ""
                                matchExact       = true
                                metricEditorMode = 0
                                metricName       = "HTTPCode_ELB_4XX_Count"
                                metricQueryType  = 0
                                namespace        = "AWS/ApplicationELB"
                                period           = ""
                                queryMode        = "Metrics"
                                refId            = "A"
                                region           = "default"
                                sqlExpression    = ""
                                statistic        = "Sum"
                            },
                            {
                                alias            = ""
                                datasource       = {
                                    type = "cloudwatch"
                                    uid  = "XnMFKnQVk"
                                }
                                dimensions       = {
                                    LoadBalancer = "app/staging-push-load-balancer/aea5ef9d0a34453a"
                                }
                                expression       = ""
                                id               = ""
                                matchExact       = true
                                metricEditorMode = 0
                                metricName       = "HTTPCode_Target_4XX_Count"
                                metricQueryType  = 0
                                namespace        = "AWS/ApplicationELB"
                                period           = ""
                                queryMode        = "Metrics"
                                refId            = "B"
                                region           = "default"
                                sqlExpression    = ""
                                statistic        = "Sum"
                            },
                        ]
                        title       = "4XX"
                        type        = "timeseries"
                    },
                ]
                tags                 = []
                # (15 unchanged attributes hidden)
            }
        )
        id           = "0:staging-push"
        # (7 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
::debug::Terraform exited with code 0.
::debug::stdout: %0ATerraform used the selected providers to generate the following execution%0Aplan. Resource actions are indicated with the following symbols:%0A  ~ update in-place%0A%0ATerraform will perform the following actions:%0A%0A  # module.monitoring.grafana_dashboard.at_a_glance will be updated in-place%0A  ~ resource "grafana_dashboard" "at_a_glance" {%0A      ~ config_json  = jsonencode(%0A          ~ {%0A              ~ panels               = [%0A                    # (6 unchanged elements hidden)%0A                    {%0A                        datasource  = {%0A                            type = "cloudwatch"%0A                            uid  = "XnMFKnQVk"%0A                        }%0A                        fieldConfig = {%0A                            defaults  = {%0A                                color      = {%0A                                    mode = "palette-classic"%0A                                }%0A                                custom     = {%0A                                    axisLabel         = ""%0A                                    axisPlacement     = "auto"%0A                                    barAlignment      = 0%0A                                    drawStyle         = "line"%0A                                    fillOpacity       = 0%0A                                    gradientMode      = "none"%0A                                    hideFrom          = {%0A                                        legend  = false%0A                                        tooltip = false%0A                                        viz     = false%0A                                    }%0A                                    lineInterpolation = "linear"%0A                                    lineWidth         = 1%0A                                    pointSize         = 5%0A                                    scaleDistribution = {%0A                                        type = "linear"%0A                                    }%0A                                    showPoints        = "auto"%0A                                    spanNulls         = false%0A                                    stacking          = {%0A                                        group = "A"%0A                                        mode  = "none"%0A                                    }%0A                                    thresholdsStyle   = {%0A                                        mode = "off"%0A                                    }%0A                                }%0A                                mappings   = []%0A                                thresholds = {%0A                                    mode  = "absolute"%0A                                    steps = [%0A                                        {%0A                                            color = "green"%0A                                            value = null%0A                                        },%0A                                        {%0A                                            color = "red"%0A                                            value = 80%0A                                        },%0A                                    ]%0A                                }%0A                            }%0A                            overrides = []%0A                        }%0A                        gridPos     = {%0A                            h = 9%0A                            w = 7%0A                            x = 0%0A                            y = 18%0A                        }%0A                        options     = {%0A                            legend  = {%0A                                calcs       = []%0A                                displayMode = "list"%0A                                placement   = "bottom"%0A                            }%0A                            tooltip = {%0A                                mode = "single"%0A                                sort = "none"%0A                            }%0A                        }%0A                        targets     = [%0A                            {%0A                                alias            = ""%0A                                datasource       = {%0A                                    type = "cloudwatch"%0A                                    uid  = "XnMFKnQVk"%0A                                }%0A                                dimensions       = {%0A                                    LoadBalancer = "app/staging-push-load-balancer/aea5ef9d0a34453a"%0A                                }%0A                                expression       = ""%0A                                id               = ""%0A                                matchExact       = true%0A                                metricEditorMode = 0%0A                                metricName       = "RequestCount"%0A                                metricQueryType  = 0%0A                                namespace        = "AWS/ApplicationELB"%0A                                period           = ""%0A                                queryMode        = "Metrics"%0A                                refId            = "A"%0A                                region           = "default"%0A                                sqlExpression    = ""%0A                                statistic        = "Sum"%0A                            },%0A                        ]%0A                        title       = "Requests"%0A                        type        = "timeseries"%0A                    },%0A                  ~ {%0A                      ~ alert       = {%0A                          ~ conditions          = [%0A                              ~ {%0A                                  ~ evaluator = {%0A                                      ~ params = [%0A                                          - 0,%0A                                          + 5,%0A                                        ]%0A                                        # (1 unchanged attribute hidden)%0A                                    }%0A                                    # (4 unchanged attributes hidden)%0A                                },%0A                              ~ {%0A                                  ~ evaluator = {%0A                                      ~ params = [%0A                                          - 0,%0A                                          + 5,%0A                                        ]%0A                                        # (1 unchanged attribute hidden)%0A                                    }%0A                                    # (4 unchanged attributes hidden)%0A                                },%0A                            ]%0A                            name                = "staging Echo Server 5XX alert"%0A                            # (8 unchanged attributes hidden)%0A                        }%0A                        # (8 unchanged attributes hidden)%0A                    },%0A                    {%0A                        datasource  = {%0A                            type = "cloudwatch"%0A                            uid  = "XnMFKnQVk"%0A                        }%0A                        fieldConfig = {%0A                            defaults  = {%0A                                color      = {%0A                                    mode = "palette-classic"%0A                                }%0A                                custom     = {%0A                                    axisLabel         = ""%0A                                    axisPlacement     = "auto"%0A                                    barAlignment      = 0%0A                                    drawStyle         = "line"%0A                                    fillOpacity       = 0%0A                                    gradientMode      = "none"%0A                                    hideFrom          = {%0A                                        legend  = false%0A                                        tooltip = false%0A                                        viz     = false%0A                                    }%0A                                    lineInterpolation = "linear"%0A                                    lineWidth         = 1%0A                                    pointSize         = 5%0A                                    scaleDistribution = {%0A                                        type = "linear"%0A                                    }%0A                                    showPoints        = "auto"%0A                                    spanNulls         = false%0A                                    stacking          = {%0A                                        group = "A"%0A                                        mode  = "none"%0A                                    }%0A                                    thresholdsStyle   = {%0A                                        mode = "off"%0A                                    }%0A                                }%0A                                mappings   = []%0A                                thresholds = {%0A                                    mode  = "absolute"%0A                                    steps = [%0A                                        {%0A                                            color = "green"%0A                                            value = null%0A                                        },%0A                                        {%0A                                            color = "red"%0A                                            value = 80%0A                                        },%0A                                    ]%0A                                }%0A                            }%0A                            overrides = []%0A                        }%0A                        gridPos     = {%0A                            h = 9%0A                            w = 7%0A                            x = 14%0A                            y = 18%0A                        }%0A                        options     = {%0A                            legend  = {%0A                                calcs       = []%0A                                displayMode = "list"%0A                                placement   = "bottom"%0A                            }%0A                            tooltip = {%0A                                mode = "single"%0A                                sort = "none"%0A                            }%0A                        }%0A                        targets     = [%0A                            {%0A                                alias            = ""%0A                                datasource       = {%0A                                    type = "cloudwatch"%0A                                    uid  = "XnMFKnQVk"%0A                                }%0A                                dimensions       = {%0A                                    LoadBalancer = "app/staging-push-load-balancer/aea5ef9d0a34453a"%0A                                }%0A                                expression       = ""%0A                                id               = ""%0A                                matchExact       = true%0A                                metricEditorMode = 0%0A                                metricName       = "HTTPCode_ELB_4XX_Count"%0A                                metricQueryType  = 0%0A                                namespace        = "AWS/ApplicationELB"%0A                                period           = ""%0A                                queryMode        = "Metrics"%0A                                refId            = "A"%0A                                region           = "default"%0A                                sqlExpression    = ""%0A                                statistic        = "Sum"%0A                            },%0A                            {%0A                                alias            = ""%0A                                datasource       = {%0A                                    type = "cloudwatch"%0A                                    uid  = "XnMFKnQVk"%0A                                }%0A                                dimensions       = {%0A                                    LoadBalancer = "app/staging-push-load-balancer/aea5ef9d0a34453a"%0A                                }%0A                                expression       = ""%0A                                id               = ""%0A                                matchExact       = true%0A                                metricEditorMode = 0%0A                                metricName       = "HTTPCode_Target_4XX_Count"%0A                                metricQueryType  = 0%0A                                namespace        = "AWS/ApplicationELB"%0A                                period           = ""%0A                                queryMode        = "Metrics"%0A                                refId            = "B"%0A                                region           = "default"%0A                                sqlExpression    = ""%0A                                statistic        = "Sum"%0A                            },%0A                        ]%0A                        title       = "4XX"%0A                        type        = "timeseries"%0A                    },%0A                ]%0A                tags                 = []%0A                # (15 unchanged attributes hidden)%0A            }%0A        )%0A        id           = "0:staging-push"%0A        # (7 unchanged attributes hidden)%0A    }%0A%0APlan: 0 to add, 1 to change, 0 to destroy.%0A
::debug::stderr: 
::debug::exitcode: 0

Action: pull_request

[chris13524]

Should have a process to review any stray 5xx's so we can continue to nail and eliminate them