Reading metrics from external DBs

[sriumcp]

Following is the result of some initial design deliberations between @Alan-Cha @kalantar and @sriumcp ...

---

1. Sample experiment
   The following experiment is in the experiment.yaml file.

# task 1: generate HTTP requests for application URL
# collect Iter8's built-in latency and error-related metrics
- task: read-metrics-from-backends
# task 2: validate service level objectives for app using
# the metrics collected in the above task
- task: assess-app-versions
  with:
    SLOs: 
    - metric: istio-prom/error-rate
      upperLimit: 0
    - metric: istio-prom/error-rate
      upperLimit: 0
    - metric: istio-prom/mean-latency
      upperLimit: 100

Istio Prometheus is a backend, contained in the istio-prom.metrics.yaml file.

url: https://prometheus.in.my.namespace
headers:
  X-super-secret: my-token
provider: Istio Prometheus add-on
method: GET
authType: bearer # basic | api-token | ...
# use app to derive everything here... 
# if app=hello, 
# baseline service (first one), has hello as the service name, hello-baseline as the deployment name
# candidate service (second one), has hello-candidate as the service name, hello-candidate as the deployment name
versionInfo:
- serviceName: hello
  deploymentName: hello-baseline
- serviceName: hello-candidate
  deploymentName: hello-candidate
metrics:
- name: mean-latency
  type: gauge
  params:
  - name: param
    # totally fake incorrect example
    value: |
      istio_requests_total{service_name={{ .version.serviceName }}, deployment_name={{ .version.deploymentName }}, namespace=my-ns }({{ .elapsedTime }})
    # some complicated prometheus query with things like $name, $namespace, $serviceName, $elapsedTime
    # stuff like $name, $namespace, and $serviceName will come from the `versionInfo` section of this metric in the experiment
  body: |
    body for POST queries
  jqExpression: my-jq-expression

CE Sysdig is another backend, contained in the ce-sysdig.metrics.yaml file.

iter8 hub -e ce-validation

iter8 run --set app=my-app
--set backends.istio-prom.url=https://prometheus.in.my.namespace
--set backends.ce-sysdig.url=https://something.else
--set backends.ce-sysdig.token={{ .SECRETS.my-pat }}

figure out the right way to do these tokens

scrub out all secrets in the logs

chart authoring

iter8 init -t slo-validation -b istio-prom -b ce-sysdig -d k8s-cron-job

Result of the above is a valid chart.

[Alan-Cha]

ce-sysdig.metrics.yaml

# endpoint where the monitoring instance is available
# https://cloud.ibm.com/docs/monitoring?topic=monitoring-endpoints#endpoints_sysdig
url: {{ .MonitoringEndpoint }}/data # e.g. https://ca-tor.monitoring.cloud.ibm.com/api/data
headers:
  # IAM token
  # to get the token, run: ibmcloud iam oauth-tokens | grep IAM | cut -d \: -f 2 | sed 's/^ *//'
  Authentication: Bearer {{ .IAMToken }}
  # GUID of the IBM Cloud Monitoring instance
  # to get the GUID, run: ibmcloud resource service-instance <NAME> --output json | jq -r '.[].guid'
  # https://cloud.ibm.com/docs/monitoring?topic=monitoring-mon-curl
  IBMInstanceID: {{ .GUID }}
provider: IBM Cloud Code Engine Sysdig
method: POST
metrics:
- name: mean-latency
  type: gauge
  params:
  - name: param
    # totally fake incorrect example
    value: |
      istio_requests_total{service_name={{ .version.serviceName }}, deployment_name={{ .version.deploymentName }}, namespace=my-ns }({{ .elapsedTime }})
    # some complicated prometheus query with things like $name, $namespace, $serviceName, $elapsedTime
    # stuff like $name, $namespace, and $serviceName will come from the `versionInfo` section of this metric in the experiment
  # available metrics: https://cloud.ibm.com/docs/codeengine?topic=codeengine-monitor#metrics-by-plan
  # extracting metrics: https://cloud.ibm.com/docs/monitoring?topic=monitoring-metrics_api
  body: |
    {
      "metrics": [
        {
          "id": "ibm_codeengine_application_revision_count"
        }
      ],
      "filter": "ibm_location = \"ca-tor\"",
      "sampling": 86400,
      "last": 86400
    }
  jqExpression: my-jq-expression

[sriumcp]

request-count, mean-latency, error-count, and error-rate are four metrics we need to populate the above metrics file with (error-rate = formula used for error count divided by formula used for request count)

[Alan-Cha]

According to CE, the following metrics are not produced anymore:

ibm_codeengine_application_request_duration_milliseconds
ibm_codeengine_application_request_bytes
ibm_codeengine_application_response_bytes

ibm_codeengine_application_request_duration_milliseconds seems to be important for calculating mean-latency. I am not sure what we can do to resolve this issue.

[sriumcp]

This is even less than I thought...

For now, lets use CE as a way to demonstrate the capabilities of this task ... perhaps with the very limited set of metrics request-count, error-count, and error-rate.

Once we have the task in place, we can:

1. Approach CE with the demo, and ask them about what other metrics might be of interest to them, and perhaps
2. Explore Istio / Linkerd and their Prometheus add ons. In the context of mirroring experiments.

[sriumcp]

Sample experiment workflow for a code engine experiment with sysdig.

This experiment validates that given revision of a CE application satisfies all the SLOs specified. The experiment may run over multiple loops, but this does not affect the experiment.yaml file, or the ce-sysdig.metrics.yaml file that are generated.

Experiment workflow

1. Download the CE sysdig experiment chart from Iter8 hub as follows.

iter8 hub -e ce-sysdig
cd ce-sysdig

2. We will load test and validate the CE application whose URL is https://ce.app.sample.url, whose name is my-app, and whose revision is named my-app-7293e. We will specify that the error rate must be 0, and the mean latency must be under 50 msec.

Launch the experiment as follows (ignore iter8 launch for now; perhaps best to think of it is as iter8 gen k8s ... | iter8 k run

iter8 launch --app=my-app \
          --set IBMInstanceId=abc-123-xyz \
          --set ce-sysdig-url=https://the.best.url/data \          
          --set ce-sysdig-token=8291-7482ns-ms6zas \
          --set url=https://ce.app.sample.url \
          --set SLOs.revision=my-app-7293e \
          --set SLOs.error-rate=0 \
          --set SLOs.mean-latency=50 

The above command creates a bunch of K8s manifests and deploys them into CE. The core of these manifests are the experiment.yaml yaml and ce-sysdig.metrics.yaml files, which we care about for this discussion.

experiment.yaml

- task: read-metrics-from-backends
- task: assess-app-versions
  with:
    SLOs:
    # the fact that this task has a `ce-sysdig` backend implies that there is also a 
    # ce-sysdig.metrics.yaml file available for iter8 to read from.
    - metric: "ce-sysdig/error-rate"
      upperLimit: 0
    - metric: "ce-sysdig/mean-latency"
      upperLimit: 50

ce-sysdig.metrics.yaml

Please look at the example in the above comment.