Skip to content
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.

Latest commit

 

History

History
257 lines (196 loc) · 9.43 KB

02-06 challenge3.md

File metadata and controls

257 lines (196 loc) · 9.43 KB
Raw
sectionid sectionclass parent-id title
monitoring
h2
upandrunning
Monitoring

You would like to monitor the performance of different components in your application, view logs and get alerts whenever your application availability goes down or some components fail.

Use a combination of the available tools to setup alerting capabilities for your application.

Tasks

Create Log Analytics workspace

Task Hints

{% collapsible %}

If you are running this lab as part of the managed lab environment, you may not be able to create the required resources to enable monitoring due to insufficient permissions on the subscription. You'll need to pre-create the Log Analytics workspace in your assigned environment resource group.

Follow the Create a Log Analytics workspace in the Azure portal instructions.

Alternatively you can create the workspace using the CLI with the command below, ensure you pick a unique name for the workspace

az resource create --resource-type Microsoft.OperationalInsights/workspaces \
 --name <workspace-name> \
 --resource-group <resource-group> \
 --location <region> \
 --properties '{}' -o table

{% endcollapsible %}

Enable the monitoring addon

Task Hints

{% collapsible %}

First get the resource id of the workspace you created, by running

az resource show --resource-type Microsoft.OperationalInsights/workspaces --resource-group <resource-group> --name <workspace-name> --query "id" -o tsv

Next enable the monitoring add-on by running the command below, replace the placeholder values and the workspace-resource-id value with the output from the previous command

az aks enable-addons --resource-group <resource-group> --name <unique-aks-cluster-name> --addons monitoring --workspace-resource-id <workspace-resource-id>

{% endcollapsible %}

Leverage integrated Azure Kubernetes Service monitoring to figure out if requests are failing, inspect Kubernetes event or logs and monitor your cluster health

Task Hints

  • View the utilization reports and charts in the Azure portal, via the "Insights" view on your AKS cluster
  • It might be several minutes before the data appears

{% collapsible %}

  • Check the cluster utilization under load Cluster utilization

  • Identify which pods are causing trouble Pod utilization

{% endcollapsible %}

View the live container logs and Kubernetes events

Task Hints

  • You can view live log data from the 'Containers' tab in the Insights view, with the "View live data (preview)" button.
  • Will get an error, this can be fixed by setting up some RBAC roles and accounts in your cluster. This is covered in the AKS documentation. You might need to refresh the page in the portal for the changes to take effect.

{% collapsible %}

If the cluster is RBAC enabled, which is the default, you have to create the appropriate ClusterRole and ClusterRoleBinding.

Save the YAML below as logreader-rbac.yaml or download it from [logreader-rbac.yaml](yaml-solutions/01. challenge-03/logreader-rbac.yaml)

apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRole 
metadata: 
   name: containerHealth-log-reader 
rules: 
   - apiGroups: [""] 
     resources: ["pods/log", "events"] 
     verbs: ["get", "list"]  
--- 
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding 
metadata: 
   name: containerHealth-read-logs-global 
roleRef: 
    kind: ClusterRole 
    name: containerHealth-log-reader 
    apiGroup: rbac.authorization.k8s.io 
subjects: 
   - kind: User 
     name: clusterUser 
     apiGroup: rbac.authorization.k8s.io

And deploy it using

kubectl apply -f logreader-rbac.yaml

If you have a Kubernetes cluster that is not configured with Kubernetes RBAC authorization or integrated with Azure AD single-sign on, you do not need to follow the steps above. Because Kubernetes authorization uses the kube-api, contributor access is required.

Head over to the AKS cluster on the Azure portal, click on Insights under Monitoring, click on the Controllers tab and pick a container to view its live logs or event logs and debug what is going on.

Azure Monitor for Containers: Live Logs

{% endcollapsible %}

Collect Prometheus metrics (optional)

{% collapsible %}

Note The minimum agent version supported by this feature is microsoft/oms:ciprod07092019 or later.

  1. Run an demo application called “prommetrics-demo” which already has the Prometheus endpoint exposed. Save the YAML below as prommetrics-demo.yaml or download it from [prommetrics-demo.yaml](yaml-solutions/01. challenge-03/prommetrics-demo.yaml)
apiVersion: v1
kind: Service
metadata:
  name: prommetrics-demo
  labels:
    app: prommetrics-demo
spec:
  selector:
    app: prommetrics-demo
  ports:
  - name: metrics
    port: 8000
    protocol: TCP
    targetPort: 8000
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prommetrics-demo
  labels:
    app: prommetrics-demo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: prommetrics-demo
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/"
        prometheus.io/port: "8000"
      labels:
        app: prommetrics-demo
    spec:
      containers:
      - name: prommetrics-demo
        image: vishiy/tools:prommetricsv5
        imagePullPolicy: Always
        ports:
        - containerPort: 8000
        - containerPort: 8080

And deploy it using

kubectl apply -f prommetrics-demo.yaml

This application on purpose generates "Bad Request 500" when traffic is generated and it exposes a Prometheus metric called prommetrics_demo_requests_counter_total.

  1. Generate traffic to the application by running curl.

Find the pod you just created.

kubectl get pods | grep prommetrics-demo

prommetrics-demo-7f455766c4-gmpjb   1/1       Running   0          2m
prommetrics-demo-7f455766c4-n7554   1/1       Running   0          2m
prommetrics-demo-7f455766c4-q756r   1/1       Running   0          2m
prommetrics-demo-7f455766c4-vqncw   1/1       Running   0          2m

Select one of the pods and login.

kubectl exec -it prommetrics-demo-7f455766c4-gmpjb bash

While logged on, execute curl to generate traffic.

while (true); do curl 'http://prommetrics-demo.default.svc.cluster.local:8080'; sleep 5; done

Note Leave the window open and keep running this. You will see "Internal Server Error" but do not close the window.

  1. Download the configmap template yaml file and apply to start scraping the metrics.

This configmap is pre-configured to scrape the application pods and collect Prometheus metric “prommetrics_demo_requests_counter_total” from the demo application in 1min interval.

Download configmap from [configmap.yaml](yaml-solutions/01. challenge-03/configmap.yaml)

interval = "1m"
fieldpass = ["prommetrics_demo_requests_counter_total"]
monitor_kubernetes_pods = true

And deploy it using

kubectl apply -f configmap.yaml
  1. Query the Prometheus metrics and plot a chart.

To access Log Analytics, go to the AKS overview page and click Logs in the TOC under Monitor. Copy the query below and run.

InsightsMetrics
| where Name == "prommetrics_demo_requests_counter_total"
| extend dimensions=parse_json(Tags)
| extend request_status = tostring(dimensions.request_status)
| where request_status == "bad"
| project request_status, Val, TimeGenerated | render timechart

You should be able to plot a chart based on the Prometheus metrics collected.

Azure Monitor for Containers: Prometheus

{% endcollapsible %}

Resources