bdc-01-k8s.ipynb

{
    "metadata": {
        "kernelspec": {
            "name": "powershell",
            "display_name": "PowerShell"
        },
        "language_info": {
            "name": "powershell",
            "codemirror_mode": "shell",
            "mimetype": "text/x-sh",
            "file_extension": ".ps1"
        }
    },
    "nbformat_minor": 2,
    "nbformat": 4,
    "cells": [
        {
            "cell_type": "markdown",
            "source": [
                "<img src=\"https://github.com/Microsoft/sqlworkshops/blob/master/graphics/solutions-microsoft-logo-small.png?raw=true\" alt=\"Microsoft\">\r\n",
                "<br>\r\n",
                "\r\n",
                "# **SQL Server 2019 big data cluster Tutorial**\r\n",
                "## **Monitoring and troubleshoot SQL Server Big Data Clusters**\r\n",
                "\r\n",
                "In this tutorial, you'll learn how to view in-depth details of a pod or other Kubernetes artifacts that are located in the big data cluster. This article also covers common tasks, such as copying files to or from a container running one of the SQL Server big data cluster services.\r\n",
                "\r\n",
                ""
            ],
            "metadata": {
                "azdata_cell_guid": "e44fc0ee-97f2-43bd-bd2d-3c18dd6db9e8"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Get status of pods**\r\n",
                "You can use the kubectl get pods command to get the status of pods in the cluster for either all namespaces or the big data cluster namespace. The following sections show examples of both."
            ],
            "metadata": {
                "azdata_cell_guid": "086ea3dc-70ac-4bb7-8c3b-0fb749f378fc"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Show status of all pods in the Kubernetes cluster**"
            ],
            "metadata": {
                "azdata_cell_guid": "3909ef05-45c1-4d9a-88ef-a3d59ef574ef"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "Run the following command to get all the pods and their statuses, including the pods that are part of the namespace that SQL Server big data cluster pods are created in:"
            ],
            "metadata": {
                "azdata_cell_guid": "a46b62da-85c8-4d84-8806-33dfbbf3be77"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl get pods --all-namespaces"
            ],
            "metadata": {
                "azdata_cell_guid": "66532331-b9d2-4eb9-a357-2a7c4a164fff"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Show status of all pods in the SQL Server big data cluster**"
            ],
            "metadata": {
                "azdata_cell_guid": "13484a21-5a8c-4b75-a3ab-ace1547dcc4a"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "Use the -n parameter to specify a specific namespace. Note that SQL Server big data cluster pods are created in a new namespace created at cluster bootstrap time based on the cluster name specified in the deployment configuration file. The default name, mssql-cluster, is used here."
            ],
            "metadata": {
                "azdata_cell_guid": "d5d52dd4-4a12-466d-aadc-67e9baa4b326"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl get pods -n mssql-cluster"
            ],
            "metadata": {
                "azdata_cell_guid": "cbfbe0c8-b07f-44dc-b486-25f8c41ebd09"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "You should see output similar to the following list for a running big data cluster:\r\n",
                "\r\n",
                "```\r\n",
                "PS C:\\> kubectl get pods -n mssql-cluster\r\n",
                "NAME              READY   STATUS    RESTARTS   AGE\r\n",
                "appproxy-f2qqt    2/2     Running   0          110m\r\n",
                "compute-0-0       3/3     Running   0          110m\r\n",
                "control-zlncl     4/4     Running   0          118m\r\n",
                "data-0-0          3/3     Running   0          110m\r\n",
                "data-0-1          3/3     Running   0          110m\r\n",
                "gateway-0         2/2     Running   0          109m\r\n",
                "logsdb-0          1/1     Running   0          112m\r\n",
                "logsui-jtdnv      1/1     Running   0          112m\r\n",
                "master-0          7/7     Running   0          110m\r\n",
                "metricsdb-0       1/1     Running   0          112m\r\n",
                "metricsdc-shv2f   1/1     Running   0          112m\r\n",
                "metricsui-9bcj7   1/1     Running   0          112m\r\n",
                "mgmtproxy-x6gcs   2/2     Running   0          112m\r\n",
                "nmnode-0-0        1/1     Running   0          110m\r\n",
                "storage-0-0       7/7     Running   0          110m\r\n",
                "storage-0-1       7/7     Running   0          110m\r\n",
                "```"
            ],
            "metadata": {
                "azdata_cell_guid": "656858ef-2164-47d8-acaf-44ed9b3426de"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Get pod details**"
            ],
            "metadata": {
                "azdata_cell_guid": "69937fab-df8b-4706-90f6-ce0235c3bd77"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "Run the following command to get a detailed description of a specific pod in JSON format output. It includes details, such as the current Kubernetes node that the pod is placed on, the containers running within the pod, and the image used to bootstrap the containers. It also shows other details, such as labels, status, and persisted volumes claims that are associated with the pod."
            ],
            "metadata": {
                "azdata_cell_guid": "b1ca9d51-36ff-439d-8019-37b0ef24854d"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl describe pod  <pod_name> -n <namespace_name>"
            ],
            "metadata": {
                "azdata_cell_guid": "f44639fe-35dd-435d-8ec4-488b04206648"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example shows the details for the master-0 pod in a big data cluster named mssql-cluster:"
            ],
            "metadata": {
                "azdata_cell_guid": "49d46456-e12b-430c-9ae8-c96c7cab677b"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl describe pod master-0 -n mssql-cluster"
            ],
            "metadata": {
                "azdata_cell_guid": "b59a3efe-9f89-4cd0-886d-9348120dda6a"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "If any errors have occurred, you can sometimes see the error in the recent events for the pod."
            ],
            "metadata": {
                "azdata_cell_guid": "94049035-1cfe-4d5a-9ac3-8a9e817e3236"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Get pod logs**"
            ],
            "metadata": {
                "azdata_cell_guid": "8eb66c73-3cc7-4dea-8c65-c6f8da967f69"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "You can retrieve the logs for containers running in a pod. The following command retrieves the logs for all containers running in the pod named master-0 and outputs them to a file name master-0-pod-logs.txt:"
            ],
            "metadata": {
                "azdata_cell_guid": "ff613a6d-10ce-46dd-8373-887db30e8cfe"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl logs master-0 --all-containers=true -n mssql-cluster > ~/temp/master-0-pod-logs.txt"
            ],
            "metadata": {
                "azdata_cell_guid": "61e9fd87-d22c-40d2-94c7-92d08457be1e"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Get status of services**"
            ],
            "metadata": {
                "azdata_cell_guid": "3c441b94-84d3-4ae3-9848-a5d2f6751d31"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "Run the following command to get details for the big data cluster services. These details include their type and the IPs associated with respective services and ports. Note that SQL Server big data cluster services are created in a new namespace created at cluster bootstrap time based on the cluster name specified in deployment configuration file."
            ],
            "metadata": {
                "azdata_cell_guid": "1ab355ea-b7ca-41c1-aad7-6899e9b31e7f"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl get svc -n <namespace_name>"
            ],
            "metadata": {
                "azdata_cell_guid": "e70bcadf-654a-4f67-8d32-8fc3cf79ebd6"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example shows the status for services in a big data cluster named mssql-cluster:"
            ],
            "metadata": {
                "azdata_cell_guid": "b0860168-2efd-4147-ae3b-958df3635017"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl get svc -n mssql-cluster"
            ],
            "metadata": {
                "azdata_cell_guid": "53d2a187-6d64-4827-97a3-a6ecac342e9e"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "| Service                 | Description                                                                 |\r\n",
                "|-------------------------|-----------------------------------------------------------------------------|\r\n",
                "| master-svc-external     | Provides access to the master instance. (EXTERNAL-IP,31433 and the SA user) |\r\n",
                "| controller-svc-external | Supports tools and clients that manage the cluster.                         |\r\n",
                "| gateway-svc-external    | Provides access to the HDFS/Spark gateway. (EXTERNAL-IP and the root user)  |\r\n",
                "| appproxy-svc-external   | Support application deployment scenarios.                                   |"
            ],
            "metadata": {
                "azdata_cell_guid": "96688f60-8d82-4190-9c6a-f62950f2bf43"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Get service details**"
            ],
            "metadata": {
                "azdata_cell_guid": "fbb010ea-1ff5-4e4d-9cf8-4f4902f9c372"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "Run this command to get a detailed description of a service in JSON format output. It will include details like labels, selector, IP, external-IP (if the service is of LoadBalancer type), port, etc."
            ],
            "metadata": {
                "azdata_cell_guid": "5653ae00-e2fe-467e-bd70-b8e2762dd82a"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl describe service <service_name> -n <namespace_name>"
            ],
            "metadata": {
                "azdata_cell_guid": "2ade95f0-21e4-4e7a-b963-d56047bddf6b"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example retrieves details for the master-svc-external service:"
            ],
            "metadata": {
                "azdata_cell_guid": "73796a1c-5115-4428-98e2-e75529fe0eaf"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl describe service master-svc-external -n mssql-cluster"
            ],
            "metadata": {
                "azdata_cell_guid": "1e3b9ac8-070b-4586-850d-928257ae80bf"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Copy files**"
            ],
            "metadata": {
                "azdata_cell_guid": "e68c4a2f-8843-4eee-89d9-e2de5e71419c"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "If you need to copy files from the container to your local machine, use kubectl cp command with the following syntax:"
            ],
            "metadata": {
                "azdata_cell_guid": "1d8e2f4a-2f19-41df-be84-a74675b982bd"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl cp <pod_name>:<source_file_path> -c <container_name> -n <namespace_name> <target_local_file_path>"
            ],
            "metadata": {
                "azdata_cell_guid": "cfa311dc-b8d9-49df-8367-070c0171a57a"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "Similarly, you can use kubectl cp to copy files from the local machine into a specific container."
            ],
            "metadata": {
                "azdata_cell_guid": "07b25a78-680c-4f0b-b825-ed0895ace9e5"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl cp <source_local_file_path> <pod_name>:<target_container_path> -c <container_name>  -n <namespace_name>"
            ],
            "metadata": {
                "azdata_cell_guid": "827120b8-f1ba-459a-a079-5f47793625bd"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Copy files from a container**"
            ],
            "metadata": {
                "azdata_cell_guid": "0c9a399a-9b12-483c-bddc-8c355c7154b8"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example copies the SQL Server log files from the container to the ~/temp/sqlserverlogs path on the local machine (in this example the local machine is a Linux client):"
            ],
            "metadata": {
                "azdata_cell_guid": "5a28aece-5485-4e39-a488-45e087a690a3"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl cp master-0:/var/opt/mssql/log -c mssql-server -n mssql-cluster ~/tmp/sqlserverlogs"
            ],
            "metadata": {
                "azdata_cell_guid": "85bcdb43-6b6c-4ad1-95e7-e4148e2415bf"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Copy files into container**"
            ],
            "metadata": {
                "azdata_cell_guid": "d7f4a159-f966-41bc-9d28-b86237b34faf"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example copies the AdventureWorks2016CTP3.bak file from the local machine to the SQL Server master instance container (mssql-server) in the master-0 pod. The file is copied to the /tmp directory in the container."
            ],
            "metadata": {
                "azdata_cell_guid": "f5c2fa94-75a3-410d-a95f-3b47ee94b5dc"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl cp ~/temp/ master-0:/tmp -c mssql-server -n mssql-cluster"
            ],
            "metadata": {
                "azdata_cell_guid": "5791dbbe-9568-41f2-8d11-b62d2c25152a"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Force delete a pod**"
            ],
            "metadata": {
                "azdata_cell_guid": "684d35ed-f747-4a68-bc5f-d98490585d82"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "It is not recommended to force-delete a pod. But for testing availability, resiliency, or data persistence, you can delete a pod to simulate a pod failure with the kubectl delete pods command."
            ],
            "metadata": {
                "azdata_cell_guid": "2c8b22c4-378e-4e5d-b549-919b8ea13bef"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl delete pods <pod_name> -n <namespace_name> --grace-period=0 --force"
            ],
            "metadata": {
                "azdata_cell_guid": "92acb6a1-ae47-4059-b6a5-789547260f23"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example deletes the storage pool pod, storage-0-0:"
            ],
            "metadata": {
                "azdata_cell_guid": "d0850ee7-f154-4585-a333-28973e760db8"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl delete pods storage-0-0 -n mssql-cluster --grace-period=0 --force"
            ],
            "metadata": {
                "azdata_cell_guid": "139b735d-24da-4ce0-a7db-b968998c8468"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Get pod IP**"
            ],
            "metadata": {
                "azdata_cell_guid": "4f8da0ce-627d-4848-b1e4-dce238cf6385"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "For troubleshooting purposes, you might need to get the IP of the node a pod is currently running on. To get the IP address, use the kubectl get pods command with the following syntax:"
            ],
            "metadata": {
                "azdata_cell_guid": "4b47a265-a6c3-4d14-a04a-2665abd6f77d"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl get pods <pod_name> -o yaml -n <namespace_name> | grep hostIP"
            ],
            "metadata": {
                "azdata_cell_guid": "8dbd772c-3ff0-47a5-a27d-a874b7361579"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "The following example gets the IP address of the node that the master-0 pod is running on:"
            ],
            "metadata": {
                "azdata_cell_guid": "41bb9818-d0ae-4189-a6f0-0427c731bdac"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl get pods master-0 -o yaml -n mssql-cluster | grep hostIP"
            ],
            "metadata": {
                "azdata_cell_guid": "ee24e449-2a15-4aab-be2c-68c123c8f9fc"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Kubernetes dashboard**\r\n",
                "You can launch the Kubernetes dashboard for additional information about the cluster. The following sections explain how to launch the dashboard for Kubernetes on AKS and for Kubernetes bootstrapped using kubeadm."
            ],
            "metadata": {
                "azdata_cell_guid": "6451faf9-0644-438d-ade6-11b08139b760"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Start dashboard when cluster is running in AKS**"
            ],
            "metadata": {
                "azdata_cell_guid": "ed21fede-e8ff-47e5-a130-275b63ee1785"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "To launch the Kubernetes dashboard run:"
            ],
            "metadata": {
                "azdata_cell_guid": "85067cb7-bc8f-43cb-9137-2294cc4489a5"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "az aks browse --resource-group <azure_resource_group> --name <aks_cluster_name>"
            ],
            "metadata": {
                "azdata_cell_guid": "b8f76465-0d7b-4b39-91f3-ffbb4db0ce62"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                ">If you get the following error: Unable to listen on port 8001: All listeners failed to create with the following errors: Unable to create listener: Error listen tcp4 127.0.0.1:8001: >bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. Unable to create listener: Error listen tcp6: address [[::1]]:8001: missing port in >address error: Unable to listen on any of the requested ports: [{8001 9090}], make sure you did not start the dashboard already from another window."
            ],
            "metadata": {
                "azdata_cell_guid": "8d13136e-6004-4094-84a5-4b1546b7e2e9"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "When you launch the dashboard on your browser, you might get permission warnings due to RBAC being enabled by default in AKS clusters, and the service account used by the dashboard does not have enough permissions to access all resources (for example, pods is forbidden: User \"system:serviceaccount:kube-system:kubernetes-dashboard\" cannot list pods in the namespace \"default\"). Run the following command to give the necessary permissions to kubernetes-dashboard, and then restart the dashboard:"
            ],
            "metadata": {
                "azdata_cell_guid": "2936c31f-bf03-4c9a-bcf0-bd61330d8f3e"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "### **Start dashboard when Kubernetes cluster is bootstrapped using kubeadm**"
            ],
            "metadata": {
                "azdata_cell_guid": "32bd1f34-3b7e-4383-b8d9-b2d8f057986d"
            }
        },
        {
            "cell_type": "markdown",
            "source": [
                "For detailed instructions how to deploy and configure the dashboard in your Kubernetes cluster, see the Kubernetes documentation. To launch the Kubernetes dashboard, run this command:"
            ],
            "metadata": {
                "azdata_cell_guid": "a11b0521-4baf-40b5-96da-b4c2a75a8036"
            }
        },
        {
            "cell_type": "code",
            "source": [
                "kubectl proxy"
            ],
            "metadata": {
                "azdata_cell_guid": "cc2ec86a-b2ff-401a-93f3-6cfe12f281ca"
            },
            "outputs": [],
            "execution_count": null
        },
        {
            "cell_type": "markdown",
            "source": [
                "## **Next Step: Viewing the status of a big data cluster using azdata and azure data studio.**\r\n",
                "\r\n",
                "Now you're ready to open the next Jupyter Notebook(optional) - [bdc-02-adstudio.ipynb](bdc-02-adstudio.ipynb) - to learn how to view the status of a big data cluster using azdata and azure data studio."
            ],
            "metadata": {
                "azdata_cell_guid": "0c8cddcc-87c0-4ecd-aa7a-ce2261f51858"
            }
        }
    ]
}