title | description | author | ms.subservice | ms.author | ms.topic | ms.custom | ms.date |
---|---|---|---|---|---|---|---|
Configure Layered Network Management on level 4 cluster |
Deploy and configure Azure IoT Layered Network Management on a level 4 cluster. |
PatAltimore |
layered-network-management |
patricka |
how-to |
ignite-2023, devx-track-azurecli |
11/15/2023 |
[!INCLUDE public-preview-note]
Azure IoT Layered Network Management Preview is one of the Azure IoT Operations Preview components. However, it can be deployed individually to the top network layer for supporting the Azure IoT Operations in the lower layer. In the top level of your network layers (usually level 4 of the ISA-95 network architecture), the cluster and Layered Network Management service have direct internet access. Once the setup is completed, the Layered Network Management service is ready for receiving network traffic from the child layer and forwards it to Azure Arc.
Meet the following minimum requirements for deploying the Layered Network Management individually on the system.
- AKS Edge Essentials - Arc-connected cluster and GitOps category in AKS Edge Essentials requirements and support matrix
- K3S Kubernetes cluster - Azure Arc-enabled Kubernetes system requirements
To set up only Layered Network Management, the prerequisites are simpler than an Azure IoT Operations deployment. It's optional to fulfill the general requirements for Azure IoT Operations in Prepare your Kubernetes cluster.
The following steps for setting up AKS Edge Essentials and K3S Kubernetes cluster are verified by Microsoft.
-
Ubuntu 22.04 LTS is the recommended version for the host machine.
-
Install Helm 3.8.0 or later.
-
Install Kubectl.
-
Install the Azure CLI. You can install the Azure CLI directly onto the level 4 machine or on another developer or jumpbox machine if you plan to access the level 3 cluster remotely. If you choose to access the Kubernetes cluster remotely to keep the cluster host clean, you run the kubectl and az" related commands from the developer machine for the rest of the steps in this article.
-
Install Azure CLI. Follow the steps in Install Azure CLI on Linux.
-
Install connectedk8s and other extensions.
az extension add --name connectedk8s az extension add --name k8s-extension
-
Install Azure CLI extension using
az extension add --name azure-iot-ops
.
-
-
Install K3S with the following command:
curl -sfL https://get.k3s.io | sh -s - --disable=traefik --write-kubeconfig-mode 644
[!IMPORTANT] Be sure to use the
--disable=traefik
parameter to disable treafik. Otherwise, you might have an issue when you try to allocate public IP for the Layered Network Management service in later steps. -
Copy the K3s configuration yaml file to
.kube/config
.mkdir ~/.kube cp ~/.kube/config ~/.kube/config.back sudo KUBECONFIG=~/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > ~/.kube/merged mv ~/.kube/merged ~/.kube/config chmod 0600 ~/.kube/config export KUBECONFIG=~/.kube/config #switch to k3s context kubectl config use-context default
-
Install Windows 11 on your device.
-
Install Helm 3.8.0 or later.
-
Install Kubectl.
-
Install AKS Edge Essentials. Follow the steps in Prepare your machines for AKS Edge Essentials.
-
Install Azure CLI. Follow the steps in Install Azure CLI on Windows.
-
Install connectedk8s using the following command:
az extension add --name connectedk8s az extension add --name k8s-extension
-
Install Azure CLI extension using
az extension add --name azure-iot-ops
.
- Verify you meet the Prerequisites section of the AKS Edge Essentials quickstart.
- Follow the Prepare your machines for AKS Edge Essentials steps to install AKS Edge Essentials on your Windows 11 machine.
- Follow the steps in the Single machine deployment article.
Use the New-AksEdgeDeployment PowerShell command to create a file named aks-ee-config.json, make the following modifications:
-
In the Init section, change the ServiceIPRangeSize property to 10.
"Init": { "ServiceIPRangeSize": 10 },
-
In the Network section, verify the following properties are added or set. Replace the placeholder text with your values. Confirm that the Ip4AddressPrefix A.B.C doesn't overlap with the IP range that is assigned within network layers.
"Network": { "NetworkPlugin": "flannel", "Ip4AddressPrefix": "<A.B.C.0/24>", "Ip4PrefixLength": 24, "InternetDisabled": false, "SkipDnsCheck": false,
For more information about deployment configurations, see Deployment configuration JSON parameters.
-
-
Sign in with Azure CLI. To avoid permission issues later, it's important that you sign in interactively using a browser window:
az login
-
Set environment variables for the setup steps. Replace values in
<>
with valid values or names of your choice. TheCLUSTER_NAME
andRESOURCE_GROUP
are created based on the names you provide:# Id of the subscription where your resource group and Arc-enabled cluster will be created $SUBSCRIPTION_ID = "<subscription-id>" # Azure region where the created resource group will be located # Currently supported regions: : "westus3" or "eastus2" $LOCATION = "WestUS3" # Name of a new resource group to create which will hold the Arc-enabled cluster and Azure IoT Operations resources $RESOURCE_GROUP = "<resource-group-name>" # Name of the Arc-enabled cluster to create in your resource group $CLUSTER_NAME = "<cluster-name>"
-
Set the Azure subscription context for all commands:
az account set -s $SUBSCRIPTION_ID
-
Register the required resource providers in your subscription:
[!NOTE] This is a one-time configuration per subscription.
az provider register -n "Microsoft.ExtendedLocation" az provider register -n "Microsoft.Kubernetes" az provider register -n "Microsoft.KubernetesConfiguration" az provider register -n "Microsoft.IoTOperationsOrchestrator" az provider register -n "Microsoft.IoTOperationsMQ" az provider register -n "Microsoft.IoTOperationsDataProcessor" az provider register -n "Microsoft.DeviceRegistry"
-
Use the az group create command to create a resource group in your Azure subscription to store all the resources:
az group create --location $LOCATION --resource-group $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
-
Use the az connectedk8s connect command to Arc-enable your Kubernetes cluster and manage it in the resource group you created in the previous step:
az connectedk8s connect -n $CLUSTER_NAME -l $LOCATION -g $RESOURCE_GROUP --subscription $SUBSCRIPTION_ID
[!TIP] If the
connectedk8s
commands fail, try using the cmdlets in Connect your AKS Edge Essentials cluster to Arc.
Once your Kubernetes cluster is Arc-enabled, you can deploy the Layered Network Management service to the cluster.
-
Run the following command. Replace the placeholders
<RESOURCE GROUP>
and<CLUSTER NAME>
with your Arc onboarding information from an earlier step.az login az k8s-extension create --resource-group <RESOURCE GROUP> --name kind-lnm-extension --cluster-type connectedClusters --cluster-name <CLUSTER NAME> --auto-upgrade false --extension-type Microsoft.IoTOperations.LayeredNetworkManagement --version 0.1.0-preview --release-train preview
-
Use the kubectl command to verify the Layered Network Management operator is running.
kubectl get pods -n azure-iot-operations
NAME READY STATUS RESTARTS AGE azedge-lnm-operator-598cc495c-5428j 1/1 Running 0 28h
Create the Layered Network Management custom resource.
-
Create a
lnm-cr.yaml
file as specified:apiVersion: layerednetworkmgmt.iotoperations.azure.com/v1beta1 kind: Lnm metadata: name: level4 namespace: azure-iot-operations spec: image: pullPolicy: IfNotPresent repository: mcr.microsoft.com/oss/envoyproxy/envoy-distroless tag: v1.27.0 replicas: 1 logLevel: "debug" openTelemetryMetricsCollectorAddr: "http://aio-otel-collector.azure-iot-operations.svc.cluster.local:4317" level: 4 allowList: enableArcDomains: true domains: - destinationUrl: "*.ods.opinsights.azure.com" destinationType: external - destinationUrl: "*.oms.opinsights.azure.com" destinationType: external - destinationUrl: "*.monitoring.azure.com" destinationType: external - destinationUrl: "*.handler.control.monitor.azure.com" destinationType: external - destinationUrl: "quay.io" destinationType: external - destinationUrl: "*.quay.io" destinationType: external - destinationUrl: "docker.io" destinationType: external - destinationUrl: "*.docker.io" destinationType: external - destinationUrl: "*.docker.com" destinationType: external - destinationUrl: "gcr.io" destinationType: external - destinationUrl: "*.googleapis.com" destinationType: external - destinationUrl: "login.windows.net" destinationType: external - destinationUrl: "graph.windows.net" destinationType: external - destinationUrl: "msit-onelake.pbidedicated.windows.net" destinationType: external - destinationUrl: "*.vault.azure.net" destinationType: external - destinationUrl: "*.k8s.io" destinationType: external - destinationUrl: "*.pkg.dev" destinationType: external - destinationUrl: "github.com" destinationType: external - destinationUrl: "raw.githubusercontent.com" destinationType: external sourceIpRange: - addressPrefix: "0.0.0.0" prefixLen: 0
For debugging or experimentation, you can change the value of loglevel parameter to debug.
-
Create the Custom Resource to create a Layered Network Management instance.
kubectl apply -f lnm-cr.yaml
-
View the Layered Network Management Kubernetes service:
kubectl get services -n azure-iot-operations
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE lnm-level-4 LoadBalancer 10.43.91.54 192.168.0.4 80:30530/TCP,443:31117/TCP,10000:31914/TCP 95s
Important
This step is for AKS Edge Essentials only.
The Layered Network Management deployment creates a Kubernetes service of type LoadBalancer. To ensure that the service is accessible from outside the Kubernetes cluster, you need to map the underlying Windows host's ports to the appropriate ports on the Layered Network Management service.
netsh interface portproxy add v4tov4 listenport=443 listenaddress=0.0.0.0 connectport=443 connectaddress=192.168.0.4
netsh interface portproxy add v4tov4 listenport=10000 listenaddress=0.0.0.0 connectport=10000 connectaddress=192.168.0.4
After these commands are run successfully, traffic received on ports 443 and 10000 on the Windows host is routed through to the Kubernetes service. When configuring customized DNS for the child level network layer, you direct the network traffic to the IP of this Windows host and then to the Layered Network Management service running on it.