Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
57 lines (39 sloc) 3.21 KB
title description author ms.author ms.reviewer ms.service ms.topic ms.custom ms.date
Apache Kafka increase scale - Azure HDInsight
Learn how to configure managed disks for Apache Kafka cluster on Azure HDInsight to increase scalability.
hrasheed-msft
hrasheed
jasonh
hdinsight
conceptual
hdinsightactive
12/09/2019

Configure storage and scalability for Apache Kafka on HDInsight

Learn how to configure the number of managed disks used by Apache Kafka on HDInsight.

Kafka on HDInsight uses the local disk of the virtual machines in the HDInsight cluster. Since Kafka is very I/O heavy, Azure Managed Disks is used to provide high throughput and provide more storage per node. If traditional virtual hard drives (VHD) were used for Kafka, each node is limited to 1 TB. With managed disks, you can use multiple disks to achieve 16 TB for each node in the cluster.

The following diagram provides a comparison between Kafka on HDInsight before managed disks, and Kafka on HDInsight with managed disks:

kafka with managed disks architecture

Configure managed disks: Azure portal

  1. Follow the steps in the Create an HDInsight cluster to understand the common steps to create a cluster using the portal. Don't complete the portal creation process.

  2. From the Configuration & Pricing section, use the Number of Nodes field to configure the number of disks.

    [!NOTE]
    The type of managed disk can be either Standard (HDD) or Premium (SSD). Premium disks are used with DS and GS series VMs. All other VM types use standard.

    cluster size section with the disks per worker node highlighted

Configure managed disks: Resource Manager template

To control the number of disks used by the worker nodes in a Kafka cluster, use the following section of the template:

"dataDisksGroups": [
    {
        "disksPerNode": "[variables('disksPerWorkerNode')]"
    }
    ],

You can find a complete template that demonstrates how to configure managed disks at https://hditutorialdata.blob.core.windows.net/armtemplates/create-linux-based-kafka-mirror-cluster-in-vnet-v2.1.json.

Next steps

For more information on working with Apache Kafka on HDInsight, see the following documents:

You can’t perform that action at this time.