Releases: Azure/HDInsight-on-AKS
Azure HDInsight on AKS Release 2024-08-05
Release date: Aug 05, 2024
This release applies to the following
- Cluster Pool Version: 1.2
- Cluster Version: 1.2.1
- AKS version: 1.27
New Features
MSI based SQL authentication
Users can now authenticate external Azure SQL DB Metastore with MSI instead of User ID password authentication. This feature helps to further secure the cluster connection with Metastore.
Configurable VM SKUs for Head node, SSH node
This functionality allows users to choose specific SKUs for head nodes, worker nodes, and SSH nodes, offering the flexibility to select according to the use case and the potential to lower total cost of ownership (TCO).
Multiple MSI in cluster
Users can configure multiple MSI for cluster admins operations and for job related resource access. This feature allows users to demarcate and control the access to the cluster and data lying in the storage account.
For example, one MSI for access to data in storage account and dedicated MSI for cluster operations.
Updated
Script action
Script Action now can be added with Sudo user permission. Users can now install multiple dependencies including custom jars to customize the clusters as required.
Library Management
Maven repository shortcut feature added to the Library Management in this release. User can now install Maven dependencies directly from the open-source repositories.
Spark 3.4
Spark 3.4 update brings a range of new features includes
- API enhancements
- Structured streaming improvements
- Improved usability and developer experience
Important
To take benefit of all these latest features, you are required to create a new cluster pool with 1.2 and cluster version 1.2.1
Known issues
- Workload identity limitation:
- There's a known limitation when transitioning to workload identity. This limitation is due to the permission-sensitive nature of FIC operations. Users can't perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion also fails.
- User Assigned Managed Identities (UAMI) support – There's a limit of 20 FICs per UAMI. You can only create 20 Federated Credentials on an identity. In HDInsight on AKS cluster, FIC (Federated Identity Credential) and SA have one-to-one mapping and only 20 SAs can be created against an MSI. If you want to create more clusters, then you are required to provide different MSIs to overcome the limitation.
Operating System version
- Mariner OS 2.0
Workload versions
| Workload | Version |
|---|---|
| Trino | 440 |
| Flink | 1.17.0 |
| Apache Spark | 3.4 |
Supported Java and Scala versions
| Workload | Java | Scala |
|---|---|---|
| Trino | Open JDK 21.0.2 | - |
| Flink | Open JDK 11.0.21 | 2.12.7 |
| Spark | Open JDK 1.8.0_345 | 2.12.15 |
The preview is available in the following regions.
If you have any more questions, contact Azure Support or refer to the Support options page. If you have product specific feedback, write us on aka.ms/askhdinsight.
Azure HDInsight on AKS Release 2024-09-05
Release date: Sep 05, 2024
IMPORTANT
We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.
This release applies to the following
- Cluster Pool Version: 1.2
- Cluster Version: 1.2.1
- AKS version: 1.27
Updated
The latest API version is as follows.
Important
To take benefit of all these latest features, you are required to create a new cluster pool with 1.2 and cluster version 1.2.1
Known issues
- Workload identity limitation:
- There's a known limitation when transitioning to workload identity. This limitation is due to the permission-sensitive nature of FIC operations. Users can't perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion also fails.
- User Assigned Managed Identities (UAMI) support – There's a limit of 20 FICs per UAMI. You can only create 20 Federated Credentials on an identity. In HDInsight on AKS cluster, FIC (Federated Identity Credential) and SA have one-to-one mapping and only 20 SAs can be created against an MSI. If you want to create more clusters, then you are required to provide different MSIs to overcome the limitation.
Operating System version
- Mariner OS 2.0
Workload versions
| Workload | Version |
|---|---|
| Trino | 440 |
| Flink | 1.17.0 |
| Apache Spark | 3.4 |
Supported Java and Scala versions
| Workload | Java | Scala |
|---|---|---|
| Trino | Open JDK 21.0.2 | - |
| Flink | Open JDK 11.0.21 | 2.12.7 |
| Spark | Open JDK 1.8.0_345 | 2.12.15 |
The preview is available in the following regions.
If you have any more questions, contact Azure Support or refer to the Support options page. If you have product specific feedback, write us on aka.ms/askhdinsight.
Full Changelog: https://github.com/Azure/HDInsight-on-AKS/commits/1.2
Azure HDInsight on AKS Release 2024-03-20
Release date: March 20, 2024
This release applies to the following
- Cluster Pool Version: 1.1
- Cluster Version: 1.1.1
- AKS version: 1.27
New Features
Apache Flink Application Mode Cluster
Application mode clusters are designed to support dedicated resources for large and long-running jobs. When you have resource-intensive or extensive data processing tasks, you can use the Application Mode Cluster. This mode allows you to allocate dedicated resources for specific Apache Flink applications, ensuring that they have the necessary computing power and memory to handle large workloads effectively.
Private Clusters for HDInsight on AKS
With private clusters, and outbound cluster settings you can now control ingress and egress traffic from HDInsight on AKS cluster pools and clusters.
- Use Azure Firewall or Network Security Groups (NSGs) to control the egress traffic, when you opt to use outbound cluster pool with load balancer.
- Use Outbound cluster pool with User defined routing to control egress traffic at the subnet level.
- Use Private AKS cluster feature - To ensure AKS control plane, or API server has internal IP addresses. The network traffic between AKS Control plane / API server and HDInsight on AKS node pools (clusters) remains on the private network only.
- Avoid creating public IPs for the cluster. Use private ingress feature on your clusters.
In place Upgrade
Upgrade your clusters and cluster pools with the latest software updates. This means that you can enjoy the latest cluster package hotfixes, security updates, and AKS patches, without recreating clusters.
Known issues
- Workload identity limitation:
- There's a known limitation when transitioning to workload identity. This limitation is due to the permission-sensitive nature of FIC operations. Users can't perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion also fails.
- User Assigned Managed Identities (UAMI) support – There's a limit of 20 FICs per UAMI. You can only create 20 Federated Credentials on an identity. In HDInsight on AKS cluster, FIC (Federated Identity Credential) and SA have one-to-one mapping and only 20 SAs can be created against an MSI. If you want to create more clusters, then you are required to provide different MSIs to overcome the limitation.
Operating System version
- Mariner OS 2.0
Workload versions
| Workload | Version |
|---|---|
| Trino | 426 |
| Flink | 1.17.0 |
| Apache Spark | 3.3.1 |
Supported Java and Scala versions
| Workload | Java | Scala |
|---|---|---|
| Trino | Open JDK 17.0.7 | - |
| Flink | Open JDK 11.0.21 | 2.12.7 |
| Spark | Open JDK 1.8.0_345 | 2.12.15 |
The preview is available in the following regions.
If you have any more questions, contact Azure Support or refer to the Support options page. If you have product specific feedback, write us on aka.ms/askhdinsight.
Azure HDInsight on AKS Release 2024-02-05
Release date: February 05, 2024
This release applies to the following
- Cluster Pool Version: 1.1
- Cluster Version: 1.1.0
- AKS version: 1.27
Tip
To create a new HDInsight on AKS cluster on 1.1.0, you are required to create a new cluster pool with version 1.1
New Features
- Workload Identity is supported by default for cluster pools on 1.1
- Trino clusters support Trino 426 from 1.1.0 release
- HDInsight on AKS now includes all changes up to Trino 426 with several notable improvements provided by the community, learn more about Trino here.
- Trino cluster shape now supports load-based autoscale from 1.1.0 release
- Trino on HDInsight on AKS now supports load-based autoscale making cluster more cost efficient. Learn more about it here.
- Trino cluster shape adds simplified hive metastore and catalogs configuration
- HDInsight on AKS has simplified external Hive metastore configuration for Trino cluster, you can now specify external metastore in config.properties and enable it for each catalog with single parameter, learn more about enhancements here.
- Trino cluster shape adds sharded sql connector
- Flink clusters now support Flink 1.17.0 from HDInsight on AKS 1.1.0 release
- HDInsight on AKS now supports Flink 1.17.0 release, with significant improvements on checkpoints, subtask level flame graph, watermark alignments. Learn more about the Flink 1.17 release here
- Flink SQL Gateway is now supported from HDInsight on AKS 1.1.0 release with Flink session clusters
Bug Fixes & CVEs
- This release includes several critical CVE fixes across the platform and open source components.
- Trino cluster shape excludes system tables from caching automatically
- Trino cluster shape improves Power BI timestamp timezones handling
Known issues
- Workload identity limitation:
- There is a known limitation when transitioning to workload identity. This is due to the permission-sensitive nature of FIC operations. Users cannot perform deletion of a cluster by deleting the resource group. Cluster deletion requests must be triggered by the application/user/principal with FIC/delete permissions. In case, the FIC deletion fails, the high-level cluster deletion will also fail.
New regions
- East Asia
Operating System version
- Mariner OS 2.0
Workload versions
| Workload | Version |
|---|---|
| Trino | 426 |
| Flink | 1.17.0 |
| Apache Spark | 3.3.1 |
Supported Java and Scala versions
| Workload | Java | Scala |
|---|---|---|
| Trino | Open JDK 17.0.7 | - |
| Flink | Open JDK 11.0.21 | 2.12.7 |
| Spark | Open JDK 1.8.0_345 | 2.12.15 |
The preview is available in the following regions.
If you have any more questions, contact Azure Support or refer to the Support options page. If you have product specific feedback, write us on aka.ms/askhdinsight.
Azure HDInsight on AKS Release 2023-12-13
Release date: December 13, 2023
This hotfix release applies to the following
- Cluster Pool Version: 1.0
- Cluster Version: 1.0.6
Known Issues
-
Secure Tenants User Interface Support
- This release addresses an issue where the open source component web URLs for HDInsight on AKS Clusters were inaccessible.
How to apply the Hotfix
- Recreate Your Cluster
- To apply this hotfix, existing users are required to recreate their cluster on an existing cluster pool.
Operating System version
- Mariner OS 2.0
Workload versions
| Workload | Version |
|---|---|
| Trino | 410 |
| Flink | 1.16 |
| Apache Spark | 3.3.1 |
Supported Java and Scala versions
| Workload | Java | Scala |
|---|---|---|
| Trino | Open JDK 17.0.7 | - |
| Flink | Open JDK 11.0.21 | 2.12.7 |
| Spark | Open JDK 1.8.0_345 | 2.12.15 |
The preview is available in the following regions.
Note
To understand about HDInsight on AKS versioning and support, refer to the versioning page.
You can refer to What's new page for all the details of the features currently in public preview for this release.
If you have any more questions, contact Azure Support or refer to the Support options page.