Skip to content

Azure HDInsight Release 2023-10-26

Choose a tag to compare

View all tags
@v-akarnase v-akarnase released this 26 Oct 12:23
2023-10-26
3aa50bc
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Release date: October 26, 2023

This release applies to HDInsight 4.x and 5.x versions. HDInsight release will be available to all regions over several days. This release is applicable for image number 2310140056. How to check the image number?

HDInsight uses safe deployment practices, which involve gradual region deployment. it might take up to 10 business days for a new release or a new version to be available in all regions.

OS versions

  • HDInsight 4.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4
  • HDInsight 5.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4
  • HDInsight 5.1: Ubuntu 18.04.5 LTS Linux Kernel 5.4

For workload specific versions, see

What's new

  • HDInsight announces the General availability of HDInsight 5.1 starting November 1, 2023. This release brings in a full stack refresh to the open source components and the integrations from Microsoft.

    • Latest Open Source Versions – HDInsight 5.1 comes with the latest stable open-source version available. Customers can benefit from all latest open source features, Microsoft performance improvements, and Bug fixes.
    • Secure – The latest versions come with the most recent security fixes, both open-source security fixes and security improvements by Microsoft.
    • Lower TCO – With performance enhancements customers can lower the operating cost, along with enhanced autoscale.

  • Cluster permissions for secure storage

    • Customers can specify (during cluster creation) whether a secure channel should be used for HDInsight cluster nodes to connect the storage account.

  • HDInsight Cluster Creation with Custom VNets.

    • To improve the overall security posture of the HDInsight clusters, HDInsight clusters using custom VNETs need to ensure that the user needs to have permission for Microsoft Network/virtualNetworks/subnets/join/action to perform create operations. Customer might face creation failures if this check is not enabled.

  • Non-ESP ABFS clusters [Cluster Permissions for Word Readable]

    • Non-ESP ABFS clusters restrict non-Hadoop group users from executing Hadoop commands for storage operations. This change improves cluster security posture.

  • In-line quota update.

    • Now you can request quota increase directly from the My Quota page, with the direct API call it is much faster. In case the API call fails, you can create a new support request for quota increase.

Coming soon

  • The max length of cluster name will be changed to 45 from 59 characters, to improve the security posture of clusters. This change will be rolled out to all regions starting upcoming release.

  • Basic and Standard A-series VMs Retirement.

    • On August 31, 2024, we will retire Basic and Standard A-series VMs. Before that date, you need to migrate your workloads to Av2-series VMs, which provide more memory per vCPU and faster storage on solid-state drives (SSDs).
    • To avoid service disruptions, migrate your workloads from Basic and Standard A-series VMs to Av2-series VMs before August 31, 2024.

If you have any more questions, contact Azure Support.

You can always ask us about HDInsight on Azure HDInsight - Microsoft Q&A

We are listening : You’re welcome to add more ideas and other topics here and vote for them - HDInsight Ideas and follow us for more updates on AzureHDInsight Community

Note

This release addresses the following CVEs released by MSRC on September 12, 2023. The action is to update to the latest image 2308221128 or 2310140056. Customers are advised to plan accordingly.

CVE Severity CVE Title Remark
CVE-2023-38156 Important Azure HDInsight Apache Ambari Elevation of Privilege Vulnerability Included on image 2308221128 or 2310140056
CVE-2023-36419 Important Azure HDInsight Apache Oozie Workflow Scheduler Elevation of Privilege Vulnerability Apply Script action on your clusters, or update to 2310140056 image

Note

We advise customers to use to latest versions of HDInsight Images as they bring in the best of open source updates, Azure updates and security fixes. For more information, see Best practices.

Assets 2
Loading