hadoop

hadoop-azure-token-providers

The package was built in order to enable a simple way of accessing data in Azure Data Lake Gen2 or OneLake when developing a PySpark solution locally in a Dev Container.

The hadoop-azure library supports supplying a custom token provider for getting accessing tokens when using the abfs(s):// protocol for accessing Azure Data Lake / OneLake.

This package contains implementations of custom token providers (currently only AzureCliCredentialTokenProvider).

Development

Open the repository in Visual Studio Code and use the provided Dev Container, which has the necessary Java SDK and Maven tools installed.

Building the package

Use Maven to build the package:

mvn clean package

Deploying the package

The package is currently configured to deploy to the public hadoop Artifacts feed in the endjin-labs Azure DevOps instance as com.endjin.hadoop:hadoop-azure-token-providers.

You need to provide credentials in a settings file to deploy to the feed. Create a settings.xml file in the ~/.m2 folder with the following contents:

<settings>
  <servers>
    <server>
      <id>hadoop</id>
      <username>endjin-labs</username>
      <password>{Personal Access Token}</password>
    </server>
  </servers>
</settings>

Replace {Personal Access Token} with a Personal Access Token generated from the endjin-labs Azure DevOps instance with permission to read/write Packages.

In the pom.xml file, set the desired version number in the <version> tag, then use Maven to deploy the package:

mvn deploy

TODO: Add GitHub Actions workflow for automatically versioning & deploying the package

Usage from PySpark

To use the custom token providers in PySpark, include the org.apache.hadoop:hadoop-azure and com.endjin.hadoop:hadoop-azure-token-providers packages when creating your Spark session. You will also need to add the endjin-labs hadoop Artifacts feed as an extra Maven repository.

e.g.

builder = (
    SparkSession.builder.appName('my-app')
    .master("local[*]")
    .config(
        "spark.jars.repositories", 
        "https://pkgs.dev.azure.com/endjin-labs/hadoop/_packaging/hadoop/maven/v1"
    )
)

extra_packages = [
    'org.apache.hadoop:hadoop-azure:3.3.3',
    'com.endjin.hadoop:hadoop-azure-token-providers:1.0.1'
]

spark = configure_spark_with_delta_pip(builder, extra_packages=extra_packages).getOrCreate()

To configure one of the custom token providers, set the relevant Hadoop configuration, e.g. for AzureCliCredentialTokenProvider:

spark.sparkContext._jsc.hadoopConfiguration().set(
    "fs.azure.account.auth.type",
    "Custom"
)
spark.sparkContext._jsc.hadoopConfiguration().set(
    "fs.azure.account.oauth.provider.type",
    "com.endjin.hadoop.fs.azurebfs.custom.AzureCliCredentialTokenProvider"
)

This will then use the token provider by default for getting access tokens for all accounts when using abfs(s):// protocol.

AzureCliCredentialTokenProvider

This token provider relies on the credentials from the Azure CLI in order to retrieve an access token for Azure Storage. See these instructions for how to install the Azure CLI in your development environment.

When AzureCliCredentialTokenProvider is configured, ensure that az login has been run (for the relevant tenant) before attempting to use Spark to access data in the Data Lake / OneLake.