Add async hook for Azure Synapse pipelines#62966
Open
SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
Open
Add async hook for Azure Synapse pipelines#62966SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
Conversation
boring-cyborg
bot
added
area:providers
provider:microsoft-azure
SameerMesiah97
force-pushed
the
AzureSynapsePipelineAsyncHook
branch
from
df4a2ca to
8872774
Compare
Introduce AzureSynapsePipelineAsyncHook as an asynchronous counterpart to the existing AzureSynapsePipelineHook. The async hook mirrors the synchronous hook’s credential resolution and client creation logic while using AsyncArtifactsClient for non-blocking pipeline run retrieval, status checks, and cancellation operations. It supports both client-secret and default credential authentication using their asynchronous Azure identity credential equivalents. Add unit tests covering async client creation, credential selection, pipeline run status retrieval, client caching, connection refresh, and proper cleanup via close() and the async context manager.
SameerMesiah97
force-pushed
the
AzureSynapsePipelineAsyncHook
branch
from
8872774 to
7b37279
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change introduces an asynchronous hook for Azure Synapse Pipelines (
AzureSynapsePipelineAsyncHook).Previously, only a synchronous hook (
AzureSynapsePipelineHook) was available. This update adds an asynchronous counterpart that mirrors the synchronous hook’s credential resolution and client creation logic usingazure.synapse.artifacts.aio.AsyncArtifactsClient.The async hook provides non-blocking methods for retrieving pipeline runs, checking pipeline run status, and cancelling pipeline runs. It supports both client-secret and default credential authentication using their asynchronous Azure identity credential equivalents.
Rationale
This change introduces the asynchronous hook as a prerequisite for implementing deferrable mode for Azure Synapse Pipeline operators.
Within the Azure provider, several services already support or are introducing deferrable execution. The Azure Data Factory (ADF) operator supports deferrable mode, and deferrable support for Azure Container Instances (ACI) is currently being introduced. Azure Synapse Pipelines should provide equivalent deferrable capabilities to maintain consistency across Azure services within the provider.
Deferrable operators require non-blocking SDK interactions within triggers running in the triggerer process. The existing synchronous Synapse hook performs blocking I/O and is therefore unsuitable for use in triggers. Introducing an async hook provides the necessary foundation for implementing deferrable support for Synapse Pipeline operators while maintaining behavioral parity with the synchronous hook.
Tests
Added unit tests verifying that:
get_pipeline_run_status()returns the expected status.AsyncClientSecretCredential.AsyncDefaultAzureCredential.ValueErroris raised when tenant ID is missing for client secret authentication.get_async_conn()reuses the cached async client.refresh_conn()recreates the async client.close()properly closes and clears the async client.close()on exit.Documentation
A docstring has been added to
AzureSynapsePipelineAsyncHookexplaining its usage.Backwards Compatibility
This change does not modify public APIs or alter existing behavior.