Deadlock when getting a token in asynchronous mode

[TiunovNN]

• Package Name: azure-core
• Package Version: 1.5.0
• Operating System: Ubuntu 14/18/20
• Python Version: 3.6.5, 3.8.2

Describe the bug
I've faced a deadlock when trying to get access to several blobs.

To Reproduce
Code:

import asyncio
import logging
import os

from azure.core.exceptions import HttpResponseError
from azure.identity.aio import ClientSecretCredential
from azure.storage.blob.aio import BlobServiceClient


logging.basicConfig(level=logging.DEBUG)

BLOB_ACCOUNT = os.getenv('AZURE_BLOB_ACCOUNT')
CONTAINER_NAME = os.getenv('AZURE_BLOB_CONTAINER')
TENANT_ID = os.getenv('AZURE_TENANT_ID')
CLIENT_ID = os.getenv('AZURE_CLIENT_ID')
SECRET = os.getenv('AZURE_SECRET')

FILENAME = 'data.txt'

DATA = b'check_data_test'


async def check_writing(client: BlobServiceClient, filename):
    print('check writing: ', filename)
    blob_client = client.get_blob_client(CONTAINER_NAME, filename)
    try:
        await blob_client.upload_blob(DATA)
    except HttpResponseError as e:
        print('Writing fail', client.account_name, str(e))
    else:
        print('Writing OK: ', client.account_name)


async def main():
    credential = ClientSecretCredential(TENANT_ID, CLIENT_ID, SECRET)
    client = BlobServiceClient(
        account_url=f'https://{BLOB_ACCOUNT}.blob.core.windows.net',
        credential=credential,
    )
    async with client:
        tasks = []
        for i in range(10):
            tasks.append(check_writing(client, FILENAME + str(i)))
        await asyncio.gather(*tasks)


if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())

1. Run this script
2. It got frozen.

Expected behavior
The script must work correctly.

Additional context
Traceback after killing of the task:

Traceback (most recent call last):
  File "/home/wolf/PycharmProjects/test/azure_test/blob_test/blob_storage-sample.py", line 48, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/home/wolf/PycharmProjects/test/azure_test/blob_test/blob_storage-sample.py", line 27, in check_writing
    await blob_client.upload_blob(DATA)
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/storage/blob/aio/_blob_client_async.py", line 256, in upload_blob
    return await upload_block_blob(**options)
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/storage/blob/aio/_upload_helpers.py", line 69, in upload_block_blob
    return await client.upload(
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/storage/blob/_generated/aio/operations_async/_block_blob_operations_async.py", line 207, in upload
    pipeline_response = await self._client._pipeline.run(request, stream=False, **kwargs)
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 212, in run
    return await first_node.send(pipeline_request)
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 3 more times]
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 81, in send
    await _await_result(self._policy.on_request, request)
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/_tools_async.py", line 32, in await_result
    return await result  # type: ignore
  File "/home/wolf/PycharmProjects/test/venv/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication_async.py", line 36, in on_request
    with self._lock:
KeyboardInterrupt

Process finished with exit code 130 (interrupted by signal 2: SIGINT)

The reason is here: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/core/azure-core/azure/core/pipeline/policies/_authentication_async.py#L24

When second task is trying to get lock for the thread, this causes the program to freeze.

The possible solution is replacing threading.Lock with asyncio.Lock

[kaerm]

Thanks @TiunovNN for reporting this! Tagging the right folks to take a look at this: @lmazuel @xiangyan99

[chlowell]

Thanks again for reporting this. I opened a PR to fix it.

@xiafu-msft, it's interesting we haven't seen this deadlock in our tests. It appears the storage test suite doesn't cover concurrent operations similar to the snippet above?

[lmazuel]

Merged for release in azure-core 1.6.0