Issue when creating Azure blob container within Azure China Cloud

[GBT]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Cloud Azure (azure_rm_storageblob)

ANSIBLE VERSION

ansible 2.4.2.0
  config file = None
  configured module search path = [u'/home/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

CONFIGURATION

Nothing changed

OS / ENVIRONMENT

N/A

SUMMARY

We are using the module azure_rm_storageblob to create a container within a blob storage.
The module use ".blob.core.windows.net" for domain name instead of ".blob.core.chinacloudapi.cn" when
cloud_environment is AzureChinaCloud

STEPS TO REPRODUCE

Within Azure Cloud China,
create a storage account, then create a blob container within the storage account created previously.

- name: "{{ storage_account_name }}"
  azure_rm_storageaccount:
    tenant: "{{ tenant }}"
    client_id: "{{ client_id }}"
    secret: "{{secret}}"
    subscription_id: "{{ subscription_id }}"
    resource_group: "{{ storage_account_resource_group_name }}"
    name: "{{ storage_account_name }}"
    location: "{{ location }}"
    cloud_environment: "{{ cloud_environment }}"
    account_type: Standard_RAGRS
    state: present
    tags :
        type_name: storageAccount

- name : Create storage feeder container
  azure_rm_storageblob:
    tenant: "{{ tenant }}"
    client_id: "{{ client_id }}"
    secret: "{{secret}}"
    subscription_id: "{{ subscription_id }}"
    resource_group: "{{ storage_account_resource_group_name }}"
    storage_account_name: "{{ storage_account_name }}"
    cloud_environment: "{{ cloud_environment }}"
    container: "mycontainer"
    state : present

EXPECTED RESULTS

Blob container is created

ACTUAL RESULTS

ConnectTimeout: HTTPSConnectionPool(host='blobname.blob.core.windows.net', port=443):

An exception occurred during task execution. To see the full traceback, use -vvv. 
The error was: azure.common.AzureException: ConnectTimeout: HTTPSConnectionPool(host='blobname.blob.core.windows.net', port=443): Max retries exceeded with url: /datalake-feeder?restype=container (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x297b110>, 'Connection to blobname.blob.core.windows.net timed out. (connect timeout=20)'))

fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_T4Toi_/ansible_module_azure_rm_storageblob.py\", line 562, in <module>\n    main()\n  File \"/tmp/ansible_T4Toi_/ansible_module_azure_rm_storageblob.py\", line 559, in main\n    AzureRMStorageBlob()\n  File \"/tmp/ansible_T4Toi_/ansible_module_azure_rm_storageblob.py\", line 256, in __init__\n    supports_tags=True)\n  File \"/tmp/ansible_T4Toi_/ansible_modlib.zip/ansible/module_utils/azure_rm_common.py\", line 285, in __init__\n  File \"/tmp/ansible_T4Toi_/ansible_module_azure_rm_storageblob.py\", line 268, in exec_module\n    self.container_obj = self.get_container()\n  File \"/tmp/ansible_T4Toi_/ansible_module_azure_rm_storageblob.py\", line 325, in get_container\n    container = self.blob_client.get_container_properties(self.container)\n  File \"/usr/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py\", line 635, in get_container_properties\n    return self._perform_request(request, _parse_container, [container_name])\n  File \"/usr/lib/python2.7/site-packages/azure/storage/storageclient.py\", line 280, in _perform_request\n    raise ex\nazure.common.AzureException: ConnectTimeout: HTTPSConnectionPool(host='blobname.blob.core.windows.net', port=443): Max retries exceeded with url: /datalake-feeder?restype=container (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x297b110>, 'Connection to blobname.blob.core.windows.net timed out. (connect timeout=20)'))\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/azure/azure_rm_storageblob.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @haroldwongms @nitzmahone @trstringer @xscript @yuwzho @zikalino
click here for bot help

[kyliel]

Thank you for reporting the bug. @yuwzho and @zikalino , could we have a quick fix for Azure China? Thanks.

[Fred-sun]

Hi @GBT ,I have test on ansible 2.4.2 and 2.6.1,all get pass result ,Could help try again, it may has been fixed, if you test OK ,we will close it , Thanks !

[Fred-sun]

Hi @GBT Could you help update the issue status? Thanks!

[yungezz]

this should be fixed already. Currently, storage client is get based on cloud environment. close the issue for now, pls cut one if met any issue using it.

[GBT]

Did you test with Azure China Cloud ? do you have an 21vianet account ?
The issue is that the FQDN of servers are not from .blob.core.windows.net domain, there are from .blob.core.chinacloudapi.cn domain.

[yungezz]

@GBT azure_rm_storageblob module doesn't hardcode specify blob FQDN name. The blob name is got via blob client.When getting blob client, ansible will get correct azure cloud resource endpoint via configuration. Code snippets:
https://github.com/VSChina/ansible/blob/ee5b761c21fde45363e204b7d886fc775e73ba06/lib/ansible/modules/cloud/azure/azure_rm_storageblob.py#L256
https://github.com/VSChina/ansible/blob/ee5b761c21fde45363e204b7d886fc775e73ba06/lib/ansible/module_utils/azure_rm_common.py#L984-L989.

To let ansible knows which azure cloud you're using, pls specify cloud environment configuration:

• by environment variable:

export AZURE_CLOUD_ENVIRONMENT=AzureChinaCloud

• in configuration file, add item

cloud_environment=AzureChinaCloud

let us know you met any other issue.

[GBT]

@yungezz I already use cloud_environment in my tasks (check example in summary)

- name: "Creation Container"
  azure_rm_storageblob:
  ...
    cloud_environment: "{{ cloud_environment }}"
  ...

It works good for Storage account creation, but it fails for container creation.

Here is below the error if it can help, pls note the host cqcnppsatest.blob.core.windows.net when cloud_environment is AzureChinaCloud

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "module_stderr": "No handlers could be found for logger \"azure.storage.common.storageclient\"\nTraceback (most recent call last):\n  File \"/tmp/ansible_Vtbrqj/ansible_module_azure_rm_storageblob.py\", line 562, in <module>\n    main()\n  File \"/tmp/ansible_Vtbrqj/ansible_module_azure_rm_storageblob.py\", line 559, in main\n    AzureRMStorageBlob()\n  File \"/tmp/ansible_Vtbrqj/ansible_module_azure_rm_storageblob.py\", line 256, in __init__\n    supports_tags=True)\n  File \"/tmp/ansible_Vtbrqj/ansible_modlib.zip/ansible/module_utils/azure_rm_common.py\", line 285, in __init__\n  File \"/tmp/ansible_Vtbrqj/ansible_module_azure_rm_storageblob.py\", line 268, in exec_module\n    self.container_obj = self.get_container()\n  File \"/tmp/ansible_Vtbrqj/ansible_module_azure_rm_storageblob.py\", line 325, in get_container\n    container = self.blob_client.get_container_properties(self.container)\n  File \"/usr/lib/python2.7/site-packages/azure/storage/blob/baseblobservice.py\", line 653, in get_container_properties\n    return self._perform_request(request, _parse_container, [container_name])\n  File \"/usr/lib/python2.7/site-packages/azure/storage/common/storageclient.py\", line 354, in _perform_request\n    raise ex\nazure.common.AzureException: ConnectionError: HTTPSConnectionPool(host='myblob.blob.core.windows.net', port=443): Max retries exceeded with url: /mycontainer?restype=container (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9243b12710>: Failed to establish a new connection: [Errno -2] Nom ou service inconnu',))\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}

[yungezz]

@GBT thanks for the info. After look at code, found a fix in PR #44802 . Could you pls try the fix in Azure China cloud to validate it? Thanks.

[GBT]

@yungezz I validated it and my test passed :-) thank you

I updated the PR for my version. Please find it below :

--- azure_rm_common.py.orig     2018-08-29 07:25:18.454348724 +0000
+++ azure_rm_common.py  2018-08-29 07:35:53.036583692 +0000
@@ -114,6 +114,7 @@
     from azure.mgmt.web import WebSiteManagementClient
     from azure.mgmt.containerservice import ContainerServiceClient
     from azure.storage.cloudstorageaccount import CloudStorageAccount
+    from azure.storage.blob import PageBlobService, BlockBlobService
 except ImportError as exc:
     HAS_AZURE_EXC = exc
     HAS_AZURE = False
@@ -573,7 +574,9 @@
 
         try:
             self.log('Create blob service')
-            return CloudStorageAccount(storage_account_name, account_keys.keys[0].value).create_block_blob_service()
+            return PageBlobService(endpoint_suffix=self._cloud_environment.suffixes.storage_endpoint,
+                                       account_name=storage_account_name,
+                                       account_key=account_keys.keys[0].value)
         except Exception as exc:
             self.fail("Error creating blob service client for storage account {0} - {1}".format(storage_account_name,
                                                                                                 str(exc)))

[yungezz]

@GBT thanks for validating. The fix will be in ansible 2.7 release. And we'll see what's next ansible 2.6 minor release train, and backport it there.

[yungezz]

#44802 merged, backporting to stable2.6 and stable 2.7.