RFE: AzureStack -- auto-discovery of Azure endpoint URLs

[bsquizz]

Instead of hard-coding URLs in the Environment, we should be able to auto-discover them given only the resource manager URL using a GET to <ARM_URL>/metadata/endpoints?api-version=1.0

This gives us details on the gallery, graph, and portal URLs, as well as the active_directory_authority and active_directory_resource_id

This will be helpful for AzureStack deployments where the underlying URLs can't be hard-coded. When adding the provider, all the admin would need to provide is the resource manager URL and we could take care of the rest.

See an example in Ruby here (search for def get_active_directory_settings(armEndpoint)):
https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-version-profiles-ruby

JSON returned by an azure stack deployment:

{
    "galleryEndpoint": "https://portal.westus.stackpoc.com:30015/",
    "graphEndpoint": "https://graph.windows.net/",
    "portalEndpoint": "https://portal.westus.stackpoc.com/",
    "authentication": {
        "loginEndpoint": "https://login.windows.net/",
        "audiences": ["https://management.poc.avahc.com/01a7976d-5a10-475d-b39d-01993f9d7d90"]
    }
}

This same API is working on the public cloud too. Example JSON:

{
    "galleryEndpoint": "https://gallery.azure.com/",
    "graphEndpoint": "https://graph.windows.net/",
    "portalEndpoint": "https://portal.azure.com/",
    "authentication": {
        "loginEndpoint": "https://login.windows.net/",
        "audiences": ["https://management.core.windows.net/", "https://management.azure.com/"]
    }
}

[bsquizz]

Based on my research, there's no reliable way to discover the storage endpoint suffix or key vault suffix, so we will still need to require the admin to manually specify those values.

We could do a "blend" of trying to discover, and falling back to requiring the values to be manually specified if they can't be determined.

A possible way to discover these is by looking at storage account info via ARM at /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.Storage/storageaccounts and looking a the URL suffixes of primaryEndpoints. However if there's no storage accounts, this obviously won't work :)

Same goes for Key Vaults, which could be listed at /subscriptions/<subscription ID>/resourceGroups/<resource group name>/providers/Microsoft.KeyVault/vaults and looking at the suffix of the vaultUri.

Example storage acct JSON:

{
    "sku": {
        "name": "Standard_LRS",
        "tier": "Standard"
    },
    "kind": "Storage",
    "id": "/subscriptions/ca3444ad-9e61-4360-85f6-fc71368a73e4/resourcegroups/resourcegroup/providers/Microsoft.Storage/storageaccounts/storageacct",
    "type": "Microsoft.Storage/storageaccounts",
    "name": "storageacct",
    "location": "westus",
    "properties": {
        "provisioningState": "Succeeded",
        "primaryEndpoints": {
            "blob": "https://storageacct.blob.westus.stackpoc.com/",
            "queue": "https://storageacct.queue.westus.stackpoc.com/",
            "table": "https://storageacct.table.westus.stackpoc.com/"
        },
        "primaryLocation": "westus",
        "statusOfPrimary": "Available",
        "encryption": {
            "services": {
                "blob": {
                    "enabled": true,
                    "lastEnabledTime": "2018-03-29T14:53:30.9400000"
                }
            },
            "keySource": "Microsoft.Storage"
        },
        "creationTime": "2018-03-29T14:53:30.9400000"
    }
}

Example Vault JSON:

{
    "id": "/subscriptions/ca3444ad-9e61-4360-85f6-fc71368a73e4/resourceGroups/resourcegroup/providers/Microsoft.KeyVault/vaults/vault",
    "name": "vault",
    "type": "Microsoft.KeyVault/vaults",
    "location": "westus",
    "tags": {},
    "properties": {
        "sku": {
            "family": "A",
            "name": "Standard"
        },
        "tenantId": "a3f33ab9-6362-4609-b44d-a669107ebd9d",
        "accessPolicies": [
            {
                "tenantId": "a3f33ab9-6362-4609-b44d-a669107ebd9d",
                "objectId": "7ad1f39b-75d9-48ba-9ff7-7158c6a4b9b3",
                "permissions": {
                    "keys": [
                        "Get",
                        "List",
                        "Update",
                        "Create",
                        "Import",
                        "Delete",
                        "Recover",
                        "Backup",
                        "Restore"
                    ],
                    "secrets": [
                        "Get",
                        "List",
                        "Set",
                        "Delete",
                        "Recover",
                        "Backup",
                        "Restore"
                    ],
                    "certificates": [
                        "Get",
                        "List",
                        "Update",
                        "Create",
                        "Import",
                        "Delete",
                        "Recover",
                        "ManageContacts",
                        "ManageIssuers",
                        "GetIssuers",
                        "ListIssuers",
                        "SetIssuers",
                        "DeleteIssuers"
                    ]
                }
            }
        ],
        "enabledForDeployment": false,
        "enabledForDiskEncryption": false,
        "enabledForTemplateDeployment": false,
        "vaultUri": "https://vault.vault.westus.stackpoc.com/"
    }
}

[bsquizz]

We can discover supported API versions/API profiles using ARM as well, using /providers/<provider type>, for example, here's a snip of the JSON for /providers/Microsoft.Network:

{
    "namespace": "Microsoft.Network",
    "resourceTypes": [
        {
            "resourceType": "connections",
            "locations": [
                "westus"
            ],
            "apiVersions": [
                "2015-06-15",
                "2015-05-01-preview"
            ],
            "apiProfiles": [
                {
                    "profileVersion": "2017-03-09-profile",
                    "apiVersion": "2015-06-15"
                }
            ]
        },
        {
            "resourceType": "loadBalancers",
            "locations": [
                "westus"
            ],
            "apiVersions": [
                "2015-06-15",
                "2015-05-01-preview"
            ],
            "apiProfiles": [
                {
                    "profileVersion": "2017-03-09-profile",
                    "apiVersion": "2015-06-15"
                }
            ]
        },

[djberg96]

@bsquizz The library will automatically determine the latest api-version for each service. For manageiq, we'll need to decide on the semantics. At the moment it looks like this:

:ems:
  :ems_azure:
    :api_versions:
      :availability_set: "2017-12-01"
      :ip_address: "2017-11-01"

As I mentioned on the call, I'm thinking we change it to something like this:

:ems:
  :ems_azure:
    :default
      :api_versions:
        :availability_set: "2017-12-01"
        :ip_address: "2017-11-01"
    :your_provider
        :availability_set: "2016-11-30" # or whatever
        :ip_address: "2016-07-30" # ditto

If the provider name matches a config entry, we use those api-version strings. Otherwise we use the default settings.

[djberg96]

@bsquizz Let's take the discussion on how to handle this over at the Azure provider repo. For now, as long as the gem has the ability to autodiscover, we can close this issue.

Once the PR is merged, I can push out a new version of the gem, and update the gemspec, and we can start using the new methods.

[bsquizz]

Sure, that's fine, just wanted to collect all 'discovery' related info I had looked into here in 1 place. I wasn't 100% certain if we did the version discovery or not.

[djberg96]

Now part of 0.9.11, which is also set in the current azure provider gemspec.