resources.GroupClient methods fail due to invalid, hard-coded API version

[axw]

I'm in the process of updating Juju to use to the latest Azure SDK for Go. One of our code paths uses resources.GroupClient to update the tags for all resources in a resource group.

The clients previously stored APIVersion as a field, which we could override, and used that as the value of the api-version query parameter in requests. Now, the API version is hard-coded inside client methods and cannot be overridden. That's fine when dealing with a specific resource type, but when dealing with generic resources, the caller must be able to specify the API version of the resource type.

machine-0: 10:26:07 ERROR juju.provider.azure error updating resource tags for "juju42638ed0f31b286323c0": getting full resource "juju42638ed0f31b286323c0": resources.GroupClient#Get: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="NoRegisteredProviderFound" Message="No registered resource provider found for location 'australiasoutheast' and API version '2016-09-01' for type 'storageAccounts'. The supported api-versions are '2017-06-01, 2016-12-01, 2016-05-01, 2016-01-01, 2015-06-15, 2015-05-01-preview'. The supported locations are 'eastus, eastus2, westus, westeurope, eastasia, southeastasia, japaneast, japanwest, northcentralus, southcentralus, centralus, northeurope, brazilsouth, australiaeast, australiasoutheast, southindia, centralindia, westindia, canadaeast, canadacentral, westus2, westcentralus, uksouth, ukwest, koreacentral, koreasouth'."

[marstr]

Howdy @axw,

This is a known issue. To get you unblocked immediately, you can hack your way through by creating a custom Preparer method to wrap the call with an adjustment to an API Version that you choose. For context, you can find a GroupClient PreparerMethod here:

azure-sdk-for-go/arm/resources/resources/groups.go

Line 153 in 63bda4f

func (client GroupsClient) CreateOrUpdatePreparer(resourceGroupName string, parameters Group) (*http.Request, error) {

However, the underlying problem you're hitting is insidious, and something we've been working to fix for quite some time now. Basically, the trouble is that each API Version of a service in Azure is independent from the others. This means that between API Versions there is no guarantee that models generated to target one API Version will match the ones need to call another. I've created an experimental branch that exposes models for all API Versions here:

azure-sdk-for-go/service/resources/management in 8150bd

The majority of the time, things should just work if you tweak the API Version parameter, but we want to expose a better more static way of doing things. Hence the proliferation of packages.

I hope this is helpful!

-Martin

[axw]

@marstr thanks for the suggestion. I should have said to start with, and saved you some time, that I worked around it. I did something similar: I just called the Preparer/Sender/Responder in sequence, changing the URL after the Preparer.

Re your branch: I'm glad to see per-version code generation, but I don't think that helps with the problem I'm dealing with, does it? In case we're talking about different things, I'll restate in a bit more detail. In my case, what I'm doing is:

• list providers in the subscription, to get the API versions for each resource type/provider
• list all resources in the resource group (returned as resources.GenericResource) to get, for each resource, ID and type
• for each resource: call GetByID, modify the GenericResource's Tags and call CreateOrUpdateByID

In the GetByID and CreateOrUpdateByID calls, we must specify resource-type specific API versions. We pick the most recent API version returned in the provider listing for each type.

(If you have any alternative suggestions on ways to update tag values for all resources in a resource group, I'd be happy to hear them.)

[marstr]

Thanks for getting back to me, @axw. I see what you're saying now. I'll go chase down what the service teams suggest in terms of dispatching on API Version based on resource type and get back to you.

[itaysk]

@axw can you please share your solution implementation? when I try to call groupClient.GetByIDPreparer it fails on No scheme detected in URL. groupClient was initialized with subscription id, but BaseURI is still empty.

[axw]

@itaysk sure. It's got all of our project's foibles woven in, so ask questions if anything's unclear.

updateResourceControllerTag is the method that calls GetByID and CreateOrUpdateByID: https://github.com/juju/juju/blob/5c90f6bd8c80efbff53d616e591105b074423eea/provider/azure/environ.go#L1312.

updateResourceControllerTag is called by AdoptResources: https://github.com/juju/juju/blob/5c90f6bd8c80efbff53d616e591105b074423eea/provider/azure/environ.go#L1242. In AdoptResources we construct a resources.GroupClient from a resources.ManagementClient we prepared earlier. The ManagementClient is contructed here: https://github.com/juju/juju/blob/5c90f6bd8c80efbff53d616e591105b074423eea/provider/azure/environ.go#L174. env.cloud.Endpoint will be "https://management.azure.com" for the main public Azure.

The code to fetch resource provider IDs, called by AdoptResources: https://github.com/juju/juju/blob/5c90f6bd8c80efbff53d616e591105b074423eea/provider/azure/utils.go#L64

[itaysk]

@axw Thanks!

[itaysk]

@axw your ResourcesClient is a handy fix in many scenarios, and I wanted to import directly from your repo, but that's not possible since it's under an "internal" path. I want to repost it as a package, probably on my github. Do you mind if I did that? would you prefer to host it under yours? (just doesn't feel right "stealing" this without your permission)

[axw]

@itaysk I'm happy for you to take a copy, but I just realised I put the wrong license on the code. It should be licensed the same as the azure-sdk-for-go code, as it was derived from it; instead it is licensed AGPLv3, like the rest of Juju. I'll fix that.

[axw]

juju/juju#7819 - just needs someone to sign off

[salameer]

Ping @marstr