migrations: implement AdoptResources for azure provider #6868

Merged
merged 5 commits into from Jan 31, 2017

Conversation

Projects
None yet
4 participants
Member

babbageclunk commented Jan 25, 2017

Description of change

The AdoptResources environ method will be used at the end of a model migration to update the controller tag for all of the resources used by the model, so that if the source controller is subsequently destroyed then these resources won't be cleaned up with it.

QA steps

None yet - this method isn't called from the migration master yet.

Documentation changes

N/A

Bug reference

This is part of the fix for https://bugs.launchpad.net/juju/+bug/1648063

provider/azure/environ.go
+ for resources.Value != nil {
+ for _, resource := range *resources.Value {
+
+ providerNamespace, parentPath, err := splitResourceType(*resource.Type)
@babbageclunk

babbageclunk Jan 25, 2017

Member

This is the bit I'm least certain about - from looking at the places constructing Resources in the same file this seems to be what the Type attribute is. It generally looks like "Microsoft.Compute/availabilitySets" - so this would have the provider namespace "Microsoft.Compute" and parent path "availabilitySets".
But I don't really get why it would be needed given that we're also passing the type as another field? (And in the resource as well.) So maybe I'm misunderstanding? Or is the redundancy just an artifact of API autogeneration?

@babbageclunk

babbageclunk Jan 25, 2017

Member

I've signed up for an Azure trial to try to work questions like this out, but I haven't managed yet.

@axw

axw Jan 25, 2017

Member

Ah, actually I don't think this bit is right.

Microsoft.Compute is the namespace, availabilitySets is the resource type; but it is not a parent resource type, as availability sets don't have a parent.

If there were other bits in between Microsoft.Compute and availabilitySets, the other bits would constitute the parent path.

@babbageclunk

babbageclunk Jan 25, 2017

Member

Ok, I had a feeling it might be something like that. I'll change the splitting to take the first component as the the namespace, the last component as the type and the parent path as anything in between.

I'll do a deploy to azure (should have done that first) and then I can see what comes back in the Type field when I call ListResources.

provider/azure/environ.go
+ logger.Errorf("error getting namespace and path for type %q: %v", *resource.Type, err)
+ }
+
+ (*resource.Tags)[tags.JujuController] = &controllerUUID
@axw

axw Jan 25, 2017

Member

maybe log.Debugf here with the details of the resource being updated?

@babbageclunk

babbageclunk Jan 25, 2017

Member

Yeah, good call - added.

provider/azure/environ.go
+ logger.Errorf("error getting namespace and path for type %q: %v", *resource.Type, err)
+ }
+
+ (*resource.Tags)[tags.JujuController] = &controllerUUID
@axw

axw Jan 25, 2017

Member

check that the resource isn't already tagged with this controller UUID? is it possible for this function to be reentrant?

@babbageclunk

babbageclunk Jan 25, 2017

Member

Good point - I meant to do that check before updating but forgot.

I'm not sure what you mean about reentrancy - you mean, could this be called multiple times with the same controller uuid? Or different ones? I guess I can see it being called more than once with the same uuid, if there was something wrong with the migration master worker.

Do you mean, if it was called in parallel with two different uuids could we end up in an inconsistent state? If the order of the resources coming back was inconsistent between two different calls I think that would cause a problem. Hmm. Or if one of them was delayed for a long time and then continued after the other one had run.

I'm not sure how to handle that - I can think of a couple of options that still seem racy, unless there's a way to make the update conditional on the value of the tag being what we expect (on the azure end rather than ours).

I don't think the migration machinery would allow this to happen (concurrent calls with different uuids) - the second migration would be prevented until the first was finished. Do you think that's good enough or should we try to handle that at this level?

(Or have I completely misunderstood the question? :)

@axw

axw Jan 27, 2017

Member

What I meant was: if this fails half way through, e.g. due to a network failure, will/can it be called again? If so, we should handle a half-processed resource group.

@babbageclunk

babbageclunk Jan 27, 2017

Member

Oh, I see! Yes, it can get restarted after crashing halfway through - the migration master will retry the phase if it errors out, so this would be called again on the half-updated group.

I think it would always handle that right (although your comment here will make it more efficient). We're finding the resources by group, based on the model id which hasn't changed. Some of the resources will have the old controller tag and some will have the new one, and they'll all be updated to the new one (barring more network errors).

Member

babbageclunk commented Jan 30, 2017

!!chittychitty!!

@babbageclunk babbageclunk changed the title from WIP migrations: implement AdoptResources for azure provider to migrations: implement AdoptResources for azure provider Jan 30, 2017

babbageclunk added some commits Jan 25, 2017

Changes from review and Azure experimentation
Experimented with updating tags in a real Azure deployment to understand
how the API behaves.
First unit test for AdoptResources
Need tests for error cases.
Member

babbageclunk commented Jan 30, 2017

!!chittychitty!!

provider/azure/environ.go
+ return errors.Trace(err)
+ }
+
+ resources, err := groupClient.ListResources(env.resourceGroup, "", nil)
@axw

axw Jan 31, 2017

Member

I know it's ugly and gross, but the API calls should be wrapped with env.callAPI to take care of rate limiting.

@babbageclunk

babbageclunk Jan 31, 2017

Member

Ah, will do - I wasn't sure what callAPI was for.

provider/azure/environ.go
+ err := env.updateResourceControllerTag(&resourceClient, resource, controllerUUID)
+ if err != nil {
+ logger.Errorf("error updating resource tags for %q: %v", *resource.Name, err)
+ failed = append(failed, *resource.Name)
@axw

axw Jan 31, 2017

Member

use to.String(resource.Name) instead of *resource.Name (in case it's nil)

@babbageclunk

babbageclunk Jan 31, 2017

Member

Thanks, I'll use these helpers.

provider/azure/environ.go
+}
+
+func (env *azureEnviron) updateResourceControllerTag(client *resources.Client, stubResource resources.GenericResource, controllerUUID string) error {
+ stubTags := *stubResource.Tags
@axw

axw Jan 31, 2017

Member

stubTags := toTags(stubResource.Tags)

provider/azure/environ.go
+ return nil
+ }
+
+ namespace, parentPath, subtype, err := splitResourceType(*stubResource.Type)
@axw

axw Jan 31, 2017

Member

to.String(stubResource.Type), etc.

provider/azure/environ.go
+
+ logger.Debugf("updating %s (%s) juju controller uuid to %s",
+ *resource.Name, *resource.Type, controllerUUID)
+ (*resource.Tags)[tags.JujuController] = &controllerUUID
@axw

axw Jan 31, 2017

Member
resourceTags := toTags(resource.Tags)
resourceTags[tags.JujuController] = controllerUUID
resource.Tags = to.StringMapPtr(resourceTags)
@babbageclunk

babbageclunk Jan 31, 2017

Member

Yeah, that's much nicer.

@babbageclunk

babbageclunk Jan 31, 2017

Member

Well, I mean it's still horrible, but better to deal with a good old map[string]string.

+func splitResourceType(resourceType string) (string, string, string, error) {
+ parts := strings.Split(resourceType, "/")
+ if len(parts) < 2 {
+ return "", "", "", errors.Errorf("expected at least 2 parts in resource type %q", resourceType)
@axw

axw Jan 31, 2017

Member

shouldn't there be at least 3? namespace, type, and name?

@babbageclunk

babbageclunk Jan 31, 2017

Member

Sometimes (fairly often) the type looks like Namespace/type, eg Microsoft.Compute/virtualMachines - in that case there's no parent path.

@axw

axw Jan 31, 2017

Member

Ah right, I was thinking we were passing in the full resource ID, but of course we're just passing in the type.

Review changes from axw
Use callAPI for rate-limiting, use to.String and friends to protect
against nil pointer dereferences.

axw approved these changes Jan 31, 2017

LGTM, thanks!

Member

babbageclunk commented Jan 31, 2017

$$merge$$

Contributor

jujubot commented Jan 31, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Jan 31, 2017

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/10153

Member

babbageclunk commented Jan 31, 2017

Spurious failure - worker/machiner/machiner_test.go:514: MachinerStateSuite.TestMachineAddresses

$$merge$$

Contributor

jujubot commented Jan 31, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

@jujubot jujubot merged commit 171cc23 into juju:2.1 Jan 31, 2017

@babbageclunk babbageclunk deleted the babbageclunk:mm-adopt-resources-azure branch Jan 31, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment