migrations: don't start model workers while a model is importing #6700

Merged
merged 3 commits into from Dec 14, 2016

Conversation

Projects
None yet
2 participants
Member

babbageclunk commented Dec 13, 2016

Fixes https://bugs.launchpad.net/juju/+bug/1646310

(Recreated for develop branch from #6678.)

Starting the migration master while the model was being imported back
into a controller that it had previously migrated out of caused it to
uninstall itself from the engine (thinking that it was finished
processing the old migration), which blocked any workers that depended
on ifNotMigrating. To fix this we wait until the model MigrationMode
has changed to MigrationModeNone - this means the watcher needs to
signal on any changes to models, rather than just life.

Add a global flag to the collectionWatcher config - this lets the watcher return
ids for any model, rather than just the current one.

The apiserver code to remove state objects from the pool still needs to
watch life specifically. If the model becomes Dead and is then
removed within the event coalescence time then the collection watcher
won't report any event, but a lifecycle watcher will.

QA steps:

  • bootstrap two controllers A and B
  • create a model m in A with an application deployed
  • migrate it to B
  • migrate it back to A
  • check the debug log has no dependency engine messages saying "fortress operation aborted" - these show the workers that are prevented from running because the migration master has exited
  • migrate it back to B - this would have failed before this fix, because the previous migration left the model workers in a bad state

babbageclunk added some commits Dec 8, 2016

Make WatchModels a collection watcher
The model worker manager was starting the model workers too early when a
model was migrated into the controller. To fix this we want it to wait
until the model MigrationMode has changed to MigrationModeNone, so the
watcher needs to signal on any changes to models, rather than just
life. Making it a collection watcher does this.

The apiserver code to remove state objects from the pool still needs to
watch life specifically, because if the model becomes Dead and is then
removed within the event coalescence time then the collection watcher
won't report any event, but a lifecycle watcher will.
Defer starting model workers when importing
Starting the migration master while the model was being imported back
into a controller that it had previously migrated out of caused it to
uninstall itself from the engine (thinking that it was finished
processing the old migration), which blocked any workers that depended
on ifNotMigrating.

Fixes https://bugs.launchpad.net/juju/+bug/1646310
Don't strip off model uuid for global watchers
If the collection it's watching is global, then the ids won't be
prefixed with the model uuid anyway - it works for the models collection
because it doesn't do anything if the prefix isn't there. If it's not a
global collection, then the model uuid will be needed to make sense of
the id that comes back.
Member

babbageclunk commented Dec 13, 2016

Doesn't need rereview - no difference between this and #6678.

Member

babbageclunk commented Dec 13, 2016

$$merge$$

Contributor

jujubot commented Dec 13, 2016

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Dec 13, 2016

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/9863

Member

babbageclunk commented Dec 13, 2016

Failed to bootstrap on lxd. ):

$$retry$$

Contributor

jujubot commented Dec 13, 2016

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Dec 14, 2016

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/9864

Member

babbageclunk commented Dec 14, 2016

Windows tests failed for something to do with Mongo:
wsarecv: An existing connection was forcibly closed by the remote host.

$$retry$$

Contributor

jujubot commented Dec 14, 2016

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

@jujubot jujubot merged commit 8a03fbc into juju:develop Dec 14, 2016

1 check passed

github-check-merge-juju Built PR, ran unit tests, and tested LXD deploy. Use !!.*!! to request another build. IE, !!build!!, !!retry!!
Details

@babbageclunk babbageclunk deleted the babbageclunk:migration-bounce branch Dec 14, 2016

Member

babbageclunk commented Dec 14, 2016

Yay!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment