/juju

Honour KVM constraints from bundles #6723

Closed
wants to merge 86 commits into
from

Conversation

Reviewers

@bz2 bz2

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
@voidspace

voidspace commented Dec 15, 2016 •
Edited 1 time

Update revision of bundlechanges. Fixes bug #1626597.

To QA
Deploy a bundle, with an application with a constraint and a KVM placement directive.
The KVM container created should honour the constraint.

natefinch and others added some commits Dec 2, 2016

@natefinch
Add Ping method to providers 
    In order to know if an endpoint is correct when adding a cloud,
    we need to be able to ping that provider to get an idea if it
    even exists.  This is long before bootstrap or even credentials,
    so we need a way to reasonably determine if a cloud of the
    correct type exists at the hostname/ip.  This is different per
    cloud, thus a per-provider endpoint.
This commit was signed with a verified signature.
@natefinch natefinch Nate Finch
GPG key ID: F5F09E8A6BAA1CCA Learn about signing commits
a185eca
@howbazaar
Refactor the peergrouper to use clock.Clock, and publish an event ove… 
…r pubsub when the collection of apiservers change.
4966a89
@howbazaar
Add a test for the publishing of the server details on the hub.
6afda6f
@natefinch
Stop using pkg/errors in juju code
This commit was signed with a verified signature.
@natefinch natefinch Nate Finch
GPG key ID: F5F09E8A6BAA1CCA Learn about signing commits
53e63fb
@howbazaar
Fix typo
632b313
@wallyworld
Implement the publish method on the remote relations facade
92ec895
@jujubot
Merge pull request #6676 from wallyworld/cmr-relation-publish 
Implement the publish method on the remote relations facade

The remote relations facade has the publish method implemented. Also adds a required GetToken() API that the worker uses.

QA:
bootstrap
juju switch controller
juju deploy mysql
juju offer mysql:db local:/u/ian/mysql
juju switch default
juju deploy wordpress
juju expose wordpress
juju relate wordpress local:/u/ian/mysql

profit
2f1c93e
@wallyworld
Delete old cmr code
c8bd437
@macgreagoir
Wait for machines on controller kill and destroy commands 
`kill-controller` and `destroy-controller` now also wait on machines yet
to be removed as well as the previous behaviour of waiting on models yet
to be destroyed. This fixes a bug in which machines in the controller
model were not given time to complete their removal before the
controller itself was destroyed.

Fixes lp:1642295
daeff6e
@jujubot
Merge pull request #6677 from wallyworld/cmr-delete-code 
Delete old cmr code

Delete old cmr code which is no longer used.
4ff3027
@jujubot
Merge pull request #6674 from macgreagoir/destroy-wait-on-machines 
Wait for machines on controller kill and destroy commands

`kill-controller` and `destroy-controller` now also wait on machines yet
to be removed as well as the previous behaviour of waiting on models yet
to be destroyed. This fixes a bug in which machines in the controller
model were not given time to complete their removal before the
controller itself was destroyed.

Fixes lp:1642295

QA steps:
 * Best seen in a manual environment, as per the bug
 * `bootstrap`, then `add-machine` to the controller model
 * `destroy-controller` and observe the destroy procedure continue until the machine in the controller model is removed, which will happen after the default model is destroyed
```
$ juju destroy-controller -y manual
Destroying controller
Waiting for hosted model resources to be reclaimed
Waiting on 0 model, 1 machine
Waiting on 0 model, 1 machine
Waiting on 0 model, 1 machine
All hosted models reclaimed, cleaning up controller machines
```
 * Repeat testing `kill-controller`
e9e1a46
@anastasiamac
Change version to 2.2.0-alpha1.
93cb93c
@howbazaar
Fix tests that patched out peergrouperNew.
3580c64
@macgreagoir
Fix race in kill suite controller machines test
c1bcc8b
@jujubot
Merge pull request #6669 from howbazaar/centralhub-4-peergrouper 
Have the peergrouper publish the apiservers over the central hub.

Also refactors the peergrouper to use clock.Clock interface rather than time directly.
b7374ef
@anastasiamac
Change version to 2.2-alpha1.
b08fea1
@jujubot
Merge pull request #6670 from natefinch/no-pkg-errors 
Stop using pkg/errors in juju code

goimports betrayed me

QA: run go test in cmd/juju/cloud
ff78c96
@jujubot
Merge pull request #6681 from macgreagoir/wait-controller-machines-race 
Fix race in kill suite controller machines test

Fixes lp:1648749

QA steps:
 * Data race tests should pass
3c9b9d5
@jujubot
Merge pull request #6680 from anastasiamac/next-version 
Change version to 2.2.0-alpha1.

What is source-tag in snapcraft.yaml? Should it be changed to 2.2... something as well?
b2f8134
@wallyworld
More tolerant call checks
6b013b0
@howbazaar
Add a test to show deletion of importing models.
b219abd
@jujubot
Merge pull request #6621 from natefinch/fix-1641970 
Add Ping method to providers

In order to know if an endpoint is correct when adding a cloud,
we need to be able to ping that provider to get an idea if it
even exists.  This is long before bootstrap or even credentials,
so we need a way to reasonably determine if a cloud of the
correct type exists at the hostname/ip.  This is different per
cloud, thus a per-provider endpoint.

This fixes https://bugs.launchpad.net/juju/+bug/1641970
164227b
@jujubot
Merge pull request #6685 from wallyworld/cmr-worker-test-fail 
More tolerant call checks

Hopefully fixes failure seen in http://juju-ci.vapour.ws:8080/job/github-merge-juju/9829/artifact/artifacts/trusty-out.log

Also drive by fmt fixes.
c3a95e5
@jujubot
Merge pull request #6688 from howbazaar/migration-restricted-apiroot 
Restrict the api calls available to users during a model's migration.

Forward port of 2.1 branch.
ba666bf
@jujubot
Merge pull request #6691 from howbazaar/block-controller-upgrade-with… 
…-active-migration

Block controller upgrade with active migration.

Block the controller from upgrading if it is in the process of importing or exporting another model.

Forward port of 2.1 commit.
0b720ae
@jujubot
Merge pull request #6689 from howbazaar/enable-delete-importing-model 
Add a test to show deletion of importing models.

It was incorrectly assumed that one would not be able to delete a model that is being imported.

Added a test to show that this can be done.
828f502
@mjs
component/all: Removed some confusing unused code
e6b4909
@mjs
component/all: Machine downloads of unit resources 
For model migrations, machine agents (specifically the migrationmaster
worker) needs to be able to download unit resources.
7c0b642
@mjs
api/migrationmaster: Client API for unit resources 
Added OpenUnitResource to support downloads of unit-level resources.
c87d515
@mjs
apiserver/migrationmaster: Report unit resources 
As well as returning application level resource revisions, Export now
returns resource revisions used by units.
d5515f1
@mjs
api/migrationmaster: Report unit revisions 
Export now reports unit resource revisions.
f0eea8a
@mjs
State functionality for setting unit resources 
Expose SetUnitResource so that unit resources can be set
directly. Previously the only way was to use OpenResourceForUniter which
only clones the application's resource to the unit.
2a91c19
@mjs
resource/resourceadapter: Use unit in URL 
Instead of using the logged in entity to determine the unit, use the
actual unit in the URL. This is required now that we have the
migratonmaster worker in machine agent downloading unit resources.
ab0e8a3
@mjs
apiserver: Support unit resource uploads 
Extended the model migration resource upload endpoint to also accept
resource uploads for individual units.
c00e25d
@mjs
api/migrationtarget: Support unit resource uploads
a2310b8
@mjs
migration: Migrate unit resources 
UploadBinaries now handles unit resources in addition to application
resources.
5d710bb
@mjs
worker/migrationmaster: Support unit resources
5d1fe64
@mjs
Remove machine access to unit resource endpoint 
It turns out there's no need to pull unit resource binaries.
83580f5
@mjs
state: Remove unneeded unit resource setting code 
... because it's not necessary to supply unit resource binaries.
ea3e146
@mjs
api/migrationmaster: Remove OpenUnitResource 
There's no need to download per-unit resource binaries.
e3983e1
@mjs
apiserver: No content for unit resources 
There's no need to upload the content for a unit resource. Only the
metadata is needed.
199000e
@mjs
api/migrationtarget: UploadUnitResource -> Set
6683e73
@mjs
migration: UploadUnitResource -> Set
fca1231
@mjs
worker/migrationmaster: UploadUnitResource -> Set
055ac2b
@mjs
gofmt fix
ca398dd
@jujubot
Merge pull request #6693 from mjs/MM-unit-resources-import-develop 
Support migration of unit resources

(NOTE: this was already reviewed when it merged into the 2.1 branch)

Units may have different resource revisions to the application and as such need to be migrated separately.

This PR adds API support for unit resources and updates the binaries migration functionality to support them.

### QA

- Migrate a resource using model and confirm the resources collection matches up such that the unit resource is recorded. Also checked gridfs and blobstore metadata.
- Migrate a resource using model where the application resource is different to a unit resource and inspect the DB is correct post-migration. Run add-unit on the application.
002c93a
@frankban
Ensure the GUI uncompressed dir can be safely renamed.
614a161
@jujubot
Merge pull request #6694 from frankban/safe-rename 
Ensure the GUI uncompressed dir can be safely renamed.

The src and target directories must be in the same file system.
This fixes juju/juju-gui#2288
3f9f3d0
@anastasiamac
Fixes lp#1597830: worker should not restart agent. 
Xenial machines with units would hang when trying to convert to state servers under HA after new revisions of systemd and dbus were introduced.

It was discovered that the conv2state worker would explicitly restart an agent. This proposal changes the behavior to throw an error instead to ensure that proper infrastructure restarts the agent cleanly.
a3311a4
@jujubot
Merge pull request #6696 from anastasiamac/develop-con2state-lp1597830 
Fixes lp#1597830 on develop: worker should not restart agent.

Xenial machines with units would hang when trying to convert to state servers under HA after new revisions of systemd and dbus were introduced.

It was discovered that the conv2state worker would explicitly restart an agent. This proposal changes the behavior to throw an error instead to ensure that proper infrastructure restarts the agent cleanly.
6fc039c
@babbageclunk
Make WatchModels a collection watcher 
The model worker manager was starting the model workers too early when a
model was migrated into the controller. To fix this we want it to wait
until the model MigrationMode has changed to MigrationModeNone, so the
watcher needs to signal on any changes to models, rather than just
life. Making it a collection watcher does this.

The apiserver code to remove state objects from the pool still needs to
watch life specifically, because if the model becomes Dead and is then
removed within the event coalescence time then the collection watcher
won't report any event, but a lifecycle watcher will.
fa9c272
@babbageclunk
Defer starting model workers when importing 
Starting the migration master while the model was being imported back
into a controller that it had previously migrated out of caused it to
uninstall itself from the engine (thinking that it was finished
processing the old migration), which blocked any workers that depended
on ifNotMigrating.

Fixes https://bugs.launchpad.net/juju/+bug/1646310
c2fe6e0
@babbageclunk
Don't strip off model uuid for global watchers 
If the collection it's watching is global, then the ids won't be
prefixed with the model uuid anyway - it works for the models collection
because it doesn't do anything if the prefix isn't there. If it's not a
global collection, then the model uuid will be needed to make sense of
the id that comes back.
a100fba
@howbazaar
Add a manifold to provide the central hub as a worker resource.
2abe574
@jujubot
Merge pull request #6700 from babbageclunk/migration-bounce 
migrations: don't start model workers while a model is importing

Fixes https://bugs.launchpad.net/juju/+bug/1646310

(Recreated for develop branch from #6678.)

Starting the migration master while the model was being imported back
into a controller that it had previously migrated out of caused it to
uninstall itself from the engine (thinking that it was finished
processing the old migration), which blocked any workers that depended
on `ifNotMigrating`. To fix this we wait until the model `MigrationMode`
has changed to `MigrationModeNone` - this means the watcher needs to
signal on any changes to models, rather than just life.

Add a global flag to the `collectionWatcher` config - this lets the watcher return
ids for any model, rather than just the current one.

The apiserver code to remove state objects from the pool still needs to
watch life specifically. If the model becomes `Dead` and is then
removed within the event coalescence time then the collection watcher
won't report any event, but a lifecycle watcher will.

QA steps:
* bootstrap two controllers A and B
* create a model m in A with an application deployed
* migrate it to B
* migrate it back to A
* check the debug log has no dependency engine messages saying "fortress operation aborted" - these show the workers that are prevented from running because the migration master has exited
* migrate it back to B - this would have failed before this fix, because the previous migration left the model workers in a bad state
8a03fbc
@jujubot
Merge pull request #6701 from howbazaar/centralhub-4.1-manifold 
Add a manifold to provide the central hub as a worker resource.

This manifold can provide the hub to other workers.

When the apiserver and the peergrouper use the dependency engine, the centralhub worker should create the central hub, but for now, we need to create it in the machine agent and pass it in through the manifold config.

## QA

juju bootstrap lxd test
juju debug-log -m controller --replay | less

confirm that the "central-hub" worker has started and not stopped.
c4c0422
@babbageclunk
Ensure resource HTTP handlers release state 
Thread the call to httpContext.release through so it can be called from
the right place in the resource HTTP handler.
28dcd96
@babbageclunk
Close state leaks from the NewResourceOpener code 
This code path is a bit trickier to close than the other handler.
0bb1a74
@mjs
apiserver: Uploads of placeholder resources 
Placeholder resources are those which haven't been downloaded by any
unit yet. The timestamp field is used to determine whether a resource is
a placeholder.
773eff3
@mjs
api/migrationtarget: Placeholder resource support 
Added a new SetPlaceholderResource method which calls the
/migrate/resource endpoint with the parameters set for a placeholder
resource.

Also cleaned up tested for the resource upload endpoints.
7de51fa
@mjs
migration: Support placeholder resources 
Placeholder resources now use the specific client side API for handling
them.
f288d2b
@mjs
worker/migrationmaster: Add SetPlaceholderResource 
... to uploadWrapper. The ResourceUploader interface was extended to
include this.
67ca29c
@babbageclunk
Fix test broken by closer change
849be37
@jujubot
Merge pull request #6703 from babbageclunk/resource-state-leak 
Ensure resource HTTP handlers release state

The resource HTTP handler didn't release the state back into the state 
pool - this would mean that it would stick around in the pool even after 
the model was migrated away, preventing the same model from being 
migrated back. (It would also leak the state if the model was destroyed.)

Thread the call to httpContext.release through so it can be called from
the right place in the resource HTTP handler.

This is the last piece of fixing https://bugs.launchpad.net/juju/+bug/1641824

QA steps:
* bootstrap 2 controllers A and B
* create model m in A
* download the Mattermost binary from https://releases.mattermost.com/3.5.1/mattermost-team-3.5.1-linux-amd64.tar.gz
* deploy Mattermost using that binary distribution
```juju deploy -m A:m cs:~cmars/mattermost --resource bdist=<path to binary distribution>```
* migrate the model to b: `juju migrate -c A m B`
* then migrate it back: `juju migrate -c B m A`
* the migration should be successful
fe83e98
@wallyworld
Fix application id hack in remote relations worker
10b7cdd
@jujubot
Merge pull request #6706 from mjs/MM-placeholder-resources-develop 
migrations: Support placeholder resources

Placeholder resources are those which are defined by a charm but have not yet been requested by any unit and are therefore not stored in the controller yet. They need special handling because they have no content and can't be opened by OpenResource.

This is a forward port of #6692.
e1bb371
@jujubot
Merge pull request #6707 from wallyworld/cmr-remove-appid-hack 
Fix application id hack in remote relations worker

Fixes a TODO in remote relations worker. The RegisterRelation call now returns the exported application id of the offered application, which is imported into the consuming model. The remote relations facade needed to gain the ImportRemoteEntity API.

The worker in the consuming model was also incorrectly exporting the remote application, not the local one; the worker has been restructured to properly register the remote relation at the right time in the lifecycle.

Finally, the export mutex is no longer needed so was deleted.

QA: bootstrap and setup a relation between wordpress and mysql in different models
9467495
@mjs
state: Expose listing of pending resources 
This is required for a migrations precheck.
d83c429
@mjs
migration: Block migration if resources pending 
A resource is pending if it is in the process of being added or
downloaded (i.e. immediately after deploy or being downloaded by a
unit). Migrations shouldn't proceed if this is the case.
d18b5e4
@mjs
Fixed missed service -> application rename
3061156
@mjs
cmd/jujud/agent: Registers components in tests 
Required to support migrationmaster worker.
f5948ba
@jujubot
Merge pull request #6714 from mjs/MM-pending-resources-develop 
Migration precheck for pending resources

Don't allow a migration to proceed if a resource is pending. A resource is pending if it is in the process of being added or downloaded (i.e. immediately after deploy or is being downloaded). The pending state is typically short-lived.

This is a forward port of #6695.
0963d5b
@mjs
state: Add missing tests for resource cleanup 
There were no tests around resource blob removal. Also added some test
helpers around the NeedCleanup tests.
b43678f
@mjs
Ignore attempts to clean up a placeholder resource 
These cleanups shouldn't get scheduled in the first place but this acts
as a backstop. See https://bugs.launchpad.net/juju/+bug/1649179
31115a3
@mjs
state: Don't schedule placeholder resource cleanup 
Placeholder resources aren't stored in the blobstore and therefore don't
need cleaning up.

Fixes https://bugs.launchpad.net/juju/+bug/1649179
c61e7cc
@mjs
state: Clearer structure for IsBlobStored
b0f40a0
@jujubot
Merge pull request #6716 from mjs/1649179-placeholder-resource-cleanu… 
…p-develop

Don't schedule placeholder resource cleanup

Placeholder resources aren't stored in the blobstore and therefore don't need cleaning up. The fix has been made in two ways:

1. Cleanups are no longer scheduled for placeholder resources.
2. The resource cleanup code now ignores cleanups without a storage path set. This is a useful backstop but also fixes stray cleanups in existing deployments.

Fixes https://bugs.launchpad.net/juju/+bug/1649179

There was no testing at all for resource cleanup so this has also been rectified.

This is a forward port #6702.
f1c20c1
@jameinel
machine.AllSpaces() LinkLayerDevicesForSpaces() 
These two functions give us the context for knowing what spaces
a container is going to need, and what host devices we will want
to associate with.
e084ac5
@jujubot
Merge pull request #6690 from jameinel/machine-all-spaces 
Track Machine Spaces

This provides 2 key functions on Machine that I think will be necessary for our work on changing how containers are getting network devices on their host machine's bridges.

Machine.AllSpaces() combines the spaces that are specified in constraints with the endpoint bindings that are part of the active primary units on the machine.

Machine.LinkLayerDevicesForSpaces() filter's the link layer devices we know about for the machine, and sorts them into what space they are associated with. Between the two functions we are able to map the desired spaces for a new container onto the host's devices. This both lets us know if we have a bridge to use, and lets us create network devices only for the right ones.
269ce39
@mjs
migration: Check source controller version 
As well as checking the source model version against the target
controller version, the source controller verison needs to be checked.

A migration is allowed as long as the major.minor of the target
controller is greater than or equal to the major.minor of the source
controller. This offers the possibility of migration back to the
original controller as long as the versions are close
together (i.e. have compatible migration APIs).
004d104
@mjs
Source controller version migration prechecks 
Make the source controller version available to migration prechecks
running on the target controller.
caf92c4
@jujubot
Merge pull request #6719 from mjs/MM-controller-version-precheck-develop 
migration: Controller version precheck

As well as checking the source model version against the target controller version, the source controller verison needs to be checked.

A migration is allowed as long as the major.minor of the target controller is greater than or equal to the major.minor of the source controller. This offers the possibility of migration back to the original controller as long as the versions are close together (i.e. have compatible migration APIs).

Forward port of #6705.
9afa12f
@bz2

bz2 approved these changes Dec 15, 2016
View changes

Dep change is correct.

@voidspace voidspace closed this Dec 15, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment