Rebase master #3788

Closed

voidspace wants to merge 511 commits into juju:maas-spaces from voidspace:maas-spaces-rebase-1

voidspace commented Nov 20, 2015

(Review request: http://reviews.vapour.ws/r/3197/)

ericsnowcurrently and others added some commits Oct 5, 2015


      Add the remote to the client config.

05171df


      Do not bother trying to chmod the socket file.

7b16946


      Add an important TODO.

e71a4f4


      Ignore errors.NotImplemented in the firewaller, for now.

20febff


      Add a TODO comment.

d92c226


      Properly set up a dummy firewaller implementation.

35c6d69


      Stop the instance before destroying it.

1614aa2


      Factor out fixSockfile.

de8de2a


      Fixes after rebasing.

4fcb394


      lxd_client -> lxdclient.

ef49a61


      Add a Config field to environs.StartInstanceResult.

56fc31e


      Try the env config from the StartInstance result before falling back …

…to env.Config().

cec41e7


      The userdata is no longer compressed by default.

05fc037


      Stub out the code to take care of an implicit remote.

e8bb0e0


      Factor out bootstrap-related code.

51a143a


      Add environs.BootstrapResult.

a9234ca


      Condense the signature of Environ.Bootstrap.

08c5a2a


      Fix environ.Bootstrap.

50e24b3


      Only propagate the "unknown" attrs from result.Config.

95dbc4f


      Factory out SupportedArchitectures into a separate interface.

d773846


      Fix default handling for "namespace".

625ea0f


      Fix the tests.

0f9b285


      Fix the "raw client" tests.

ec33116


      added client_key, client_cert to lxd config

37f5e75


      Add cert handling.

5e86de1


      Add remote handling.

e53dad9


      Add remotes to Config.

89879af


      Drop ParsePEM.

8d54d57


      Adjust to changes in lxdclient.

5362fd8


      Always apply the config before connecting.

ea1c92e


      Partially fold RemoteInfo into Remote.


      remote -> remote-url.

9ecc20a


      Do not expose the lxd container type to environments.

5e06ce2


      Adding build tag for lxd provider

753890b


      Added build tag for container factory

ab32450


      build tags on stuff

391f0a1


      fix a weird compiler issue and move lxd container tests to go1.3 file

6e5f503


      Add build constraints to more files.

fd69d87


      Mark client-cert/key as immutable.

c532de5


      Add tests.

1857d2d


      Update the config based on the client config.

f8e76c4


      Fix order of operations in SetDefaults.

32d861a


      Add the remote to the LXC config file.

ae124cd


      Properly initialize the config dir.

ee7d842


      Update the remote even when it is already set.

81e62d1


      Always use a temp dir for LXD configs during tests.

22572aa


      Add "remote" field to lxdclient.Config.

0fb6a6b


      Use the client's remote as the image remote.

d006bc1


      Only call exposeHostAPI if using the Local remote.

bfeb773


      Fix up a TODO.

c4d1167


      Drop auth*File (and change config*file -> config*File).

aca831f


      Always save the config.

f8ed4e9


      Certificate -> Cert.

9743afe


      Drop GenerateCert.

c23702d


      Clarify why we call LoadConfig/SaveConfig.

7abf69d


      Drop some dead code.

61f5382


      Add test skips for unimplemented tests.

00cfdac


      Add some explanatory comments.

67ba537


      Drop Config.Apply.

8a5d716


      Drop some dead code.

b4d914c

Martin Packman and others added some commits Nov 18, 2015


      Drop table based tests and use several cases

8b16c5b


      state: remove unit assignment docs after assigning

We were removing the docs in one case, but not the
other. Crucially, we weren't removing them in the
typical case where the unit is assigned to a new
machine.

Fixes https://bugs.launchpad.net/juju-core/+bug/1516144

1265e93


      Merge pull request #3768 from axw/lp1516144-assignunit-remove

state: remove unit assignment docs after assigning

We were removing the docs in one case, but not the
other. Crucially, we weren't removing them in the
typical case where the unit is assigned to a new
machine.

The tests are still woefully inadequate, but at least
they're non-zero and cover the linked bug.

Fixes https://bugs.launchpad.net/juju-core/+bug/1516144

(Review request: http://reviews.vapour.ws/r/3176/)

283d21b


      Merge pull request #3751 from dimitern/master-lp-1499426-panic-space-…

…no-subnets

Fixed lp:1499426 - spaces with no subnets not allowed for deployments

Straight forward port of #3749 from 1.25 to master, no other changes.

Provisioner no longer accepts spaces with no subnets as a deployment
target. See bug http://pad.lv/1499426 for details.

Added a unit test and also live tested on EC2 trying to deploy a service
or add units to one, when spaces constraints are present and include a
known space without subnets.

(Review request: http://reviews.vapour.ws/r/3158/)

badcb76


      Merge pull request #3748 from dimitern/ec2-use-subnets-from-constrain…

…ts-master

ec2: Subnets are now used when specified in constraints

Forward port of #3319 from 1.25 to master.

With this PR, the EC2 provider respects spaces constraints when
specified and provisions instances inside subnets of the space, rather
than all available AZs. Instance distribution (per service or for all
instances) is still balanced among the subnets of the space, the same
way as instances are distributed among AZs (without spaces constraints).

Live tested on EC2, by bootstrapping and then adding spaces and subnets
to them using the AWS shared account and eu-central-1 region:

```bash
for i in default dmz apps backend; do juju space create $i; done
for i in 172.31.0.0/20 172.31.16.0/20; do juju subnet add $i default; done
for i in 172.31.50.0/24 172.31.51.0/24; do juju subnet add $i dmz; done
for i in 172.31.100.0/24 172.31.110.0/24; do juju subnet add $i apps; done
for i in 172.31.200.0/24 172.31.210.0/24; do juju subnet add $i backend; done
```

Then deployed a few services using:

```bash
juju deploy haproxy --constraints spaces=dmz
juju add-unit haproxy
juju deploy mediawiki -n 2 --constraints spaces=apps
juju deploy memcached -n 2 --constraints spaces=apps
juju deploy mysql --constraints spaces=backend
juju add-relation mediawiki memcached
juju add-relation haproxy mediawiki
juju add-relation mediawiki:db mysql:db
juju expose haproxy
```

All instances came up on the right subnets, verified by their private
addresses seen in AWS web console. Finally, tested bundle-based
deployments with spaces constraints work:

```bash
juju deploy wordpress-spaces-bundle.yaml
```

Original bundle was https://jujucharms.com/u/jorge/wordpress/5 and the
only needed changes where adding "constraints: spaces=apps" for
wordpress, "constraints: spaces=backend" for mysql, and renaming the
mysql service to wp-mysql in order not to clash with the other one
deployed earlier.

(Review request: http://reviews.vapour.ws/r/3155/)

4df0698


      Merge pull request #3746 from dimitern/lp-1483879-maas-devices-master

Fixed lp:1483879: Use MAAS 1.8+ devices for containers by default

Forward port of #3730 from 1.25 to master. The following is the original
PR's description.

This PR changes the default behavior around starting and stopping both
LXC containers and KVM instances, regardless whether the feature flag
"address-allocation" is enabled or not. It aims to resolve a few corner
cases where container resources (IP addresses or DHCP leases) can leak
over time, especially if the same underlying MAAS substrate is being
reused over and over again (i.e. CI/CD automated tests, OIL, etc.).

The cases are:
$ juju destroy-environment --force
$ juju destroy-machine # --force
$ juju destroy-machine #

See the related LP bug http://pad.lv/1483879

With this patch, before any container is started, Juju will now make a
best effort to register the container as a MAAS 1.8+ device with a pre-
generated MAC address, and then explicitly claim a sticky IP address
for it. In case older MAAS version is used or a non-MAAS provider in
general, the process goes on as expected. Also, if using the experimental,
feature-flagged addressable containers, the existing behavior is preserved
(allocating a static IP, tracking its lifecycle, releasing it by the
addresser worker).

Manually tested on MAAS 1.9.0rc1 in the following combinations:
1. precise/trusty/vivid/wily MAAS node hosting one or more LXC/KVM
containers with matching series (without the feature flag enabled).
2. same as 1., but with the address-allocation feature flag on.
3. same as 1., but with MAAS 1.8.3 (stable).
4. same as 3., but with the feature flag on.

In all cases tested so far, it was verified all containers get addresses
as expected, and without the feature flag, there are matching devices
created on MAAS, and the latter are removed as the containers are stopped.

(Review request: http://reviews.vapour.ws/r/3153/)

754b146


      Merge pull request #3692 from voidspace/pick-provider-first-master

Change algorithm for picking addresses to always prefer provider

When choosing a preferred public or private address for a machine always prefer the provider over machine addresses, even if the scope doesn't exactly match. This is because the provider usually will have a better idea what addresses should be used for a machine and the machine addresses should only be used as a fallback.

(Review request: http://reviews.vapour.ws/r/3097/)

7d70037


      Merge pull request #3760 from juju/lxd-provider

Merge LXD Provider into Master



(Review request: http://reviews.vapour.ws/r/3168/)

de417eb


      Merge pull request #3363 from mattyw/keep-for-24-hours

state/metrics: Metrics should be deleted 24 hours after they are sent



(Review request: http://reviews.vapour.ws/r/2746/)

95c4832


      Merge pull request #3720 from tasdomas/d001-deploy-plan-auth

Plan authorization when deploying charms.



(Review request: http://reviews.vapour.ws/r/3126/)

d52a4b3


      Merge pull request #3732 from fwereade/dependency-self-deadlock

address flaky reports test; fix self-manifold deadlock



(Review request: http://reviews.vapour.ws/r/3138/)

f45e69a


      Merge pull request #3762 from bz2/cinder_endpoints

Use cinder endpoint as given to openstack provider

Fixes lp:1512399 "environment destruction failed: ... record overflow".

Picks up fix from goose to how cinder api requests were constructed,
which was causing https requests to be sent to a http cinder endpoint.

Because the previous code also hardcoded '/v2/' in the urls being used,
logic is added in the openstack provider to prefer a v2 endpoint
where available.

As keystone does not provide a standard mechanism to specify the
version of an endpoint, this is somewhat fragile. Our Openstack
charms use a different type in the catalog for v2 api cinder, but
not all deployments will follow this scheme.

(Review request: http://reviews.vapour.ws/r/3170/)

d7092bd


      utils: switch to yaml.v2

Update tests expecting the previous error output.

14132c7


      all: remove juju/utils.go yaml fork, revert to juju/utils repo

Removes the last file from juju/juju/utils, so remove that package.

d0cb1b7


      Merge pull request #3724 from davecheney/upgrade-yaml-v2

all: upgrade to yaml.v2

- utils: switch to yaml.v2
  Update tests expecting the previous error output.
- all: remove juju/utils.go yaml fork, revert to juju/utils repo
  Removes the last file from juju/juju/utils, so remove that package.


(Review request: http://reviews.vapour.ws/r/3130/)

9d8b143


      Merge pull request #3743 from mjs/api-safe-loggedin

api: Ensure safe access to state.loggedIn

The loggedIn field on api.state was being accessed in a potentially un-goroutine-safe way.

(Review request: http://reviews.vapour.ws/r/3150/)

ab509f7


      cmd/jujud/agent/unit: removed APIInfo gate

This gate is no longer in use. The problem it was dealing with is now
handled in another way.

9632e71


      Merge pull request #3717 from mjs/dep-eng-logging

Increase the dependency engine logging level in 2 places

This change increases the dependency engine's log level when manifolds have started and when they have stopped (or failed to start). Tweaking these 2 sites provides valuable information about manifold execution, especially as agents are starting, without contributing excessive log spam.

(Review request: http://reviews.vapour.ws/r/3123/)

2e4de3d


      Merge pull request #3766 from davecheney/fixedbugs/1516498

api/unitassigner: fix data race in test

Fixes LP 1516498

(Review request: http://reviews.vapour.ws/r/3174/)

8bd466d


      Merge pull request #3741 from mjs/remove-apiinfo-gate

cmd/jujud/agent/unit: removed APIInfo gate

This gate is no longer in use. The problem it was dealing with is now handled in another way.

(Review request: http://reviews.vapour.ws/r/3148/)

d776dea


      provider/azure: Azure Resource Manager provider

Implement a new Azure provider, using the newer
Azure Resource Manager APIs. This is a completely
new provider for Azure, but we retain support for
existing Azure environments by keeping the old
("legacy") provider and switching based on user
configuration.

6ef3675


      Update to new Bootstrap signature

80c161f


      Merge pull request #3742 from axw/peergrouper-fixes

worker/peergrouper: fix race

The peergrouper was using a racey method of
waiting for a timer:

   t := time.NewTimer(0)
   t.Stop()
   select {
   ...
   case <-t.C:
   }

The problem with this is that the timer can
trigger before the Stop method is reached.
Once this happens, the select will proceed
even if Stop() is called. The tests assume
that the timer event does not trigger.

Additional changes:
 - stop using globals for injected errors,
   attach to the mock State instead
 - delete the "external" tests, which aren't
   external at all and add zero additional
   coverage

(Review request: http://reviews.vapour.ws/r/3149/)

9dd7d7c


      Merge pull request #3772 from axw/azure-arm-merge-master

provider/azure: Azure Resource Manager provider

Implement a new Azure provider, using the newer Azure Resource Manager
APIs. This is a completely new provider for Azure, but we retain support
for existing Azure environments by keeping the old
("legacy") provider and switching based on user configuration.

The code here has been reviewed in a feature branch.

(Review request: http://reviews.vapour.ws/r/3180/)

fbd3035


      worker/rsyslog: use net.JoinHostPort

fmt.Sprintf("%s:%d", ...) doesn't do the right thing for IPv6
addresses.

9c792a1


      Merge pull request #3776 from mjs/rsyslog-address-joining-master

worker/rsyslog: use net.JoinHostPort

(forward port)

fmt.Sprintf("%s:%d", ...) doesn't do the right thing for IPv6 addresses.



(Review request: http://reviews.vapour.ws/r/3184/)

4acbd26


      Merge pull request #3731 from frobware/lp-1516036

maas: change testing.invalid DNS name to have leading "-"

Certain ISP resolvers positively resolve "testing.invalid" but
".invalid" should not resolve at all, and it doesn't with 8.8.8.8. The
net result is that in some environments the unit tests will fail and in
others they will pass. This is fixed by changing the name to have a
leading "-" (i.e., -testing.invalid).

The tests pass on CI so they must be using an upstream resolver that
fails to resolve "testing.invalid".

Fixes [LP:#1516036](https://bugs.launchpad.net/juju-core/+bug/1516036)

(Review request: http://reviews.vapour.ws/r/3137/)

8a3d71c


      all: use charm.URL rather than charm.Reference

29ca55b


      Merge pull request #3745 from rogpeppe/068-remove-charm.Reference

all: use charm.URL rather than charm.Reference

This is necessary to use the latest version of the charm package,
which removes the charm.Reference type.

Also update other dependencies to later versions.


(Review request: http://reviews.vapour.ws/r/3152/)

b5cda73


      Merge pull request #3778 from natefinch/fix-1382556-mark2

Fix 1382556 mark2

This is a copy of http://reviews.vapour.ws/r/3173/ with an added test to ensure that start is being called serially and wait is being called concurrently.

In addition, make the cancel channel for StartCommandOnMachine into a more generic chan struct{}.

(Review request: http://reviews.vapour.ws/r/3186/)
Conflicts:
	apiserver/client/export_test.go
	utils/ssh/run.go

1e9ad6f


      update deps to point to modifications in juju/utils

d592108


      update deps to point to merged fp of ssh code

7124ea4


      Use the latest util & testing repos and adjust to a changed signature…

… in utils/fslock.

Conflicts:
	dependencies.tsv

b1854f0


      update to latest juju/utils

97012aa


      update tests for new lock signature

f1012f1


      network: remove racy InitaliseFromConfig

The package singleton network.globalPreferIPV6 is very racy.

Replace this with a simple atomic variable and a getter and setter.
Also remove the unnecessary PreferIPV6Getter interfaces.

60350f6


      Merge pull request #3783 from natefinch/fix-fslock

Fix fslock in core

there were incompatible changes to juju/utils without corresponding fixes to juju/juju ... this fixes that and updates to the latest of juju/testing as well.

(Review request: http://reviews.vapour.ws/r/3192/)

18bf2c6


      Merge pull request #3785 from davecheney/fix-global-ipv6-race

network: remove racy InitaliseFromConfig

Fixes LP 1517744

The package singleton network.globalPreferIPV6 is very racy.

Replace this with a simple atomic variable and a getter and setter.
Also remove the unnecessary PreferIPV6Getter interfaces.

(Review request: http://reviews.vapour.ws/r/3194/)

d1ec2da


      Merge pull request #3782 from natefinch/fix-1382556-mark2-fp

Fix 1382556 mark2 fp

Fix for http://pad.lv/1382556

We make the fork/exec of the ssh processes serially, but wait for the responses in parallel, so we'll never more than double our RAM burden on the system at any one time.

This code relies on juju/utils#176 which copies over the ssh changes (but the ssh code got moved to the other repo).

(Review request: http://reviews.vapour.ws/r/3191/)

d690b20


      Implement SupportsSpaces for maas 1.9

b1b4829


      Test

a1ec9ee


      Extended test

22023ee


      Rename capability

e29578a


      Add server url back

18b0a16


      Properly check result of calling SupportsSpaces

6fff5c4


      Remove unneeded logging

6f39cbd


      Implemented spaces constraints and initial interface bindings handlin…

…g for node selection

383654d


      Changes after review

df03fb8


      Fixed imports ordering

80a4edc

voidspace commented Nov 20, 2015

$$merge$$

Contributor

jujubot commented Nov 20, 2015

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Nov 20, 2015

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/5525

voidspace commented Nov 20, 2015

$$merge$$

Contributor

jujubot commented Nov 20, 2015

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Nov 20, 2015

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/5526

Contributor

bz2 commented Nov 21, 2015

$$badrecordMAC$$

This looks like a merge of master into the feature branch, not a rebase at all? Guessing it's the correct thing though.

Contributor

jujubot commented Nov 21, 2015

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Nov 21, 2015

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/5527

voidspace closed this Nov 23, 2015

voidspace deleted the voidspace:maas-spaces-rebase-1 branch Oct 6, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

You can't perform that action at this time.