juju/juju

Extract resources apiserver parts into the apiserver package

## Description of change

This PR is part of the cleanup of the apiserver facade registration. All the resources related apiserver code has been moved to the apiserver package, include HTTP endpoints. This means that resources related APIs are now defined in the same way as all other APIs.

Some highlights:

* The Resources and ResourceHookContext facades now exist under apiserver.
* Some mime handling backport code for Go < 1.6 has been removed. We require at least Go 1.7 now.
* Client and unit HTTP endpoints for resource upload and download have been moved to the apiserver package.
* Many layers of completely unnecessary and confusing abstraction have been removed.
* Removed the now-unused apiserver plumbing to support for the component architecture. Both the payloads and resources features have been extracted to apiserver now.
* Drive-by test fixes for Go 1.8.

This PR results in a net removal of over 1700 lines of code, for no functional change!

## QA steps

```
$ juju deploy cs:~cmars/mattermost
# Confirmed that deploy works and resource downloads on to unit.

$ juju list-resources mattermost                          
[Service]
Resource  Supplied by  Revision
bdist     charmstore   3

# Now test with the dummy-resource charm from the QA team

$ cd ~/canonical/repository/charms/dummy-resource
$ juju deploy . --resource foo=foo.txt --resource bar=bar.txt
# Charm emits resource info as status. Confirmed that resource was deployed.

$ juju status | grep Path:
dummy-resource/0*  maintenance  idle   1        10.0.8.199             Path: /var/lib/juju/agents/unit-dummy-resource-0/resources/bar/bar.txt Time: 0 sec Size: 17 bytes

# Update the resource
$ juju attach dummy-resource bar=foo.txt

$ Confirm that the resource is updated
$ juju status | grep Path:
dummy-resource/0*  maintenance  idle   1        10.0.8.199             Path: /var/lib/juju/agents/unit-dummy-resource-0/resources/bar/bar.txt Time: 0 sec Size: 27 bytes
# (note the change in size)

$ juju list-resources dummy-resource
[Service]
Resource  Supplied by  Revision
foo       admin        2017-20-03T00:03
bar       admin        2017-20-03T00:13
```

## Documentation changes

No user visible effects.

## Bug reference

N.A.

Bump juju/cmd and loggo dependencies

## Description of change

This pulls in the latest loggo change which moves loggo's color
support to different package and the changes to juju/cmd which make
the "help" and "version" output consistent with other Juju commands.

## QA steps

Code and tests still compile. "juju debuglog" has colored output. "juju help commands" now has correctly formatted command summaries for all commands.

## Documentation changes

N.A.

## Bug reference

N.A.

This pulls in the latest loggo change which moves loggo's color
support to different package and the changes to juju/cmd which make
the "help" and "version" output consistent with other Juju commands.

Cmr find list cleanup

## Description of change

Please look at the just second commit as this PR builds on #7124
2c965bd

There's several pieces of cleanup:
- separate CMR list and find APIs
- list and find return different data
-- list includes charm and connection info
- new state support for querying the number of CMR connections
- fix for NPE in firewaller
- log stacktrace when a worker panics for easier diagnosis
- CMR CLI help cleanup

## QA steps

bootstrap and deploy a CMR scenario
run the various CMR commands - list, find, show
check the output

httptest.ResponseRecorder only got a Result method after Go 1.6.

Show progress for kvm containers

## Description of change

KVM containers were only showing "Starting container, this might take a while". But not giving any sense for what it was actually doing. This makes it slightly better by giving a progress message (number of bytes copied, progress %) while the container is being started, and then changing the final message to indicate the machine is running.

## QA steps

```
  $ juju bootstrap maas
  $ juju switch controller
  $ juju add-machine kvm:0
  $ juju status
  $ watch juju show-machine 0
```

Without this patch, there would only be a message:
```
        instance-id: juju-3cab08-0-kvm-1
        machine-status:
          current: allocating
          message: Creating container; it might take some time
```

With this patch, you should see messages like:
```
 copying http://cloud-images.ubuntu.com/server/releases/trusty/release-20170307/ubuntu-14.04-server-cloudimg-amd64-disk1.img 5% 1.4MB/s
```
And a final status of:
```
        machine-status:
          current: running
          message: Container started
          since: 19 Mar 2017 16:25:39+04:00
```

this matches the LXD container status.  The progress message should be pretty close, and the final status is the same.

## Documentation changes

This *is* user visible, but as it is informative, I don't think it needs documentation changes.

## Bug reference

[lp:1674074](https://bugs.launchpad.net/juju/+bug/1674074)

Switch to dustin/go-humanize instead of a bespoke BPS check.
Document the 'progress' parameter for Sync.

Fix command summaries

## Description of change

A number of client commands had help summaries that weren't consistent with the standard approach. This PR fixes all of them except the ones that come from github.com/juju/cmd (separate PR).

## QA steps

$ juju help commands | egrep '^\S+ +[a-z]'
$ juju help commands | egrep '.+[^.]$'

## Documentation changes

N.A.

## Bug reference

N.A.

Register the commands defined by the resource and payload "components"
so that the general tests for command registrations and help text check
those too.

Instead of assuming registeredCommands and registeredEnvCommands are nil
to begin with, set them in test setup (and restore their values afterwards).

We were already testing for capitalisation but not the trailing full
stop.

These were missed because they were for a "juju charm" subcommand.

This makes them consistent with all the other commands.

Due to these commands not being implemented in the standard way, the
help summaries weren't updated when the style was standardised for 2.0

This is more in line with other facades. Ultimately we would like all
facade to not deal with State directly and just expose the backend
interfaces they need.

Appears to already be covered by other tests.

resourcesMigrationUploadHandler is a longer but less fighter-jettish
name.

Put {attach,detach}-storage behind feature flag

## Description of change

Introduce the "persistent-storage" feature flag,
and put the attach-storage and detach-storage
commands behind it. The remove-storage command
is complete, and so has not been put behind the
flag. We'll remove the feature flag in develop
once 2.2 has branched.

## QA steps

1. juju detach-storage -h
`ERROR unrecognized command: juju detach-storage`
2. juju attach-storage -h
`ERROR unrecognized command: juju attach-storage`
3. export JUJU_DEV_FEATURE_FLAGS=provider-storage
4. juju detach-storage -h
```
Usage: juju detach-storage [options] <storage> [<storage> ...]
...
```
5. juju attach-storage -h
```
Usage: juju attach-storage [options] <unit> <storage> [<storage> ...]
...
```

## Documentation changes

No, these commands are not yet documented.

## Bug reference

None.

Some tests were testing for error messages which have changed in Go
1.8. The regexes have been made less specific.

The resources and payload facades are now always registered and don't
require the component mechanism to do this.

The apiserver infrastrcture to support the component architecture bits
are now unused and have been removed.

Backend does not need to be exposed so don't.

This is no longer used.

This is the last component architecture apiserver part to be
moved. Much unnecessary abstraction and dead code has been removed. The
code is now much more straightforward.

Move all this out from resource/api/server to apiserver. Much
unnecessary abstraction and stupid test technique has also been removed.