Add instance types #6564

perrito666 · 2016-11-14T22:13:51Z

Added InstanceTypes API endpoint that allows to learn
the available instance types in the model region/zone
to make better deployment decisions.

Tests are missing for 3 out of 4 providers implemented but I am keen to get the rest of the code reviewed as this needs to be merged soon.

This follows: https://docs.google.com/document/d/1m6iNWOMYyGwHbDbl8-u_VSZozeZQklxUOgM5t_0wbMQ/edit#heading=h.kc1rayquamxb

###QA
Run Tests.
Create an API client call that points to the controller and Perform the call for the different providers to obtain lists of available types. (This is intended to be used by the GUI)

perrito666 · 2016-11-15T00:27:13Z

!!build!!

perrito666 · 2016-11-15T00:29:33Z

!!try!!

perrito666 · 2016-11-15T09:59:04Z

!!try!!

axw

Mostly looking good. I think we should get input from @urosj or someone on GUI about cost/currency details.

axw · 2016-11-16T01:10:33Z

apiserver/provider/backend.go

+package provider
+
+import (
+	names "gopkg.in/juju/names.v2"


drop alias (goimports is doing this to you too?)

yes, I just realized that, mm, Ill have to check what is going on with goimports so I can disable it

axw · 2016-11-16T01:11:56Z

apiserver/provider/provider.go

+
+// NewAPI returns an API pointer.
+func NewAPI(backend Backend, authorizer facade.Authorizer, envNew environs.NewEnvironFunc) (*API, error) {
+	if !authorizer.AuthModelManager() {


The instance types API is intended for users to call.

axw · 2016-11-16T01:16:56Z

apiserver/provider/provider.go

+	}, nil
+}
+
+type modelConfigAble interface {


use environs.ConfigGetter?

Does not satisfy what we need here (lacks err)

axw · 2016-11-16T01:22:56Z

apiserver/provider/provider.go

+		return params.InstanceTypesResults{}, errors.Trace(err)
+	}
+
+	for i, c := range cons.Constraints {


Please add a TODO to cache the results. We need to avoid repeatedly querying the cloud, but cannot poll the cloud for reasons described in the spec.

axw · 2016-11-16T01:23:30Z

apiserver/provider/provider_test.go

+
+var _ = gc.Suite(&providerTypesSuite{})
+
+var over9kCPUCores uint64 = 9001


axw · 2016-11-16T01:34:02Z

provider/cloudsigma/instance_information.go

@@ -0,0 +1,16 @@
+package cloudsigma


axw · 2016-11-16T01:34:11Z

provider/dummy/instance_information.go

@@ -0,0 +1,16 @@
+package dummy


copyright
(etc.)

axw · 2016-11-16T01:34:17Z

provider/dummy/instance_information.go

+package dummy
+
+import (
+	"github.com/juju/errors"


axw · 2016-11-16T01:40:51Z

provider/ec2/instance_information.go

+		return instances.InstanceTypesWithCostMetadata{}, errors.Trace(err)
+	}
+	for i, t := range iTypes {
+		iTypes[i].Cost = t.Cost / 1000


Nope. I must have been unclear, sorry.

What I meant was to include a divisor in the result struct, not to truncate/drop precision in the cost field. i.e. the results should include:

the cost (e.g. in deci-cents)

a divisor (e.g. of 1000) to get to dollars, or whatever the unit is

the currency/unit (e.g. $USD/h)

e.g. with cost=1234, divisor=1000, currency/unit=$USD/h, the GUI could display "1.234 $USD/h".

gotcha, will do

axw · 2016-11-16T01:43:25Z

provider/gce/instance_information.go

+	}
+
+	result := make([]instances.InstanceType, len(resultUnique))
+	i := 0


for i, it := range resultUnique {

once again, map. :p

axw

LGTM, but let's get feedback from @urosj before committing to this.

axw · 2016-11-16T03:06:47Z

apiserver/params/instance_information.go

+	InstanceTypes []InstanceType `json:"instance-types,omitempty"`
+	CostUnit      string         `json:"cost-unit,omitempty"`
+	CostCurrency  string         `json:"cost-currency,omitempty"`
+	Divisor       uint64         `json:"divisor,omitempty"`


CostDivisor/cost-divisor please

axw · 2016-11-16T03:08:49Z

environs/instances/instancetype.go

+	CostCurrency string
+	// Divisor indicates a number that must be applied to InstanceType.Cost to obtain
+	// a number that is in CostUnit.
+	Divisor uint64


CostDivisor

I think we should document that setting (Cost)Divisor to zero will mean that the cost is treated as already being in specified unit, as the (cost-)divisor field will be omitted from the API response. i.e. the zero value is equivalent to 1.

axw · 2016-11-16T04:00:10Z

provider/gce/instance_information.go

@@ -0,0 +1,60 @@
+package gce


axw · 2016-11-16T04:00:17Z

provider/joyent/instance_information.go

@@ -0,0 +1,23 @@
+package joyent


axw · 2016-11-16T04:00:41Z

provider/lxd/instance_information.go

@@ -0,0 +1,15 @@
+package lxd


axw · 2016-11-16T04:02:12Z

provider/rackspace/instance_information.go

+
+var _ environs.InstanceTypesFetcher = (*environ)(nil)
+
+func (e environ) InstanceTypes(c constraints.Value) (instances.InstanceTypesWithCostMetadata, error) {


do we need this one? I think the openstack impl is sufficient?

axw · 2016-11-16T04:02:18Z

provider/vsphere/instance_information.go

@@ -0,0 +1,16 @@
+package vsphere


axw · 2016-11-16T04:02:24Z

provider/vsphere/instance_information.go

+package vsphere
+
+import (
+	"github.com/juju/errors"


axw · 2016-11-16T04:02:27Z

provider/rackspace/instance_information.go

@@ -0,0 +1,15 @@
+package rackspace


axw · 2016-11-16T04:02:31Z

provider/rackspace/instance_information.go

+package rackspace
+
+import (
+	"github.com/juju/errors"


rogpeppe · 2016-11-16T12:28:41Z

apiserver/provider/provider.go

+		costUnit := instanceTypes.CostUnit
+		costCurrency := instanceTypes.CostCurrency
+		divisor := instanceTypes.CostDivisor
+		if err != nil {


Move this next to the call to InstanceTypes ?

rogpeppe · 2016-11-16T12:29:57Z

apiserver/provider/provider.go

+
+		res[i] = params.InstanceTypesResult{
+			InstanceTypes: toParamsInstanceTypeResult(allTypes),
+			CostUnit:      costUnit,


Inline? i.e. use instanceTypes.CostUnit instead of using a temp variable.

rogpeppe · 2016-11-16T13:07:32Z

environs/interface.go

+
+// InstanceTypesFetcher is an interface that allows for instance information from
+// a provider to be obtained.
+type InstanceTypesFetcher interface {


Is it really worth having a separate interface type for this single method? Why not just make it part of Environ?

Separation of concerns

axw

What's there LGTM, but the apiserver facade changes are missing.

axw · 2016-11-20T01:10:46Z

apiserver/params/instance_information.go

+
+// ModelInstanceTypesConstraint contains a constraint applied when filtering instance types.
+type ModelInstanceTypesConstraint struct {
+	// Value, if specified, contains the constraints to filter


Can you please change the name to Constraints, and tag to "constraints", to match the field in CloudInstanceTypesConstraints? That way one is a superset of the other.

Ping. This one is not so small, please don't forget about this

axw · 2016-11-20T01:11:15Z

apiserver/params/instance_information.go

+	InstanceTypes []InstanceType `json:"instance-types,omitempty"`
+	CostUnit      string         `json:"cost-unit,omitempty"`
+	CostCurrency  string         `json:"cost-currency,omitempty"`
+	// CostDivisor Will be present only when the Cost is not expressed in CostUnit.


s/Will/will/

axw · 2016-11-20T01:13:13Z

provider/azure/instance_information.go

+	}
+	result := make([]instances.InstanceType, len(types))
+	i := 0
+	for _, iType := range types {


@perrito666 like rogpeppe, I do the same when going from map to slice. Only because it's more compact and reads better (to me).

axw · 2016-11-20T01:15:18Z

provider/gce/google/testing_test.go

+		},
+	}
+	return &machineType, nil
+


delete empty line

axw · 2016-11-20T01:15:27Z

provider/gce/instance_information.go

+)
+
+var _ environs.InstanceTypesFetcher = (*environ)(nil)
+var virtType = "kvm"


if I use a const I cannot pass its addres afterwards (instances.InstanceType.VirtType is a pointer)

axw · 2016-11-20T04:00:51Z

apiserver/cloud/instance_information.go

+	"github.com/juju/juju/environs"
+	"github.com/juju/juju/state"
+	"github.com/juju/juju/state/stateenvirons"
+	names "gopkg.in/juju/names.v2"


move me, drop alias

axw · 2016-11-20T04:00:57Z

apiserver/cloud/instance_information.go

+package cloud
+
+import (
+	"github.com/juju/errors"


axw · 2016-11-20T04:06:47Z

apiserver/cloud/instance_information.go

+// EnvironConfigGetter implements environs.EnvironConfigGetter
+// in terms of a *state.State.
+type cloudEnvironConfigGetter struct {
+	*state.State


You only need state.CloudAccessor and the GetModel function. Backend (apiserver/cloud/backend.go) already embeds state.CloudAccessor, so you just need to add the GetModel method, and you can use Backend instead of State.

You could otherwise just ignore the ModelTag, and use the existing Backend.ControllerModel method.

I think it might be simpler though if you just pass the CloudSpec (or perhaps Environ?) into the common.InstanceTypes function? i.e. have the facades pull apart the args, get the appropriate model, cloud, and region, and construct a CloudSpec from them. You may as well then get the Environ then too, and avoid passing the backend in just to get ModelConfig?

axw · 2016-11-20T04:12:43Z

apiserver/cloud/instance_information.go

+		if cons.Constraints != nil {
+			value = *cons.Constraints
+		}
+		backend := cloudEnvironConfigGetter{


Please parse the cons.CloudTag here, and check that it is the same as the controller model's cloud. We may relax this later, but it's better to be strict to avoid having to deal with sloppily written clients later.

axw · 2016-11-20T04:13:31Z

apiserver/cloud/instance_information.go

+
+// InstanceTypes returns instance type information for the cloud and region
+// in which the current model is deployed.
+func (api *CloudAPI) InstanceTypes(cons params.CloudInstanceTypesConstraints) (params.InstanceTypesResults, error) {


we need a test or two for this

axw · 2016-11-20T04:16:35Z

apiserver/cloud/instance_information.go

+		itCons := common.NewInstanceTypeConstraints(backend, value, modelTag)
+		it, err := common.InstanceTypes(itCons)
+		if err != nil {
+			return params.InstanceTypesResults{}, errors.Trace(err)


An error due to one of them failing should not cause all of them to fail.

Either assume that common.InstanceTypes never fills in the Error field, and if it returns an error, have the caller (Cloud or MachineManager facade) fill it in; or change common.InstanceTypes to not have an error result.

axw · 2016-11-20T04:18:38Z

apiserver/common/instancetypes.go

+	environs.EnvironConfigGetter
+	ModelTag() names.ModelTag
+	Close() error
+	GetModel(names.ModelTag) (*state.Model, error)


please not state.Model, use an interface

This is mirroring state, there is nothing I can do.

Don't be so defeatist ;)

Look at ControllerModel in apiserver/cloud/backend.go for an example of how to do this. Anyway, I think this will be moot if you pass the Environ in?

I realized that seconds after writing that and fixed it with ashim, and am now writing the tests

axw · 2016-11-20T04:28:38Z

apiserver/common/instancetypes.go

+// InstanceTypes returns a list of the available instance types in the provider according
+// to the passed constraints.
+func InstanceTypes(cons instanceTypeConstraints) (params.InstanceTypesResult, error) {
+	m, err := cons.backend.GetModel(cons.modelTag)


The fact that we're calling GetModel multiple times for what's always going to be the same model is a bit of a smell I think. As mentioned before, I think we should pass the CloudSpec (perhaps Environ?) in.

If this were split out, then it would be reasonable to fail the entire API call if we fail to get the CloudSpec in the case of the MachineManager facade. For the Cloud facade, it would be reasonable to fail the entire API call if we fail to get the Model; and then fail individual results if we can't get the CloudSpec (e.g. if an invalid CloudTag or region is specified).

axw

Looking better, thanks.

axw · 2016-11-22T01:09:45Z

apiserver/common/instancetypes.go

+// information.
+type InstanceTypeBackend interface {
+	environs.EnvironConfigGetter
+	Close() error


Unused, drop me. I don't think it's appropriate to hand off ownership/responsibility to InstanceTypes anyway.

axw · 2016-11-22T01:34:20Z

apiserver/common/instancetypes.go

+	Config() (*config.Config, error)
+}
+
+// NewModelEnvironConfigGetter can create


Can you please move this, modelEnvironConfigGetter, and InstanceTypeBackend to a separate file? It's only tangentially related to instance types. Also, InstanceTypeBackend should be renamed to be something to do with model/environ config.

It might be simpler just to have this as a struct of two functions? e.g.

type EnvironConfigGetterFuncs struct { ModelConfigFunc func() (*config.Config, error) CloudSpecFunc func(names.ModelTag) (environs.CloudSpec, error) } func (f EnvironConfigGetterFuncs) ModelConfig() (*config.Config, error) { return f.ModelConfigFunc() } ...

Just a thought, feel free to ignore that.

The EnvironConfigGetter interface is kinda weird. Both methods should probably be taking a ModelTag...

axw · 2016-11-22T01:36:25Z

apiserver/common/instancetypes.go

+
+// InstanceTypes returns a list of the available instance types in the provider according
+// to the passed constraints.
+func InstanceTypes(cons instanceTypeConstraints) (params.InstanceTypesResult, error) {


Why pass around an unexported type?

axw · 2016-11-22T01:37:16Z

apiserver/machinemanager/instance_information.go

+// InstanceTypes returns instance type information for the cloud and region
+// in which the current model is deployed.
+func (mm *MachineManagerAPI) InstanceTypes(cons params.ModelInstanceTypesConstraints) (params.InstanceTypesResults, error) {
+	st := mm.st.(*state.State)


Please do the same for this method as you did for the other one, and use interfaces (stateInterface).

axw · 2016-11-22T23:27:55Z

apiserver/machinemanager/instance_information_test.go

 	jujutesting.Stub

 	results map[constraints.Value]instances.InstanceTypesWithCostMetadata
 }

 func (m *mockEnviron) InstanceTypes(c constraints.Value) (instances.InstanceTypesWithCostMetadata, error) {
+	fmt.Println(c)


axw · 2016-11-22T23:29:12Z

apiserver/machinemanager/state.go

+	return m, nil
+}
+
+func (s stateShim) Cloud(cloudName string) (cloud.Cloud, error) {


these Cloud* methods aren't needed? stateShim embeds State, and State already has those methods

axw

LGTM once remaining comments are addressed (those from this review and ones from before, unless we've already talked about them.)

axw · 2016-11-23T01:30:03Z

apiserver/cloud/instance_information.go

+	"github.com/juju/juju/state/stateenvirons"
+)
+
+// EnvironConfigGetter implements environs.EnvironConfigGetter


cloudEnvironConfigGetter

axw · 2016-11-23T01:32:41Z

apiserver/cloud/instance_information.go

+		return params.InstanceTypesResults{}, errors.Trace(err)
+	}
+	modelTag := c.ModelTag()
+	m, err := api.backend.GetModel(modelTag)


you end up with the same model as returned by ControllerModel

just do m, err := api.backend.ControllerModel()

then you can get rid of the ModelTag method from Backend

axw · 2016-11-23T01:37:47Z

apiserver/common/environ_config.go

+	"github.com/juju/juju/environs/config"
+)
+
+// EnvironConfigGetterFuncs holds implements environs.EnvironConfigGetter


axw · 2016-11-23T01:39:53Z

apiserver/params/instance_information.go

+
+// ModelInstanceTypesConstraint contains a constraint applied when filtering instance types.
+type ModelInstanceTypesConstraint struct {
+	// Value, if specified, contains the constraints to filter


Ping. This one is not so small, please don't forget about this

frankban

Looks good and works well. Just a couple of minor issues. Did you talk about opening the Cloud call up to anonymous users?

frankban · 2016-11-23T12:04:44Z

apiserver/cloud/instance_information.go

+		if m.Cloud() != cloudTag.Id() {
+			result[i] = params.InstanceTypesResult{Error: common.ServerError(errors.NotValidf("asking %s cloud information to %s cloud", cloudTag.Id(), m.Cloud()))}
+			continue
+		}


I'd check also that cons.CloudRegion is not empty here. otherwise I guess the subsequent error would be ugly and not really informative. This case should also be tested.

frankban · 2016-11-23T12:13:29Z

apiserver/machinemanager/instance_information_test.go

+	) (environs.Environ, error) {
+		return &env, nil
+	}
+	r, err := machinemanager.InstanceTypes(&api, fakeEnvironGet, cons)


Should we also test the case no constraints are passed? In this case I'd expect an empty result.

axw · 2016-11-23T23:17:29Z

apiserver/cloud/instance_information.go

@@ -75,6 +75,10 @@ func instanceTypes(api *CloudAPI,
 			result[i] = params.InstanceTypesResult{Error: common.ServerError(errors.NotValidf("asking %s cloud information to %s cloud", cloudTag.Id(), m.Cloud()))}
 			continue
 		}
+		if cons.CloudRegion == "" {


Not all clouds have regions. I'm not sure if the validation belongs here or not, but if we do validate, we need to check that the cloud supports regions before requiring that it's empty.

axw · 2016-11-23T23:18:38Z

@frankban

Looks good and works well. Just a couple of minor issues. Did you talk about opening the Cloud call up to anonymous users?

Yes we have talked about it, but as we agreed we'll defer this bit. It's not feasible to make that change for 2.1.

Added InstanceTypes API endpoint that allows to learn the available instance types in the model region/zone to make better deployment decisions.

perrito666 · 2016-11-24T12:54:19Z

!!try!!

perrito666 · 2016-11-24T22:12:33Z

$$merge$$

jujubot · 2016-11-24T22:14:12Z

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

perrito666 force-pushed the add_instance_types branch from a9f76ca to 4aa7e30 Compare November 15, 2016 00:26

perrito666 force-pushed the add_instance_types branch 2 times, most recently from 2035d3d to 1d9b90c Compare November 15, 2016 00:43

axw suggested changes Nov 16, 2016

View reviewed changes

axw approved these changes Nov 16, 2016

View reviewed changes

rogpeppe reviewed Nov 16, 2016

View reviewed changes

axw reviewed Nov 20, 2016

View reviewed changes

perrito666 force-pushed the add_instance_types branch from a7aea2a to 1d46062 Compare November 20, 2016 02:43

axw suggested changes Nov 20, 2016

View reviewed changes

axw reviewed Nov 22, 2016

View reviewed changes

axw approved these changes Nov 23, 2016

View reviewed changes

frankban approved these changes Nov 23, 2016

View reviewed changes

perrito666 force-pushed the add_instance_types branch from 5ee6359 to 48377ee Compare November 23, 2016 18:54

axw reviewed Nov 23, 2016

View reviewed changes

Add InstanceTypes API endpoint.

6dcfaf0

Added InstanceTypes API endpoint that allows to learn the available instance types in the model region/zone to make better deployment decisions.

perrito666 force-pushed the add_instance_types branch from f7ce73c to 6dcfaf0 Compare November 24, 2016 12:47

jujubot merged commit acfb0ce into juju:develop Nov 24, 2016


		var _ = gc.Suite(&providerTypesSuite{})

		var over9kCPUCores uint64 = 9001


		var _ environs.InstanceTypesFetcher = (*environ)(nil)

		func (e environ) InstanceTypes(c constraints.Value) (instances.InstanceTypesWithCostMetadata, error) {

Add instance types #6564

Add instance types #6564

Conversation

perrito666 commented Nov 14, 2016

perrito666 commented Nov 15, 2016

perrito666 commented Nov 15, 2016

perrito666 commented Nov 15, 2016

axw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

axw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment