2.3 direct models 1732384 #8128

Merged
merged 51 commits into from Nov 27, 2017

Conversation

Projects
None yet
3 participants
Owner

jameinel commented Nov 26, 2017

Description of change

juju models was implemented very inefficiently. It spent a lot of time re-reading the same tables multiple times, and then all other tables it ended up reading one object at a time, rather than doing large scans over the various tables. We also were returning information that wasn't particularly useful (we computed all the information we knew about every model, only to show a very small summary.)

While we have fixed a few of the issues around missing indexes in 2.3, this takes it all a step further and makes it much more efficient overall.

QA steps

$ juju bootstrap
# Add 200 models
$ for k in 0 1; do for j in `seq 0 9`; do for i in `seq 0 9`; juju add-model m$k$j$i & done; time wait; done; done
$ time juju models

In juju 2.2 we also had things like instantiating State objects rather than using the state pool, missing indexes, etc. However, with 200 models, the differences are:

1.11 +/- 0.18 juju-2.2
0.93 +/- 0.10 juju-2.3
0.25 +/- 0.05 juju-2.3-proposed

If I do the sampling at the same time on all models, to help show what happens under load, the numbers change to:

2.56 +/- 0.36 juju-2.2 (x2.3)
1.91 +/- 0.23 juju-2.3 (x2.05)
0.46 +/- 0.15 juju-2.3-proposed (x1.84)

Note also that 2.2 always reads all machines for every model (including their status and hardware characteristics), while the proposed 2.3 only computes the summary. So I would expect even better long term scaling on real models.

Documentation changes

I don't think this needs documentation changes. It does change the format of juju models --format=yaml but we believe it is not something that users depended upon. The information is still available in juju show-model, we just don't always compute all the details across all models.

Bug reference

lp:1732353
lp:1732384

jameinel and others added some commits Nov 15, 2017

Try inlining all the model access checks.
Rather than using a bunch of abstractions that make us hit the same
database multiple times, just set up the queries with the information we
want to have. This avoids all the StatePool stuff, and makes it all
significantly faster.
Work in progress.
Start trying to pull all the stuff that is layered behind 20-different
abstractions to get all the way back to its-just-strings-in-a-database.
Instead of loading everything one-by-one and doing 100s of round trips
to the database, just pull it in a single query and update our data
structure.

Things are mostly progressing.
Try to match client up with new controller.
This should let us at least have a snapshot of what might be working in
the future.
Add model users and status.
This had been the really expensive part that we were computing.
Hopefully this shows a big win so it is worth the time spent.
Refactor Permissions to be batch.
Also start tracking LastConnection time.
Added equivalent coverage to listmodelswithinfo based on current list…
…models coverage - beginning of full on unit tests.

Added ModelStatus to facades interfaces.
Added new state methods to test mock.
Move code into modeldetails.go
Start adding direct tests for interesting code.
We're starting with the modelQueryForUser testing,
to make sure that SuperUser gets access to everything,
but the other users only see models they are added to.
Change of focus.
We'll continue to use ModelInfo because it was present if someone used
  juju models --format=yaml

However, we'll now allow the client to say "I'm not going to do anything
with that information, so don't bother computing it", and we can update
'juju models' to know whether it is doing tabular or details output.
Actually implement some of the internals of machine info.
This might actually look like the final API? Maybe we have
to expose the Machine count information when we're suppressing
the actual machine details.
Break a bunch of stuff trying to get the shape right.
We can drop the User and Machine map from the State objects.
This currently breaks ModelManager apiserver code, but it
gets us closer.
Merge remote-tracking branch 'jameinel/2.3-direct-models-1732384' int…
…o apiserver-params-adjustment

# Conflicts:
#	state/modeluser.go
Merge remote-tracking branch 'jameinel/2.3-direct-models-1732384' int…
…o apiserver-params-adjustment

# Conflicts:
#	state/modeluser.go
Rename Details to Summary.
This gets us to line up better with what we're calling it in the API, which
hopefully allows us to line everything up. (Subject to final confirmation that
this is the route we want to go.)
Update tests around more attributes.
Check that we set the right LastConnection and Access information.
Tests for Machine information.
Check that we compute the Machine Count and Core Count correctly.
Some tests around migrations.
Importing models should be suppressed. Not sure what
needs to be done with migration attempts.
Add 'all' as a State flag.
Update the API Server to be able to take All as a parameter.

We still need to update all of the client side code to pass
in the new structure and handle the new parameter.

We are so close \o/
Thank you for under-taking this and completing it in such a short time :)
I've left some comments but this is a lot better than what we currently have and I'd love to see this in RC for users to take for a spin. We can follow up on any fall-outs/test gaps later...

@@ -30,6 +30,8 @@ type ModelManagerBackend interface {
ModelUUID() string
ModelUUIDsForUser(names.UserTag) ([]string, error)
+ ModelBasicInfoForUser(user names.UserTag) ([]state.ModelAccessInfo, error)
@anastasiamac

anastasiamac Nov 26, 2017

Member

Did we want to keep both ModelBasicInfoForUser and ModelSummariesForUser?

@jameinel

jameinel Nov 27, 2017

Owner

I don't like the naming of ModelBasicInfoForUser but it is a necessary backing for "ListModels"

We might be able to get rid of ModelUUIDsForUser, though.

@@ -160,6 +162,7 @@ func (st modelManagerStateShim) GetModel(modelUUID string) (Model, func() bool,
// Model implements ModelManagerBackend.
func (st modelManagerStateShim) Model() (Model, error) {
+ st.State.Model()
@anastasiamac

anastasiamac Nov 26, 2017

Member

Maybe worth a comment why this line is needed?... Are we loading model into state here?

@jameinel

jameinel Nov 27, 2017

Owner

I actually think it shouldn't be there. I'm a bit surprised that it is. (removed)

@@ -0,0 +1,217 @@
+// Copyright 2015 Canonical Ltd.
@anastasiamac

anastasiamac Nov 26, 2017

Member

Oops, I must have wanted to be younger... This should be 2017.

+ attrs["agent-version"] = jujuversion.Current.String()
+ cfg, err := config.New(config.UseDefaults, attrs)
+ c.Assert(err, jc.ErrorIsNil)
+
@anastasiamac

anastasiamac Nov 26, 2017

Member

Not sure a new line here does much :)

-// ListModels returns the models that the specified user
-// has access to in the current server. Only that controller owner
+// ListModelSummaries returns models that the specified user
+// has access to in the current server. Only the controller owner
@anastasiamac

anastasiamac Nov 26, 2017

Member

Since we do not have the concept of 'controller owner', maybe we should say here 'controller admin'?

+ if err == nil {
+ summary.UserAccess = access
+ }
+ // TODO (anastasiamac 2017-11-24) what happens here if there is no migration in progress?
@anastasiamac

anastasiamac Nov 26, 2017

Member

We do not need this TODO :)

+}
+
+// ListModels returns the models that the specified user
+// has access to in the current server. Only that controller owner
@jameinel

jameinel Nov 27, 2017

Owner

Replaced with:
// ListModels returns the models that the specified user
// has access to in the current server. Controller admins (superuser)
// can list models for any user. Other users
// can only ask about their own models.

good?

- OwnerTag: model.Owner().String(),
+ Name: mi.Name,
+ UUID: mi.UUID,
+ OwnerTag: names.NewUserTag(mi.Owner).String(),
@anastasiamac

anastasiamac Nov 26, 2017

Member

Maybe worth checking that mi.Owner is a valid user?

@jameinel

jameinel Nov 27, 2017

Owner

we don't have a place to inject the error as a response.
It would be a bug in our output of ModelBasicInfoForUser (rather than a bug in user input).
But sure, we'll do as such.
Panic() in production code is bad mojo.

cmd/juju/controller/listmodels.go
+
+ haveModels := false
+ if modelmanagerAPI.BestAPIVersion() > 3 {
+ // New code path
@anastasiamac

anastasiamac Nov 26, 2017

Member

Probably do not need this comment now :)

cmd/juju/controller/listmodels.go
+ w.Println("Access", "Last connection")
+}
+
+// formatTabular takes model summaries set to adhere to the cmd.Formatter interface
@anastasiamac

anastasiamac Nov 26, 2017

Member

// tabularSummaries

cmd/juju/controller/listmodels_test.go
+// This test is only needed for older api versions as
+// whether the user has an access to a model will be checked on
+// the api side and the model data will not be sent.
+func (s *BaseModelsSuite) TestAllModelsWithOneUnauthorised(c *gc.C) {
@anastasiamac

anastasiamac Nov 26, 2017

Member

My thinking was - either the test comment is incorrect or this should reside on ModelSuiteV3 only. It looks like the test is passing for both v3 and v4 suites, so maybe get rid of the comment?

@jameinel

jameinel Nov 27, 2017

Owner

removed the comment.

featuretests/cmd_juju_controller_test.go
@@ -99,12 +99,14 @@ func (s *cmdControllerSuite) TestCreateModelAdminUser(c *gc.C) {
func (s *cmdControllerSuite) TestAddModelNormalUser(c *gc.C) {
s.createModelNormalUser(c, "new-model", false)
context := s.run(c, "list-models", "--all")
+ // TODO (anastasiamac 2017-11-23) currently logged in user does not have any right to the model,
@anastasiamac

anastasiamac Nov 26, 2017

Member

This comment can be deleted.

state/modelsummaries.go
+ Status status.StatusInfo
+
+ // Needs ModelUser, ModelUserLastConnection, and Permissions collections
+ // This information will only be filled out if includeUsers is true and the user has at least Admin (write?) access
@anastasiamac

anastasiamac Nov 26, 2017

Member

This comment block needs to be updated/re-phrased :)

+
+ cfg, err := config.New(config.NoDefaults, doc.Settings)
+ if err != nil {
+ // err on one model should kill all the other ones?
@anastasiamac

anastasiamac Nov 26, 2017

Member

No! Please do not :)

@jameinel

jameinel Nov 27, 2017

Owner

well, it currently does, what errors should be considered ok to just skip and go on to the next.
Is it just mgo.NotFound?
What errors can config.New give?
We have lots of potential validation errors.
It doesn't quite feel right that an invalid config on a model would cause that model to disappear from "juju models"
Should we be adding an 'err' field on ModelSummaries?

+ }
+ if !remaining.IsEmpty() {
+ // XXX: What error is appropriate? Do we need to care about models that its ok to be missing?
+ return errors.Errorf("could not find settings/config for models: %v", remaining.SortedValues())
@anastasiamac

anastasiamac Nov 26, 2017

Member

I'd prefer to log this as a Warning and return nil (i.e. continue processing up the stack)

@jameinel

jameinel Nov 27, 2017

Owner

I'd like to discuss with you the general things around getting errors on a particular model.
We should be adding tests for it, and improve the behavior.
Should we just be adding cards around this?

state/modelsummaries.go
+}
+
+// removeExistingUsers takes a set of usernames, and removes any of them that are known-valid users.
+// it leaves behind names that could not be validated
@anastasiamac

anastasiamac Nov 26, 2017

Member

Capitalize :) and finish the sentence?..

@jameinel

jameinel Nov 27, 2017

Owner

actually, we can just drop this whole function because we don't include users anymore.
removed.

state/modelsummaries.go
+ for i, uuid := range p.modelUUIDs {
+ statusIds[i] = uuid + ":" + modelGlobalKey
+ }
+ // TODO: Track remaining and error if we're missing any
@anastasiamac

anastasiamac Nov 26, 2017

Member

If we plan to leave TODOs in this code, could we please follow the format TODO (name)...
Although there is 'git blame', the code could be shuffled and then it can become harder to track the person with the context :)

@jameinel

jameinel Nov 27, 2017

Owner

updated all the TODO I found in this file.
We should probably just go through them all as potential cards and decide if we actually want to do them.

state/modelsummaries.go
+ }
+ modelIdx, ok := p.indexByUUID[modelUUID]
+ if !ok {
+ // ??
@anastasiamac

anastasiamac Nov 26, 2017

Member

Might be worthwhile to elaborate here :)

@jameinel

jameinel Nov 27, 2017

Owner

elaborated, added a TODO

state/modelsummaries_test.go
+ "gopkg.in/juju/names.v2"
+ "gopkg.in/mgo.v2/bson"
+ "github.com/juju/utils"
+ //"gopkg.in/macaroon.v1"
@anastasiamac

anastasiamac Nov 26, 2017

Member

If we don't need these imports, let's delete them altogether rather than comment out :)

state/modelsummaries_test.go
+}
+
+func (s *ModelSummariesSuite) TestModelsForUser2(c *gc.C) {
+ // User1 is only added to the model they own and the shared model
@anastasiamac

anastasiamac Nov 26, 2017

Member

You probably wanted "User2"

+ c.Check(names, gc.DeepEquals, []string{"shared", "user2model"})
+}
+
+func (s *ModelSummariesSuite) TestModelsForUser3(c *gc.C) {
@anastasiamac

anastasiamac Nov 26, 2017

Member

Does this test the same as TestModelForUser2? (just for a different user but the same in principal...)

@jameinel

jameinel Nov 27, 2017

Owner

User 3 is admin on the shared model, user 2 is read access on the shared model, user 1 is write access.
I updated the comments to indicate as such.

+ // Since the new model is importing, when we do the list we shouldn't see it.
+ names := s.modelNamesForUser(c, "user3admin")
+ c.Check(names, gc.DeepEquals, []string{"shared", "user3model"})
+ // Superuser doesn't see importing models, either
@anastasiamac

anastasiamac Nov 26, 2017

Member

Completely off-side but .... I wonder why we have decided that this is how we want things? Would not the importing user, superuser in this case, want to know that there is a model that is about to come in because it is being imported?

I mean is it possible that the user will import, not see the model in the list and will try to import again?

@jameinel

jameinel Nov 27, 2017

Owner

I have the feeling you're right. the reason we were stripping importing modules is because it caused "juju models" to fail, not because we didn't want them in the output.
We should put this as a card.

+ c.Check(names, gc.DeepEquals, []string{"shared", "testenv", "user1model", "user2model", "user3model"})
+}
+
+func (s *ModelSummariesSuite) TestContainsConfigInformation(c *gc.C) {
@anastasiamac

anastasiamac Nov 26, 2017

Member

And we'd want a test where there is no config for a model. We can follow it up once this PR lands...

state/modeluser.go
+func (st *State) isUserSuperuser(user names.UserTag) (bool, error) {
+ access, err := st.UserAccess(user, st.controllerTag)
+ if err != nil {
+ // We don't suppress NotFound becaus
@anastasiamac

anastasiamac Nov 26, 2017

Member

Please continue :)

@jameinel

jameinel Nov 27, 2017

Owner

actually, the change to --all probably means we have introduced a bug.
Namely, you can now ask for st.ModelSummariesForUser for users that don't exist at all.
Maybe that's not a problem, but it may be better to give a "no such user" rather than an empty list of models.
Thoughts?
I added a TODO so that we can decide how we want this to behave.

state/settings.go
@@ -43,7 +43,7 @@ type settingsDoc struct {
Settings settingsMap `bson:"settings"`
// Version is a version number for the settings,
- // and is increased every time the settings change.
+ // and is increased every time the settingsechange.
@anastasiamac

anastasiamac Nov 26, 2017

Member

I think we can un-do this :D

Owner

jameinel commented Nov 27, 2017

I didn't realize this was 2.5k lines patch.
I'm pretty sure I've addressed everything, at least leaving TODO in the code for us to address any remaining items.
$$merge$$

Contributor

jujubot commented Nov 27, 2017

Status: merge request accepted. Url: http://ci.jujucharms.com/job/github-merge-juju

Review feedback from Anastasia.
Lots of little tweaks to code, some TODO, etc.
Owner

jameinel commented Nov 27, 2017

$$merge$$ (I had committed the fixes, but push failed because I had updated the branch from another machine)

Contributor

jujubot commented Nov 27, 2017

Build failed: Tests failed
build url: http://ci.jujucharms.com/job/github-merge-juju/590

Owner

jameinel commented Nov 27, 2017

$$merge$$ when responding to the review comments I broke ListModels (ParseUserTag takes a tag object, not a user string.)

Contributor

jujubot commented Nov 27, 2017

Status: merge request accepted. Url: http://ci.jujucharms.com/job/github-merge-juju

@jujubot jujubot merged commit fbb8949 into juju:develop Nov 27, 2017

1 check passed

continuous-integration/jenkins/pr-merge This commit looks good
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment