Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
state: Split logs collection into a collection per model #7441
Conversation
babbageclunk
added some commits
Jun 1, 2017
|
Job got killed for some reason. !!chittychitty!! |
babbageclunk
added some commits
Jun 2, 2017
babbageclunk
changed the title from
WIP state: Split logs collection into a collection per model
to
state: Split logs collection into a collection per model
Jun 2, 2017
|
!!buildplease!! |
| @@ -29,7 +29,7 @@ import ( | ||
| // TODO(wallyworld) - lp:1602508 - collections need to be defined in collections.go | ||
| const ( | ||
| logsDB = "logs" | ||
| - logsC = "logs" | ||
| + logsPrefix = "logs." |
| @@ -745,15 +746,15 @@ func PruneLogs(st MongoSessioner, minLogTime time.Time, maxLogsMB int) error { | ||
| // Do further pruning if the logs collection is over the maximum size. |
| @@ -395,6 +395,9 @@ func (st *State) NewModel(args ModelArgs) (_ *Model, _ *State, err error) { | ||
| return nil, nil, errors.Annotate(err, "granting admin permission to the owner") | ||
| } | ||
| + if err := InitDbLogs(session, uuid); err != nil { | ||
| + return nil, nil, errors.Trace(err) |
| @@ -235,7 +235,7 @@ func DropOldLogIndex(st *State) error { | ||
| // If the log collection still has the old e,t index, remove it. | ||
| key := []string{"e", "t"} | ||
| db := st.MongoSession().DB(logsDB) | ||
| - collection := db.C(logsC) | ||
| + collection := db.C("logs") |
wallyworld
Jun 2, 2017
Owner
The "logs" collection won't necessarily be there will it? Once we upgrade to 2.2 it will be gone. So this DropOldLogIndex step can be deleted.
babbageclunk
Jun 3, 2017
•
Edited 1 time
-
babbageclunk
Jun 3, 2017
Member
-
Jun 3, 2017
It's used in steps_21 - although I guess if someone is upgrading to 2.1 they aren't running this binary, are they? So then the only use for this is if someone is upgrading from 2.0 to 2.2? You're right - I'll delete the step. It feels slightly weird to be deleting something from the 2.1 steps because of something we're changing in 2.2 though.
| +func SplitLogCollections(st *State) error { | ||
| + session := st.MongoSession() | ||
| + db := session.DB(logsDB) | ||
| + oldLogs := db.C("logs") |
wallyworld
Jun 2, 2017
Owner
This collection won't necessarily be there once we drop it. I guess we drop it below so even though we create it here, it will be gone later. But if there's an error in the upgrade step, and logs had already been dropped before, we will now leave it behind when we shouldn't
babbageclunk
Jun 3, 2017
•
Edited 1 time
-
babbageclunk
Jun 3, 2017
Member
-
Jun 3, 2017
I'll add a check that exits if the collection doesn't exist - I have a feeling that the remove all might raise an error if the collection doesn't exist. (I'll add a test.)
babbageclunk
Jun 3, 2017
Member
It turned out @howbazaar already had handling for the remove failing if the collection didn't exist - calling db.C("logs") doesn't create the collection, it is created lazily on insert. I added a test for that case and verified that it failed if the dropping-nonexistent-collection error handling wasn't there.
| + db := session.DB(logsDB) | ||
| + oldLogs := db.C("logs") | ||
| + | ||
| + // If we haven't seen any particular environment, we need to initialise |
| + seen.Add(newCollName) | ||
| + } | ||
| + | ||
| + delete(doc, "e") // old env uuid |
| + delete(doc, "e") // old env uuid | ||
| + | ||
| + if err := newLogs.Insert(doc); err != nil { | ||
| + return errors.Annotate(err, "failed to insert log record") |
|
Thanks for the comments @wallyworld, they all make sense. I'll address them when I can later on. Check build failed because the windows build crapped out. !!trytryagain!! |
|
$$merge$$ |
|
Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju |
babbageclunk commentedJun 1, 2017
•
Edited 1 time
-
babbageclunk
Jun 2, 2017
Description of change
When destroying a model on a controller with lots of models, deleting the logs for that model is by far the most expensive part of the operation. Splitting the log collection so we have one log collection per model will mean that we can drop the collection instead, and save space by not storing the model UUID on each row and in the indexes.
QA steps