2.2 force upgrade #7958

jameinel · 2017-10-24T14:55:09Z

Description of change

This addresses an issue that has come up a few times, where we are asking people to upgrade their Controller, but they are unable to because of some agents that aren't working properly.

It does 3 primary things:

Update the State code to allow an override so we will skip the agent version check.
Update the API to expose it so that new Controllers can be triggered by new Juju clients (juju upgrade-juju --ignore-agent-versions)
Brings in Tim's 'force-upgrade' script as something you can run directly on controllers for when their current version is to old.

QA steps

$ juju bootstrap lxd
$ juju deploy ubuntu
$ juju ssh ubuntu/0 "sudo service jujud-unit-ubuntu-0 stop"
$ juju upgrade-juju
$ juju upgrade-juju # should fail because unit-ubuntu-0 didn't get upgraded
$ juju upgrade-juju --ignore-agent-versions # should succeed

Documentation changes

The flag should be documented, along with when it is appropriate to use it.

Bug reference

lp:1726893

At the State level, allow a force flag that ignores the agent-version check. Expose this with a direct client that can trigger an upgrade to any version you want.

Rename 'force-upgrade' to 'juju-force-upgrade'. Expose a new value in the API that allows us to pass Force through to the state code.

Change the flag from 'force' to 'ignore-agent-versions' because it is more descriptive about what we're actually doing with the flag. And we don't actually let you force things like models newer than controller, etc.

nskaggs · 2017-10-24T15:14:11Z

As a drive-by, can you confirm the model version check is accurate? I saw a failure comparing versions with rc in it for instance..

(Basically, can you add a test for versions with strings in them and confirm)

anastasiamac

Looks great overall \o/
Thank you for a nice doc string explaining the parameter and awesome QA steps on the PR!
I'd love to see the best facade check in CLI.
Also are you sure you don't want to bump a facade version? You have after all changed an input type...
Leaner api layer tests is just a wish but it would be nice if we started cleaning them up as we go rather than fixing & modifying existing full-stack, heavy tests....

anastasiamac · 2017-10-24T23:35:33Z

cmd/juju/commands/upgradejuju.go

@@ -363,7 +369,8 @@ func (c *upgradeJujuCommand) Run(ctx *cmd.Context) (err error) {
 				return block.ProcessBlockedError(err, block.BlockChange)
 			}
 		}
-		if err := client.SetModelAgentVersion(context.chosen); err != nil {


Any specific reasons why you did not want to use BestFacadeVersion check to determine which call to make?

anastasiamac · 2017-10-24T23:37:22Z

api/client_test.go

@@ -527,7 +527,7 @@ func (s *clientSuite) TestSetModelAgentVersionDuringUpgrade(c *gc.C) {
 	_, err = s.State.EnsureUpgradeInfo(machine.Id(), agentVersion, nextVersion)
 	c.Assert(err, jc.ErrorIsNil)

-	err = s.APIState.Client().SetModelAgentVersion(nextVersion)
+	err = s.APIState.Client().SetModelAgentVersion(nextVersion, false)


I appreciate that this is an old, full-stack test. It'd be great to make it leaner \o/ Especially since the only thing we'd want to check here is that all the parameters are passed in.

nskaggs · 2017-10-25T15:23:09Z

@jameinel 2.2.6 is going right now -- can we ship this?

jameinel · 2017-10-25T16:00:48Z

$$merge$$

…

On Wed, Oct 25, 2017 at 7:23 PM, nskaggs ***@***.***> wrote: @jameinel <https://github.com/jameinel> 2.2.6 is going right now -- can we ship this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7958 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMYfUqoT3Qg7uLZS6G8guAzn7zaiDMXks5sv1JegaJpZM4QEkSh> .

jujubot · 2017-10-25T16:02:07Z

Status: merge request accepted. Url: http://ci.jujucharms.com/job/github-merge-juju

nskaggs · 2017-10-26T13:43:37Z

@pmatulis This needs documented. It shipped in 2.2.6. It's kind of a get of a jail card, if a model is really broken, this can let you fix it. Not sure how we want to highlight it, or the extent to which we do.

pmatulis · 2017-10-26T18:38:24Z

@jameinel
For doc purposes,

Is this flag only to be used in the context of a controller model?
- If so, is there any problem using it in a non-controller model (user mistake)?
- If so, what to do about troubled units in other models if an upgrade is not possible?
Are there any risks associated with the usage of this flag?

jameinel · 2017-10-29T04:10:39Z

This is valid for both controller and non controller models. You can still only upgrade non-controller models to agents that are <= the controller agent version.
Before this flag, we would check that all agents are == 'currentVersion' before we would allow you to upgrade. (So when going 2.1.1 => 2.1.3 => 2.2.2, we would require that all agents report 2.1.3 before we would let you upgrade to 2.2.2.)
That is slightly safer, because then we know that you don't end up in a weird skewed system where you have a big spread of all the versions of all the agents. (some are 2.1.1, some 2.1.3, some 2.2.2.)
In practice, what we've actually seen is that the only time agent versions end up being skewed is because specific agents/machines are down/broken. And having to fix those agents before we can upgrade the rest of the agents has generally been prohibitive, so this gives a way out.

The 'juju-force-upgrade' command exists to trigger the same upgrade (still checks versions, still ensures controllers are higher version than individual models, just ignores agent version skew). It
a) Works on controllers that don't support the 'IgnoreAgentVersions' flag.
b) Requires that you run on one of the controller machines (as root so you can read /var/log/juju/agents/machine-*/agent.conf)

Once 2.2.6+ is prevalent enough, I would expect juju-force-upgrade to be deprecated and retired.

pmatulis · 2017-10-29T23:29:01Z

@jameinel

So if some agents/machines are down/broken the user may end up with version skew?
Where do I find juju-force-upgrade? I cannot find it.
What is the connection between the deprecation of juju-force-upgrade and 2.2.6+?
Similar to the last question, why is juju-force-upgrade only in 2.2(.6) and not in 2.3?

jameinel · 2017-10-30T06:20:43Z

1. As an example: If you did "juju bootstrap --agent-version=2.1.0" and then "juju upgrade-juju --agent-version=2.1.1", and some of the agents were down, they will be flagged inside the controller as still being 2.1.0. Thus when you try to do "juju upgrade-juju --agent-version 2.1.2" we will refuse, because not all the agents are reporting 2.1.1. The "juju upgrade-juju --ignore-agent-versions" is a way to do the upgrade anyway. (the original check is still in place, because if those are not at 2.1.1 simply because they haven't caught up yet, it is reasonable to wait for them.) 2. scripts/juju-force-upgrade should be the source tree for it. If you did "go install github.com/juju/juju/..." then it will be in $GOPATH/bin. I don't know how Nicholas is packaging things as to whether it will be generally released and available, or whether it is just a build-from-source-when-you-need-it. 3. If you are running a 2.2.6+ controller, then you can use "juju upgrade-juju --ignore-agent-versions" instead of using juju-force-upgrade. If you are running say a 2.1 controller and want to upgrade *to* 2.2.6 then you probably need "juju-force-upgrade" because the 2.1.X controller won't accept the --ignore-agent-versions flag. 4. I do think it will be in 2.3 (I think it is there now). It was just a case of landing the code in 2.2, and then merging from 2.2 into 2.3 and it took a day or two for me to do the merge because we ended up having a merge conflict. I *think* that should all be sorted out now, and the code for juju-force-upgrade should be in 2.3-beta2, but its possible I missed something there. (And I'm also not sure how Nicholas is planning on publishing things.) One option is that we could just publish a snap of juju-force-upgrade and make it accessible in that fashion.

…

On Mon, Oct 30, 2017 at 3:29 AM, Peter Matulis ***@***.***> wrote: @jameinel <https://github.com/jameinel> 1. So if some agents/machines are down/broken the user may end up with version skew? 2. Where do I find juju-force-upgrade? I cannot find it. 3. What is the connection between the deprecation of juju-force-upgrade and 2.2.6+? 4. Similar to the last question, why is juju-force-upgrade only in 2.2(.6) and not in 2.3? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7958 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMYffOo5-qIaZODF-odmfOykLGIZihWks5sxQo-gaJpZM4QEkSh> .

pmatulis · 2017-10-30T20:40:47Z

The upgrade will abort unless all agents are only 1 patch version behind the targeted version. By overriding that we are getting users into a position we don't want them to be in?
The docs will refrain from mentioning juju-force-upgrade until the command is available for regular users.
What is the oldest version that supports the --ignore-agent-versions flag?
Please let the docs team know what happens.

New question:

How does one discover what values for --agent-version can be used (with bootstrap or upgrade-juju)?

jameinel · 2017-11-01T07:51:53Z

1. By default we refuse if there is any skew between any agent version and what is flagged as "current". --ignore-agent-versions would allow you to get into a heavily skewed view, but only for agents that are unable to upgrade (which has always only actually happened when the agents aren't actually running). 2. agreed 3. 2.2.6 4. sure b1: Its whatever versions of Juju that we have published. There is 'juju upgrade-juju --dry-run' which reports the version it would use, but I don't believe we have a Juju command to expose what versions are available.

…

On Tue, Oct 31, 2017 at 12:40 AM, Peter Matulis ***@***.***> wrote: 1. The upgrade will abort unless all agents are only 1 patch version behind the targeted version. By overriding that we are getting users into a position we don't want them to be in? 2. The docs will refrain from mentioning juju-force-upgrade until the command is available for regular users. 3. What is the oldest version that supports the --ignore-agent-versions flag? 4. Please let the docs team know what happens. New question: 1. How does one discover what values for --agent-version can be used (with bootstrap or upgrade-juju)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7958 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMYfeihIjfh4rSigK9UQNUH1ASm4bJFks5sxjRQgaJpZM4QEkSh> .

pmatulis · 2017-11-02T18:53:44Z

For an aborted upgrade, does upgrade-juju give sufficient information for the user to rectify the situation (and retry)?
Similarly, if the flag is used, is the user provided enough information to rectify (and retry with the flag)?

jameinel added 3 commits October 24, 2017 14:01

Allow forcing a desired model version.

c8cfee0

At the State level, allow a force flag that ignores the agent-version check. Expose this with a direct client that can trigger an upgrade to any version you want.

Expose 'juju upgrade-juju --force'

45af426

Rename 'force-upgrade' to 'juju-force-upgrade'. Expose a new value in the API that allows us to pass Force through to the state code.

Expose the flag all the way through the CLI.

1640dc0

Change the flag from 'force' to 'ignore-agent-versions' because it is more descriptive about what we're actually doing with the flag. And we don't actually let you force things like models newer than controller, etc.

anastasiamac suggested changes Oct 24, 2017

View reviewed changes

jujubot merged commit abaa606 into juju:2.2 Oct 25, 2017

pmatulis mentioned this pull request Oct 26, 2017

forced upgrades juju/docs#2256

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.2 force upgrade #7958

2.2 force upgrade #7958

jameinel commented Oct 24, 2017

nskaggs commented Oct 24, 2017 •

edited

anastasiamac left a comment

anastasiamac Oct 24, 2017

anastasiamac Oct 24, 2017

nskaggs commented Oct 25, 2017

jameinel commented Oct 25, 2017 via email

jujubot commented Oct 25, 2017

nskaggs commented Oct 26, 2017

pmatulis commented Oct 26, 2017

jameinel commented Oct 29, 2017

pmatulis commented Oct 29, 2017

jameinel commented Oct 30, 2017 via email

pmatulis commented Oct 30, 2017

jameinel commented Nov 1, 2017 via email

pmatulis commented Nov 2, 2017

2.2 force upgrade #7958

2.2 force upgrade #7958

Conversation

jameinel commented Oct 24, 2017

Description of change

QA steps

Documentation changes

Bug reference

nskaggs commented Oct 24, 2017 • edited

anastasiamac left a comment

Choose a reason for hiding this comment

anastasiamac Oct 24, 2017

Choose a reason for hiding this comment

anastasiamac Oct 24, 2017

Choose a reason for hiding this comment

nskaggs commented Oct 25, 2017

jameinel commented Oct 25, 2017 via email

jujubot commented Oct 25, 2017

nskaggs commented Oct 26, 2017

pmatulis commented Oct 26, 2017

jameinel commented Oct 29, 2017

pmatulis commented Oct 29, 2017

jameinel commented Oct 30, 2017 via email

pmatulis commented Oct 30, 2017

jameinel commented Nov 1, 2017 via email

pmatulis commented Nov 2, 2017

nskaggs commented Oct 24, 2017 •

edited