merge 2.3 changes into 2.4 #9181
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We'll use it to check for free space before downloading and unpacking the agent binary from the controller.
I'm about to add another one (for checking disk space), and it's getting a bit ridiculous.
Before the upgrader starts downloading a binary to unpack and install it will check that there is at least 250MiB free on the volumes for temp storage and the Juju data dir.
Available is the number of bytes that can be allocated to any user. Free includes the blocks reserved by the filesystem for the superuser to perform recovery/emergency actions when the disk is full. We shouldn't be filling up that space with downloaded binaries.
Only log every minute to reduce logspam but keep the message visible.
…rade juju#9180 ## Description of change If the disk was full when trying to download the agent binary for an upgrade, Juju would log fairly cryptic errors that didn't clearly spell out the problem. Instead of allowing the download or unpacking to just fail, we now check for 250MiB available space before beginning. If there's not enough space the error message logged is: ``` not enough free disk space on "/var/lib/juju" for upgrade: 103MiB available, require 250MiB ``` The free space check is done for the Juju data directory as well as wherever the OS temp storage is (for unpacking). This exposes and uses `upgrades.CheckFreeDiskSpace` so it can be used from the worker. The function now uses `DiskUsage.Available()`, because in manual testing I couldn't ever get the disk space check to fail using `.Free()` and after some research Available seems like the right choice. I had trouble finding information about the difference between the fields in the underlying `statfs` syscall - the man page just says `f_bfree: free blocks in filesystem` versus `f_bavail: free blocks available to unprivileged user` without any indication of why or in what circumstances you'd use one over the other. [This discussion](https://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why) has some detail about superuser-reserved space that suggests it's really/mostly for emergency recovery in nearly-full disk situations. That indicates that Available is the right field for this check - we shouldn't be filling up emergency space with downloaded agent binaries just because the agent is running as root. ## QA steps * Deploy an application, on a machine in the application use `fallocate -l <size> bigfile` to create a file that fills *almost* all the available space reported by `df -h` - so that there's less than 250MiB free. * Run `juju upgrade-juju -m controller --build-agent && sleep 20 && juju upgrade-juju ` to upgrade the model. * Look in the model log, you should see the uniter on the full machine complaining that there's not enough space in /var/lib/juju. * Delete bigfile; the upgrade should continue and succeed. ## Bug reference https://bugs.launchpad.net/juju/+bug/1782367
This brings the fix for https://pad.lv/1782367 forward.
anastasiamac
approved these changes
Sep 10, 2018
|
|
jameinel
added a commit
to jameinel/juju
that referenced
this pull request
Oct 25, 2018
Including: * Merge pull request juju#9359 from juju/2.3 * Merge pull request juju#9355 from howbazaar/2.4-engine-backoff * Merge pull request juju#9345 from anastasiamac/only-clear-current-when-asked-lp1796148 * Merge pull request juju#9335 from babbageclunk/mux-worker * Merge pull request juju#9333 from wallyworld/merge-2.3-20181018 * Merge pull request juju#9323 from ExternalReality/fix_logging_path * Merge pull request juju#9327 from mitechie/enable-export-bundle * Merge pull request juju#9315 from manadart/2.4-acceptance-maas-network-bind * Merge pull request juju#9300 from manadart/2.4-provisioner-testing-with-mocks * Merge pull request juju#9287 from manadart/2.4 * Merge pull request juju#9297 from anastasiamac/24-better-debug-lp1779677 * Merge pull request juju#9298 from wallyworld/cmr-controller-connect-robust-2.4 * Merge pull request juju#9251 from howbazaar/bundle-export-series * Merge pull request juju#9293 from babbageclunk/bundle-deploy-panic-2.4 * Merge pull request juju#9285 from howbazaar/2.4-fix-multiwatcher * Merge pull request juju#9264 from mitechie/forward-9235 * Merge pull request juju#9263 from wallyworld/goal-state-units * Merge pull request juju#9255 from howbazaar/2.4-bionic-tests * Merge pull request juju#9203 from wallyworld/go-1.11-2.4 * Merge pull request juju#9191 from howbazaar/2.4-status-retry * Merge pull request juju#9178 from hmlanigan/fixes * Merge pull request juju#9177 from SimonRichardson/ec2-instance-types-2.4 * Merge pull request juju#9181 from babbageclunk/2.3-merge-into-2.4 * Merge pull request juju#9169 from juju/2.3 * Merge pull request juju#9167 from babbageclunk/caasfirewaller-test-fix-2.4 * Merge pull request juju#9166 from babbageclunk/s390x-test-fix-2.4 * Merge pull request juju#9152 from jameinel/2.3-into-2.4 * Merge pull request juju#9080 from vinu2003/juju_exportBundle_timeDelayFix * Merge pull request juju#9138 from wallyworld/charm.v6-dep-update * Merge pull request juju#9137 from anastasiamac/test-order-fix-24 * Merge pull request juju#9134 from anastasiamac/merge-23-into-24-20180829 * Merge pull request juju#9132 from wallyworld/improve-charm-versionstring * Merge pull request juju#9126 from anastasiamac/empty-default-settings-lp1789415-24 * Merge pull request juju#9117 from anastasiamac/init-cmd-lp1785205-24 * Merge pull request juju#9104 from wallyworld/merge-2.3-20180823 Bugs * https://bugs.launchpad.net/juju/+bug/1798485 * https://bugs.launchpad.net/juju/+bug/1793245 * https://bugs.launchpad.net/juju/+bug/1796148 * https://bugs.launchpad.net/juju/+bug/1798001 * https://bugs.launchpad.net/juju/+bug/1793284 * https://bugs.launchpad.net/juju/+bug/1796106 * https://bugs.launchpad.net/juju/+bug/1779677 * https://bugs.launchpad.net/juju/+bug/1795499 * https://bugs.launchpad.net/juju/+bug/1773357 * https://bugs.launchpad.net/juju/+bug/1792299 * https://bugs.launchpad.net/juju/+bug/1782803 * https://bugs.launchpad.net/juju/+bug/1782367 * https://bugs.launchpad.net/juju/+bug/1790626 * https://bugs.launchpad.net/juju/+bug/1789211 * https://bugs.launchpad.net/juju/+bug/1789447 * https://bugs.launchpad.net/juju/+bug/1789415 * https://bugs.launchpad.net/juju/+bug/1785205 * https://bugs.launchpad.net/juju/2.3/+bug/1788554
Merged
jujubot
added a commit
that referenced
this pull request
Oct 26, 2018
#9372 ## Description of change Merge of the current tip of 2.4 into 2.5 bringing in patches: prdesc Merge pull request #9359 from juju/2.3 prdesc Merge pull request #9355 from howbazaar/2.4-engine-backoff prdesc Merge pull request #9345 from anastasiamac/only-clear-current-when-asked-lp1796148 prdesc Merge pull request #9335 from babbageclunk/mux-worker prdesc Merge pull request #9333 from wallyworld/merge-2.3-20181018 prdesc Merge pull request #9323 from ExternalReality/fix_logging_path prdesc Merge pull request #9327 from mitechie/enable-export-bundle prdesc Merge pull request #9315 from manadart/2.4-acceptance-maas-network-bind prdesc Merge pull request #9300 from manadart/2.4-provisioner-testing-with-mocks prdesc Merge pull request #9287 from manadart/2.4 prdesc Merge pull request #9297 from anastasiamac/24-better-debug-lp1779677 prdesc Merge pull request #9298 from wallyworld/cmr-controller-connect-robust-2.4 prdesc Merge pull request #9251 from howbazaar/bundle-export-series prdesc Merge pull request #9293 from babbageclunk/bundle-deploy-panic-2.4 prdesc Merge pull request #9285 from howbazaar/2.4-fix-multiwatcher prdesc Merge pull request #9264 from mitechie/forward-9235 prdesc Merge pull request #9263 from wallyworld/goal-state-units prdesc Merge pull request #9255 from howbazaar/2.4-bionic-tests prdesc Merge pull request #9203 from wallyworld/go-1.11-2.4 prdesc Merge pull request #9191 from howbazaar/2.4-status-retry prdesc Merge pull request #9178 from hmlanigan/fixes prdesc Merge pull request #9177 from SimonRichardson/ec2-instance-types-2.4 prdesc Merge pull request #9181 from babbageclunk/2.3-merge-into-2.4 prdesc Merge pull request #9169 from juju/2.3 prdesc Merge pull request #9167 from babbageclunk/caasfirewaller-test-fix-2.4 prdesc Merge pull request #9166 from babbageclunk/s390x-test-fix-2.4 prdesc Merge pull request #9152 from jameinel/2.3-into-2.4 prdesc Merge pull request #9080 from vinu2003/juju_exportBundle_timeDelayFix prdesc Merge pull request #9138 from wallyworld/charm.v6-dep-update prdesc Merge pull request #9137 from anastasiamac/test-order-fix-24 prdesc Merge pull request #9134 from anastasiamac/merge-23-into-24-20180829 prdesc Merge pull request #9132 from wallyworld/improve-charm-versionstring prdesc Merge pull request #9126 from anastasiamac/empty-default-settings-lp1789415-24 prdesc Merge pull request #9117 from anastasiamac/init-cmd-lp1785205-24 prdesc Merge pull request #9104 from wallyworld/merge-2.3-20180823 ## QA steps See individual patches. ## Documentation changes See individual patches. ## Bug reference prdesc https://bugs.launchpad.net/juju/+bug/1798485 prdesc https://bugs.launchpad.net/juju/+bug/1793245 prdesc https://bugs.launchpad.net/juju/+bug/1796148 prdesc https://bugs.launchpad.net/juju/+bug/1798001 prdesc https://bugs.launchpad.net/juju/+bug/1793284 prdesc https://bugs.launchpad.net/juju/+bug/1796106 prdesc https://bugs.launchpad.net/juju/+bug/1779677 prdesc https://bugs.launchpad.net/juju/+bug/1795499 prdesc https://bugs.launchpad.net/juju/+bug/1773357 prdesc https://bugs.launchpad.net/juju/+bug/1792299 prdesc https://bugs.launchpad.net/juju/+bug/1782803 prdesc https://bugs.launchpad.net/juju/+bug/1782367 prdesc https://bugs.launchpad.net/juju/+bug/1790626 prdesc https://bugs.launchpad.net/juju/+bug/1789211 prdesc https://bugs.launchpad.net/juju/+bug/1789447 prdesc https://bugs.launchpad.net/juju/+bug/1789415 prdesc https://bugs.launchpad.net/juju/+bug/1785205 prdesc https://bugs.launchpad.net/juju/2.3/+bug/1788554
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Forward-merge fix for this bug into the 2.4 branch (for PR #9180).
QA steps
Bug reference
https://bugs.launchpad.net/juju/+bug/1782367