clean up uniter downloads #6325
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 6 commits
September 27, 2016 10:28
The Download and Downloader were unnecessarily complicated with more than one way to do things and confused aborting of downloads. - Check req.Verify closer to where it's used to minimise the possibility of mistakes. - Fixed/removed racy tests for stopping downloads. This also shaved 10s of the package test runtime. - Removed the use of both a tomb and a channel to abort. downloads. Downloads are now only aborted via a channel passed in at download start time. - The downloaded file is now removed if verification fails. - The download filename is passed around instead of using a *os.File which needs to be re-seeked back to the start of file all the time. - Removed Downloader most tests which were actually functionality which is already being tested on Download.
Previously the calls waiting for a download to be finish could be aborted but the downloads themselves would continue.
Do't annotate to avoid unnecessary information in the error.
This can be used to clear out any temporary download files that may be hanging around in a unit's charm download directory.
The uniter will now clear any temporary charm download files for itself when it starts. Although this shouldn't happen now, previous versions of Juju could leave a signiciant amount of data behind from failed download attempts. Fixes https://bugs.launchpad.net/juju/+bug/1626304
howbazaar
reviewed
Sep 26, 2016
| status := <-dl.Done() | ||
| c.Assert(status.Err, jc.ErrorIsNil) | ||
|
|
||
| filename, err := dl.Wait() |
| @@ -71,11 +70,12 @@ func (d *BundlesDir) download(info BundleInfo, target string, abort <-chan struc | |||
| expectedSha256, err := info.ArchiveSha256() | |||
| req := downloader.Request{ | |||
| URL: curl, | |||
| TargetDir: d.downloadsPath(), | |||
| TargetDir: downloadsPath(d.path), // XXX check this | |||
There was a problem hiding this comment.
To remind me to check that this was definitely correct when QAing (which I did). Removing the XXX now.
| // downloadsPath returns the path to the directory into which charms are | ||
| // downloaded. | ||
| func (d *BundlesDir) downloadsPath() string { | ||
| return path.Join(d.path, "downloads") | ||
| func downloadsPath(bunsDir string) string { |
There was a problem hiding this comment.
bunsDir? Why not just 'dir' ? Or 'base' ?
There was a problem hiding this comment.
To make it clear that it's supposed to receive the bundles directory. bunsDir is already used throughout this file so I was being consistent.
This was actioned.
howbazaar
approved these changes
Sep 26, 2016
|
|
|
Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes https://bugs.launchpad.net/juju/+bug/1626304 for 2.0. It begins with significant refactoring of the downloads package, removing unused functionality and unnecessary complexity. This also ensure that temporary download files are always removed if downloads or download verification fails.
The second part of the PR changes the uniter worker to remove its temporary downloads directory in case files from previous failed download attempts have been left behind.
QA
Deploy a charm while running
while true; do echofind /var/lib/juju/agents/; sleep 0.05; doneon the machine. With this it was possible to see the charm's download files appear in the correct location and then moved to the correct final location.A large bundle (openstack-base) was deployed and logs were checked to ensure that charm downloads succeeded.
A unit's charm downloads directory was populated with files and the agent was restarted. The downloads directory was removed on startup.