Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make downloading more robust #22

Merged
merged 2 commits into from Sep 20, 2016
Merged

Make downloading more robust #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2958838
Retry on all curl errors, not just timeouts
Sep 19, 2016
469182f
Synchronise downloading of files
Sep 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 source/backends/debian/debpkg.d
View file
Open in desktop
@@ -62,7 +62,9 @@ public:
@property string filename () const {
if (debFname.isRemote) {
immutable path = buildNormalizedPath (tmpDir, debFname.baseName);
downloadFile (debFname, path);
synchronized (this) {
downloadFile (debFname, path);
}
return path;
}
return debFname;
22 source/backends/debian/debpkgindex.d
View file
Open in desktop
@@ -74,15 +74,17 @@ public:
bool[string] ret;

try {
const inReleaseContents = getFileContents (inRelease);
synchronized (this) {
const inReleaseContents = getFileContents (inRelease);

foreach (const ref entry; inReleaseContents) {
auto match = entry.matchFirst (translationregex);
foreach (const ref entry; inReleaseContents) {
auto match = entry.matchFirst (translationregex);

if (match.empty)
continue;
if (match.empty)
continue;

ret[match[1]] = true;
ret[match[1]] = true;
}
}
} catch (Exception ex) {
logWarning ("Could not get %s, will assume 'en' is available.", inRelease);
@@ -112,7 +114,9 @@ public:
"Translation-%s.%s".format(lang, "%s"));

try {
fname = downloadIfNecessary (rootDir, tmpDir, fullPath);
synchronized (this) {
fname = downloadIfNecessary (rootDir, tmpDir, fullPath);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this to only synchronize if we actually download anything? Because otherwise this will slow down things even if the file is already downloaded and present on disk.

Copy link
Collaborator Author

@iainlane iainlane Sep 20, 2016 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meh, hardly; the code basically just returns the filename (after checking if it exists) if it's not remote or if it is downloaded already.

If you insist, then I have to look at the filename in debpkgindex.d which makes it more ugly (currently remote file handling is all taken care of in utility methods).

}
} catch (Exception ex) {
logDebug ("No translations for %s in %s/%s", lang, suite, section);
continue;
@@ -175,7 +179,9 @@ public:
{
immutable path = buildPath ("dists", suite, section, "binary-%s".format (arch));

return downloadIfNecessary (rootDir, tmpDir, buildPath (path, "Packages.%s"));
synchronized (this) {
return downloadIfNecessary (rootDir, tmpDir, buildPath (path, "Packages.%s"));
}
}

protected DebPackage newPackage (string name, string ver, string arch)
4 source/utils.d
View file
Open in desktop
@@ -352,7 +352,7 @@ in { assert (url.isRemote); }
body
{
import core.time;
import std.net.curl : CurlTimeoutException, HTTP, FTP;
import std.net.curl : CurlException, HTTP, FTP;

size_t onReceiveCb (File f, ubyte[] data)
{
@@ -377,7 +377,7 @@ body
downloader.perform();
}
logDebug ("Downloaded %s", url);
} catch (CurlTimeoutException e) {
} catch (CurlException e) {
if (retryCount > 0) {
logDebug ("Failed to download %s, will retry %d more %s",
url,