-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support retrying downloads if they transiently fail #1594
Closed
Closed
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
73226ae
lib/repo-pull: Support retrying requests on transient network errors
pwithnall 5b78601
lib/fetcher-soup: Map more SoupStatus codes to known GIOErrors
pwithnall 63b1ba4
ostree/trivial-httpd: Add --random-408s command line option
pwithnall 30b6c08
tests: Test pull behaviour when network timeouts occur
pwithnall 3c51ee4
fixup! lib/repo-pull: Support retrying requests on transient network …
pwithnall d8ad578
lib/repo-pull: Add some missing assertions for progress statistics
pwithnall ba8a265
lib/fetcher: Factor out HTTP status code handling from soup and curl
pwithnall e283a05
fixup! lib/fetcher: Factor out HTTP status code handling from soup an…
pwithnall File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1070,19 +1070,13 @@ on_request_sent (GObject *object, | |
soup_uri_to_string (soup_request_get_uri (pending->request), FALSE); | ||
|
||
GIOErrorEnum code; | ||
|
||
switch (msg->status_code) | ||
{ | ||
case SOUP_STATUS_NOT_FOUND: | ||
case SOUP_STATUS_FORBIDDEN: | ||
case SOUP_STATUS_GONE: | ||
code = G_IO_ERROR_NOT_FOUND; | ||
break; | ||
/* These statuses are internal to libsoup, and not standard HTTP ones: */ | ||
case SOUP_STATUS_CANCELLED: | ||
code = G_IO_ERROR_CANCELLED; | ||
break; | ||
case SOUP_STATUS_REQUEST_TIMEOUT: | ||
code = G_IO_ERROR_TIMED_OUT; | ||
break; | ||
case SOUP_STATUS_CANT_RESOLVE: | ||
case SOUP_STATUS_CANT_CONNECT: | ||
code = G_IO_ERROR_HOST_NOT_FOUND; | ||
|
@@ -1092,10 +1086,11 @@ on_request_sent (GObject *object, | |
code = G_IO_ERROR_BROKEN_PIPE; | ||
#else | ||
code = G_IO_ERROR_CONNECTION_CLOSED; | ||
#endif | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Did you mean to move this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. argh, no, the problems of squashing two diffs |
||
break; | ||
#endif | ||
default: | ||
code = G_IO_ERROR_FAILED; | ||
code = _ostree_fetcher_http_status_code_to_io_error (msg->status_code); | ||
break; | ||
} | ||
|
||
{ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing that feels odd to me is that the way this is structured, we'll be retrying the whole set of mirrors if the last one on the list returns a transient error. So e.g. the first N-1 mirrors could return
NOT_FOUND
and if the Nth mirror times out, we'll be retrying all N mirrors, right? Whereas I think what we want is to retry only the Nth mirror. It seems like fixing this properly would probably require indeed teaching each backend-specific fetcher about retries.Anyway, I think it's definitely worth getting the current approach in and iterate on that since AFAIK mirrorlists are not widely used yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’d definitely prefer to get the current approach in first. That said, I’m not sure whether I will have time afterwards to work on improvements to mirror handling (I’m working on this to fix a problem with our Endless downloads timing out, and mirrors don’t really impact on that).
However, I’m not entirely convinced about the situation you’re considering. Are we ever expecting the majority of mirrors to return
NOT_FOUND
? Sounds like an unlikely corner case to me. Even so,OstreeFetcherSoup
maintains state about its mirrors, so I think the problem could be addressed by storing a bit more state there, rather than by rearchitecting things. Specifically, instead of storing a singlemirrorlist_idx
pointing to the best mirror to try, store some flags for each mirror, and track whether each mirror has returnedNOT_FOUND
, or returned a transient error, at all. Prioritise requesting from mirrors which haven’t. (This could also be implemented by storing a priority for each mirror, and demoting it onNOT_FOUND
or transient errors.)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify, I wasn't suggesting you would fix those up. Just something that'd need to be done down the road. As I said, I'm not aware of any big users of mirrorlists yet, so it's not a pressing concern.
Heh, sure. The
NOT_FOUND
was an example; anything "non-transient" would work. Anyway, I do agree with you that it could be done without lowering the retry logic down a level.