Use Soup for downloads and set timeouts #59
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
July 12, 2013 09:38
Downloading via GIO uses gvfs internally. Unfortunately gvfs does not set or handle connection timeouts, meaning that bad connectivity can lead to a total hang. Fixing gvfs is not easy: https://mail.gnome.org/archives/gvfs-list/2013-July/msg00000.html And while looking at the bug, I realised just how much overkill this is. gvfs runs in a separate process and does the HTTP communication, communicating back to gio over dbus, and has a few layers of abstraction. The gio developers on IRC recommended just to use libsoup directly, which is not hard. And we can set timeouts appropriately in order to handle bad connectivity. This required some tweaks to the internal Downloader API - we can now handle cancellation much more directly.
Re-use the general downloader class for downloading activity metadata, which means that interrupted connections will now error and time out correctly. This involved adding a "download to memory" interface to the Downloader class, which has a new unit test.
Re-use the general downloader class for downloading activity metadata, which means that interrupted connections will now error and time out correctly. This involved adding a get_size() request to the downloader, and a got-chunk signal for users that don't require the entire download presented as a single buffer.
Increase code re-use by using the Downloader class instead of urllib2. The advantage here is that Soup keeps around connections for a while, so instead of re-creating TCP streams for each partial read, we just re-use the existing ones. This results in a noticable speedup when doing metadata lookups. Another advantage is that we can run the main loop while we are waiting for a response. This means that there is no need to create a separate thread for metadata lookups.
Closing the activity updater when the list of updates was visible, then reopening it again, resulted in an exception. Handle this cancellation case properly.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Downloading via GIO uses gvfs internally. Unfortunately gvfs does not
set or handle connection timeouts, meaning that bad connectivity can
lead to a total hang.
Fixing gvfs is not easy:
https://mail.gnome.org/archives/gvfs-list/2013-July/msg00000.html
And while looking at the bug, I realised just how much overkill this
is. gvfs runs in a separate process and does the HTTP communication,
communicating back to gio over dbus, and has a few layers of abstraction.
The gio developers on IRC recommended just to use libsoup directly,
which is not hard. And we can set timeouts appropriately in order to
handle bad connectivity.
replaces #57