Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Allow to stream the downloads when appropriate
Some API calls will download possibly large data, resulting in a high memory usage and out-of-memory errors. For these API calls use the requests streaming capabilities and download chunked data. The caller is responsible of providing a callable to actually store the data. The default callable just prints the data on stdout.
- Loading branch information
Gauvain Pocentek
committed
Jul 17, 2016
1 parent
8e6a944
commit 94aea52
Showing
5 changed files
with
75 additions
and
23 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
class _StdoutStream(object): | ||
def __call__(self, chunk): | ||
print(chunk) | ||
|
||
|
||
def response_content(response, streamed, action, chunk_size): | ||
if streamed is False: | ||
return response.content | ||
|
||
if action is None: | ||
action = _StdoutStream() | ||
|
||
for chunk in response.iter_content(chunk_size=chunk_size): | ||
if chunk: | ||
action(chunk) |