Skip to content
This repository has been archived by the owner on Apr 8, 2021. It is now read-only.

Commit

Permalink
topmodel: amend/add comments about chunk size
Browse files Browse the repository at this point in the history
  • Loading branch information
matt-stripe committed Mar 10, 2016
1 parent bf9e9fd commit 856e7a8
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions topmodel/file_system.py
Expand Up @@ -58,14 +58,15 @@ def write_file(self, path, data):
if key is None:
key = self.bucket.new_key(self.subdirectory + path)

# Take the data (a giant string of JSON), and turn it into a file
# stream we can read from
# Take the data (a giant string), and turn it into a file IO stream
# we can read from
f = StringIO.StringIO()
f.write(data)
f.seek(0)

# Count the number of chunks we're going to have to upload
size = len(data)
# By default, set the chunk size to 50MB
chunk_size = 1024 * 1024 * 50
chunks = int(math.ceil(size/float(chunk_size)))

Expand Down

0 comments on commit 856e7a8

Please sign in to comment.