Skip to content

Commit

Permalink
Stream zipfile contents when unpacking
Browse files Browse the repository at this point in the history
Some of these things are 10GB+. Takes forever to read.
  • Loading branch information
Mike Graves committed May 11, 2017
1 parent 8e2d9d8 commit e1ed590
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions slingshot/app.py
Expand Up @@ -33,8 +33,12 @@ def unpack_zip(source, destination):
raise "Only shapefiles are currently supported"
for f in [m for m in zf.namelist() if not m.endswith('/')]:
f_dest = os.path.join(destination, os.path.basename(f))
with open(f_dest, 'wb') as fp:
fp.write(zf.read(f))
with open(f_dest, 'wb') as fp, zf.open(f) as zp:
while True:
chunk = zp.read(8192)
if not chunk:
break
fp.write(chunk)


def create_record(bag, public, secure, **kwargs):
Expand Down

0 comments on commit e1ed590

Please sign in to comment.