Skip to content
Writing large ZIP archives without memory inflation
Python
Branch: master
Clone or download

Latest commit

Latest commit c393053 May 25, 2020

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dist new dist May 25, 2020
examples BufferPredictionSize; example with linux xenial May 24, 2020
zipfly.egg-info Package info May 25, 2020
zipfly update version; init version May 25, 2020
.travis.yml BufferPredictionSize; xenial May 24, 2020
LICENSE Create LICENSE May 16, 2020
README.md README.md May 25, 2020
setup.cfg readme Mar 24, 2020
setup.py update version May 25, 2020

README.md

Build Status GitHub release (latest by date) Downloads

Buzon - ZipFly

ZipFly is a zip archive generator based on zipfile.py. It was created by Buzon.io to generate very large ZIP archives for immediate sending out to clients, or for writing large ZIP archives without memory inflation.

Requirements

Python 3.6+

Install

pip3 install zipfly

Basic usage, compress on-the-fly during writes

Basic use case is compressing on the fly. Some data will be buffered by the zipfile deflater, but memory inflation is going to be very constrained. Data will be written to destination at fairly regular intervals.

ZipFly defaults arguments:

  • paths: [ ]
  • mode: (write) w
  • chunksize: (bytes) 16384
  • compression: Stored
  • allowZip64: True
  • compresslevel: None
  • storesize: (bytes) 0

paths list of dictionaries:

.
fs Should be the path to a file on the filesystem
n (Optional) Is the name which it will have within the archive
(by default, this will be the same as fs)

    import zipfly

    paths = [
        {
            'fs': '/path/to/large/file'
        },
    ]

    zfly = zipfly.ZipFly( paths = paths )

    generator = zfly.generator()
    print (generator)
    # <generator object ZipFly.generator at 0x7f74d52bcc50>


    with open("large.zip", "wb") as f:
        for i in generator:
            f.write(i)

Examples

Streaming multiple files in a zip with Django or Flask Send forth large files to clients with the most popular frameworks

Create paths Easy way to create the array paths from a parent folder.

Predict the size of the zip file before creating it Use the BufferPredictionSize to compute the correct size of the resulting archive before creating it.

Streaming a large file Efficient way to read a single very large binary file in python

Set a comment Your own comment in the zip file

Maintainer

Santiago Debus (@santiagodebus.com)

Santiago's open-source projects are supported by his Patreon. If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.

License

This library was created by Buzon.io and is released under the MIT. Copyright 2020 Grow HQ, Inc.

You can’t perform that action at this time.