Skip to content
Fastzip is an opinionated Zip archiver and extractor with a focus on speed.
Go
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
internal
.travis.yml
LICENSE
README.md
archiver.go
archiver_options.go
archiver_test.go
archiver_unix.go
archiver_windows.go
extractor.go
extractor_options.go
extractor_test.go
extractor_unix.go
extractor_windows.go
go.mod
go.sum
register.go
util.go

README.md

fastzip

godoc Build Status

Fastzip is an opinionated Zip archiver and extractor with a focus on speed.

  • Archiving and extraction of files and directories can only occur within a specified directory.
  • Permissions, ownership (uid, gid on linux/unix) and modification times are preserved.
  • Buffers used for copying files are recycled to reduce allocations.
  • Files are archived and extracted concurrently.
  • By default, the excellent github.com/klauspost/compress/flate library is used for compression and decompression.

Example

Archiver

// Create archive file
w, err := os.Create("archive.zip")
if err != nil {
  panic(err)
}
defer w.Close()

// Create new Archiver
a, err := fastzip.NewArchiver(w, "~/fastzip-archiving")
if err != nil {
  panic(err)
}
defer a.Close()

// Register a non-default level compressor if required
// a.RegisterCompressor(zip.Deflate, fastzip.FlateCompressor(1))

// Walk directory, adding the files we want to add
files := make(map[string]os.FileInfo)
err = filepath.Walk("~/fastzip-archiving", func(pathname string, info os.FileInfo, err error) error {
	files[pathname] = info
	return nil
})

// Archive
if err = a.Archive(files); err != nil {
  panic(err)
}

Extractor

// Create new extractor
e, err := fastzip.NewExtractor("archive.zip", "~/fastzip-extraction")
if err != nil {
  panic(err)
}
defer e.Close()

// Extract archive files
if err = e.Extract(); err != nil {
  panic(err)
}

Benchmarks

Archiving and extracting a Go 1.13 GOROOT directory, 342M, 10308 files.

StandardFlate is using compress/flate, NonStandardFlate is klauspost/compress/flate, both on level 5. This was performed on a server with an SSD and 24-cores. Each test was conducted using the WithArchiverConcurrency and WithExtractorConcurrency options of 1, 2, 4, 8 and 16.

$ go test -bench Benchmark* -archivedir go1.13 -benchtime=30s -timeout=20m

goos: linux
goarch: amd64
pkg: github.com/saracen/fastzip
BenchmarkArchiveStore_1-24                            57         759214314 ns/op         437.98 MB/s     9402776 B/op     266273 allocs/op
BenchmarkArchiveStandardFlate_1-24                     2        16635779058 ns/op         19.99 MB/s    11278584 B/op     257235 allocs/op
BenchmarkArchiveStandardFlate_2-24                     4        8958114992 ns/op          37.12 MB/s    15874110 B/op     260757 allocs/op
BenchmarkArchiveStandardFlate_4-24                     7        4513852744 ns/op          73.67 MB/s    18144582 B/op     260842 allocs/op
BenchmarkArchiveStandardFlate_8-24                    14        2337987195 ns/op         142.23 MB/s    22054485 B/op     260969 allocs/op
BenchmarkArchiveStandardFlate_16-24                   15        2112499873 ns/op         157.41 MB/s    27918230 B/op     261194 allocs/op
BenchmarkArchiveNonStandardFlate_1-24                  5        6313696025 ns/op          52.67 MB/s    15156160 B/op     257217 allocs/op
BenchmarkArchiveNonStandardFlate_2-24                  9        3741371342 ns/op          88.88 MB/s    30771747 B/op     261645 allocs/op
BenchmarkArchiveNonStandardFlate_4-24                 18        1906735146 ns/op         174.39 MB/s    35048840 B/op     261690 allocs/op
BenchmarkArchiveNonStandardFlate_8-24                 33        1046698073 ns/op         317.69 MB/s    49022688 B/op     261807 allocs/op
BenchmarkArchiveNonStandardFlate_16-24                36         889191472 ns/op         373.96 MB/s    75247073 B/op     262044 allocs/op
BenchmarkExtractStore_1-24                            24        1491304034 ns/op         220.91 MB/s    22625127 B/op     330125 allocs/op
BenchmarkExtractStore_2-24                            43         717933301 ns/op         458.88 MB/s    22280720 B/op     329243 allocs/op
BenchmarkExtractStore_4-24                            98         362879118 ns/op         907.86 MB/s    22231260 B/op     328635 allocs/op
BenchmarkExtractStore_8-24                           174         207128645 ns/op        1590.53 MB/s    22369458 B/op     328445 allocs/op
BenchmarkExtractStore_16-24                          232         153516463 ns/op        2145.99 MB/s    22326276 B/op     328391 allocs/op
BenchmarkExtractStandardFlate_1-24                     6        5134276149 ns/op          25.20 MB/s    92484254 B/op     525571 allocs/op
BenchmarkExtractStandardFlate_2-24                    12        2735566653 ns/op          47.29 MB/s    91496545 B/op     521703 allocs/op
BenchmarkExtractStandardFlate_4-24                    25        1413308343 ns/op          91.53 MB/s    91636712 B/op     519786 allocs/op
BenchmarkExtractStandardFlate_8-24                    38         917288285 ns/op         141.02 MB/s    93474913 B/op     519425 allocs/op
BenchmarkExtractStandardFlate_16-24                   48         628763448 ns/op         205.74 MB/s    95599041 B/op     519594 allocs/op
BenchmarkExtractNonStandardFlate_1-24                  6        5173029578 ns/op          25.01 MB/s    64130228 B/op     352198 allocs/op
BenchmarkExtractNonStandardFlate_2-24                 13        2685023282 ns/op          48.18 MB/s    63327702 B/op     353975 allocs/op
BenchmarkExtractNonStandardFlate_4-24                 25        1460385105 ns/op          88.58 MB/s    63177508 B/op     354165 allocs/op
BenchmarkExtractNonStandardFlate_8-24                 36         901423814 ns/op         143.51 MB/s    64567442 B/op     357130 allocs/op
BenchmarkExtractNonStandardFlate_16-24                57         635525487 ns/op         203.55 MB/s    65348558 B/op     360551 allocs/op

You can’t perform that action at this time.