Split and compress large files with multi-threading in parallel
- uses multi-threading
- can output a Makefile for manual editing
- allows to stop and resume a job
- doesn't require parallel(1)
parallel: Warning: --blocksize >= 64K causes problems on Cygwin.
- split
cat "$FILENAME" | split -d -a3 --bytes="$CHUNKSIZE" --filter='gzip > "$FILENAME".gz' - "$FILENAME."
- GNU parallel
parallel --pipepart -a "$FILENAME" --block "$CHUNKSIZE" 'gzip > $(({#}-1)).gz'
parallel --pipepart -a "$FILENAME" --block "$CHUNKSIZE" '[[ $(({#}-1)) -ge "$CHUNKSTART" ]] && [[ $(({#}-1)) -le "$CHUNKEND" ]] && gzip > $(({#}-1)).gz'
Usage: chunker [OPTION]... FILE TARGET_DIR CHUNK_SIZE
Split FILE in CHUNK_SIZE byte chunks compressing chunks with gzip in parallel.
--dry-run just output the makefile
--start=n start processing at chunk number n
--end=n end processing at chunk number n
--help display this help and exit
CHUNK_SIZE must be an integer with an optional KB, MB or GB suffix (powers of 1000)
Example:
chunker.sh "SYSTEM-bak01.VHD" "./parts" 500MB
echo " --compress=none don't compress"
compress
gzip, 7z, bzip2
single threaded
- specify number of threads
- make load limit --load-average
full support for dd/split unit suffixes for chunk_size
alternative: add makefile output with dd to split
This software is available under the following licenses:
- Apache 2.0