Skip to content

jbd/packo

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Please consider using fpart from martymac. Inspired by packo, fpart is a much more powerful tool, written in C.

Packo

packo is a simple program from a very specific need : I needed to transfer multiple terabytes of data using rsync. In my environment (file server, connectivity, etc...) I can use multiple rsync in parallel to improve the throughput but didn't want to think about what data each rsync should transfer, that's why i wrote this program. it may be of interest to you.

It gathers file size information recursively given a path and splits the whole list in sets of approximatively equal size using a greedy algorithm with a hack. Oups, i should say "heuristic".

By default, the program just output what it can do. For example, i want to build 5 sets of files from /usr/share/ :

$ python packo.py /usr/share/doc 5
11202/11202, 220.7 MBytes (Memsize: 1.1 MBytes)

Pack 0: 44.1 MBytes / 2239 files
Pack 1: 44.1 MBytes / 2240 files
Pack 2: 44.1 MBytes / 2239 files
Pack 3: 44.1 MBytes / 2240 files
Pack 4: 44.1 MBytes / 2244 files

You can see that there are 11202 files here and the process requires approximatively 1.1 MBytes of memory to run. I print this information because my implementation is simple : i'm building the full files list in memory. It shouldn't be a concern for most usage. In this example, each set of files is about 44 MBytes.

If a want to write the list of files, i just add a filename name :

$ python packo.py /usr/share/doc 5 mylistoffiles
11202/11202, 220.7 MBytes (Memsize: 1.1 MBytes)

Pack 0: 44.1 MBytes / 2239 files
Pack 1: 44.1 MBytes / 2240 files
Pack 2: 44.1 MBytes / 2239 files
Pack 3: 44.1 MBytes / 2240 files
Pack 4: 44.1 MBytes / 2244 files

It will write 5 files that can be used with the rsync '--files-from' option :

$ head  mylistoffiles*
==> mylistoffiles0 <==
/usr/share/doc/ipython/manual/ipython.pdf.gz
/usr/share/doc/gir1.2-gtk-3.0/changelog.gz
...

==> mylistoffiles1 <==
/usr/share/doc/chromium-inspector/copyright
/usr/share/doc/libgtk-3-0/changelog.gz
...

==> mylistoffiles2 <==
/usr/share/doc/chromium/copyright
/usr/share/doc/libgtk-3-bin/changelog.gz
...

==> mylistoffiles3 <==
/usr/share/doc/chromium-browser/copyright
/usr/share/doc/libgail-3-0/changelog.gz
...

==> mylistoffiles4 <==
/usr/share/doc/xserver-common/changelog.gz
/usr/share/doc/python2.6/html/genindex-all.html
...

If you try to relaunch the command line, the program will exit to prevent overwriting :

$ python packo.py /usr/share/doc 5 mylistoffiles
'mylistoffiles0' already exists, aborting.

Nothing magic or very interesting here :)

Just for fun :

$ python packo.py /usr 42 out
112950/112950, 3.7 Gbytes (Memsize: 12.0 MBytes)

Pack 0: 134.2 MBytes / 1 files
Pack 1: 88.0 MBytes / 2592 files
Pack 2: 88.0 MBytes / 2612 files
Pack 3: 88.0 MBytes / 2655 files
Pack 4: 88.0 MBytes / 2655 files
Pack 5: 88.0 MBytes / 2655 files
Pack 6: 88.0 MBytes / 2655 files
Pack 7: 88.0 MBytes / 2655 files
Pack 8: 88.0 MBytes / 6802 files
...
Pack 41: 88.0 MBytes / 2657 files

$ cat out0
/usr/share/icons/oxygen/icon-theme.cache
$ ls -h /usr/share/icons/oxygen/icon-theme.cache
$ ls -lh /usr/share/icons/oxygen/icon-theme.cache
-rw-r--r-- 1 root root 135M Sep 30 20:54 /usr/share/icons/oxygen/icon-theme.cache

Dependencies

python>=2.6 (including python 3)

About

Group files in sets of same total size

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages