The script dies with "Cannot allocate memory" error on a relatively large S3 bucket

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start the sync on a relatively large bucket (in my case that was about 10 
million files)
2. After some time, the script simply dies:

ubuntu@ip-10-86-150-235:~/bin$ PYTHONUNBUFFERED=1 
./2011-10-14-multi_cloud_mirror.py --debug 1 's3://mybucket->cf://mybucket' > 
/mnt/tmp/s3-to-cf.log
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 302, in _handle_workers
    pool._maintain_pool()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 206, in _maintain_pool
    self._repopulate_pool()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 199, in _repopulate_pool
    w.start()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 120, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory 

It works fine on smaller buckets.

Apparently the script tries to built the complete sync scenario before actually 
doing anything - not really how a sync tool should work.

Original issue reported on code.google.com by m...@ilyasemenov.com on 23 Aug 2012 at 5:48