dbrestore runs out of memory on large compressed backups

[MattFisher]

Trying to restore a db backup that's about 200MB compressed and running out of memory.
Is it possible to modfy utils.uncompress_file to avoid reading the whole file into memory, like http://stackoverflow.com/questions/339053/how-do-you-unzip-very-large-files-in-python ?

Stack trace:

Reading file xxx-xxx-2015-10-25-180000.psql.gz
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
    return original_func(self, *args, **kwargs)
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/dbbackup/management/commands/dbrestore.py", line 58, in handle
    self.restore_backup()
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/dbbackup/management/commands/dbrestore.py", line 94, in restore_backup
    uncompressed_file, input_filename = utils.uncompress_file(inputfile, input_filename)
  File "/srv/apps/staging/env/local/lib/python2.7/site-packages/dbbackup/utils.py", line 249, in uncompress_file
    outputfile.write(zipfile.read())
  File "/usr/lib/python2.7/gzip.py", line 249, in read
    self._read(readsize)
  File "/usr/lib/python2.7/gzip.py", line 308, in _read
    self._add_read_data( uncompress )
  File "/usr/lib/python2.7/gzip.py", line 326, in _add_read_data
    self.extrabuf = self.extrabuf[offset:] + data
MemoryError

[ZuluPro]

Does compression job have the same trouble ?

[MattFisher]

Finally got around to testing this.
Compression seems to work fine in terms of memory, but I'm having some problems with running out of disk space.
The dbbackup command creates an uncompressed temp file, then compresses it if --compress is set, then encrypts it if --encrypt is set.
In my case the uncompressed file is 3+ GB, and the compressed file is ~400MB, so if I do something like pg_dump mydb | gzip > mydb.psql.gz I can compress it on the fly and save it, but dbbackup runs out of disk space (the server instances only have ~7GB of HDD space each).

[ZuluPro]

Ok, thanks a lot for the troubleshooting,
I study how to improve and make it as soon as possible.

[ZuluPro]

Hello @MattFisher,
Could you test this branch make the job for you: https://github.com/ZuluPro/django-dbbackup/tree/gzip
?

[ZuluPro]

Hello, @MattFisher,
With PR #178 merged you should not have MemoryError anymore (in master future v3).
Code is now supposed to use SpooledTemporaryFile which uses your filesystem if memory isn't big enough.