Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream ZIP exports to S3 #448

Closed
jnm opened this issue May 22, 2018 · 3 comments
Closed

Stream ZIP exports to S3 #448

jnm opened this issue May 22, 2018 · 3 comments
Assignees

Comments

@jnm
Copy link
Member

jnm commented May 22, 2018

Right now, the ZIP export process looks like:

  1. Download every single media attachment from S3;
  2. Add it to a ZIP file in /tmp/;
  3. Upload the ZIP file to S3;
  4. Return the result to the user.

These temporary ZIP files are often exhausting our servers' disk space and causing many wasted hours. It's time to stop the madness.

@jnm jnm self-assigned this May 22, 2018
@jnm
Copy link
Member Author

jnm commented May 30, 2018

In our sights:

with NamedTemporaryFile('wb+', prefix='media_zip_export_', suffix='.zip') as temporary_file:
create_attachments_zipfile(attachments, temporary_file=temporary_file)
export_filename = get_storage_class()().save(
file_path,
File(temporary_file, file_path))

@jnm
Copy link
Member Author

jnm commented May 30, 2018

I think I'm hitting this again: jschneier/django-storages#449

@jnm
Copy link
Member Author

jnm commented May 31, 2018

Will need to use my fork until jschneier/django-storages#504 is merged.

jnm added a commit that referenced this issue May 31, 2018
without using temporary files. Use a custom fork of django-storages that
properly supports multipart uploads. Increase the default S3 buffer size to 50
megabytes. Fixes #448.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant