Added noDisk option that keeps each part's content in memory instead of in a temp file #6

Closed
wants to merge 4 commits into from

4 participants

@Zugwalt

Platform as a service offerings such as nodejitsu either forbid or limit disk writes. With the current implementation I was unable to large amounts of text (giant generated CSV files) to S3 as the application would throw errors.

I've added a noDisk option which when set to true keeps the data for each part in-memory instead of writing to a temporary file. It defaults to false.

Also added trivial change of including status code when an error is thrown because response code is not 200, as this was helpful for debugging.

Aaron Silverman added some commits Apr 6, 2013
Aaron Silverman provide status code when upload fails--helpful for debugging 71b5127
Aaron Silverman ability to optionally keep parts in-memory instead of writing to temp…
… files. Useful for PaaS offerings that don't let you write much (or anything) to disk
e9967df
@shahriman

@Zugwalt I have a different solution which might be relevant:
shahriman/knox-mpu@530c06f

@mikermcneil
Collaborator

+1

This would solve the lingering ENOENT errors that pop up

@nathanoehlman

Hi @Zugwalt - you make a good case for buffering the file into memory instead of the file - my initial version that I used did this, but I made a deliberate design decision to switch to files instead because I didn't want to have to put a bit of logic along the lines of what @shahriman proposed at that stage in order to prevent people accidentally crashing their servers.

I intend on refactoring your commit a little in the future to use an in memory stream (so that the file and memory streams are easily interchangeable), as well as take into account @shahriman's changes - but seeing as how I'm a little short strapped for time at the moment, I'll just pull yours in for the moment, and refactor later on.

Thanks :)

@Zugwalt

Awesome Nathan! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment