New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consuming alot of RAM #135

Open
mohamedjmustafa opened this Issue May 21, 2014 · 15 comments

Comments

Projects
None yet
8 participants
@mohamedjmustafa

mohamedjmustafa commented May 21, 2014

Iam using JSzip to zip a big file (total about 300MB) , i have noticed a huge consumption of RAM about 6 - 6.5 GB , then the program crashed due to memory leak !!!

also when iam trying to zip a small files (les than 5 MB) , it consumes about 350 - 500 MB of RAM !!!

@dduponchel

This comment has been minimized.

Show comment
Hide comment
@dduponchel

dduponchel May 21, 2014

Collaborator

That a lot of memory.
Could you share the code sample using JSZip ? (and the files used if possible).
Also, what is the browser or the version of nodejs you use ?

Collaborator

dduponchel commented May 21, 2014

That a lot of memory.
Could you share the code sample using JSZip ? (and the files used if possible).
Also, what is the browser or the version of nodejs you use ?

@mohamedjmustafa

This comment has been minimized.

Show comment
Hide comment
@mohamedjmustafa

mohamedjmustafa May 21, 2014

iam working on a javascript windows store application
and this is the code :

function zipFolder(foldername) {
zip = new JSZip();
//get all folder and subfolders
var options = new Windows.Storage.Search.QueryOptions(Windows.Storage.Search.CommonFileQuery.defaultQuery, ['*']);
options.folderDepth = Windows.Storage.Search.FolderDepth.deep;
Windows.Storage.KnownFolders.documentsLibrary.getFolderAsync("eduBookData").done(
function (eduBookData) {
eduBookData.getFolderAsync(foldername).done(
function (myFolder) {
//get all file ...
myFolder.createFileQueryWithOptions(options).getFilesAsync().then(
function (files) {
fileindex = 1;
files.forEach(
function (file) {
getFileAsUint8Array(file).done(
function (Filecontent) {
spath = file.path.split("");
folderindex = spath.indexOf(myFolder.name) + 1;
var fpath = spath[folderindex]
for (i = folderindex + 1; i < spath.length ; i++) {
fpath += "" + spath[i] //generate the path of each file inside the zip file
}
zip.file(fpath, Filecontent);
//document.getElementById('mylog').innerHTML += fpath + "
";
if (fileindex == files.length) {
content = zip.generate({ compression: 'DEFLATE', type: 'uint8array' });
var buffer = Windows.Security.Cryptography.CryptographicBuffer.createFromByteArray(content);
eduBookData.createFileAsync(myFolder.name + ".zip",
Windows.Storage.CreationCollisionOption.replaceExisting).done(
function (myFile) {
return Windows.Storage.FileIO.writeBufferAsync(myFile, buffer).then(
function () {
new Windows.UI.Popups.MessageDialog("Writing Done! ", "Zip").showAsync().done();
});
});
}
fileindex++;
});
});
});
});
});
}

function getFileAsUint8Array(file) { //convert the file to uint array
return Windows.Storage.FileIO.readBufferAsync(file).then(function (buffer) { //Read the file into a byte array
var fileContents = new Uint8Array(buffer.length);
var dataReader = Windows.Storage.Streams.DataReader.fromBuffer(buffer);
dataReader.readBytes(fileContents);
dataReader.close();
return fileContents;
});
}

mohamedjmustafa commented May 21, 2014

iam working on a javascript windows store application
and this is the code :

function zipFolder(foldername) {
zip = new JSZip();
//get all folder and subfolders
var options = new Windows.Storage.Search.QueryOptions(Windows.Storage.Search.CommonFileQuery.defaultQuery, ['*']);
options.folderDepth = Windows.Storage.Search.FolderDepth.deep;
Windows.Storage.KnownFolders.documentsLibrary.getFolderAsync("eduBookData").done(
function (eduBookData) {
eduBookData.getFolderAsync(foldername).done(
function (myFolder) {
//get all file ...
myFolder.createFileQueryWithOptions(options).getFilesAsync().then(
function (files) {
fileindex = 1;
files.forEach(
function (file) {
getFileAsUint8Array(file).done(
function (Filecontent) {
spath = file.path.split("");
folderindex = spath.indexOf(myFolder.name) + 1;
var fpath = spath[folderindex]
for (i = folderindex + 1; i < spath.length ; i++) {
fpath += "" + spath[i] //generate the path of each file inside the zip file
}
zip.file(fpath, Filecontent);
//document.getElementById('mylog').innerHTML += fpath + "
";
if (fileindex == files.length) {
content = zip.generate({ compression: 'DEFLATE', type: 'uint8array' });
var buffer = Windows.Security.Cryptography.CryptographicBuffer.createFromByteArray(content);
eduBookData.createFileAsync(myFolder.name + ".zip",
Windows.Storage.CreationCollisionOption.replaceExisting).done(
function (myFile) {
return Windows.Storage.FileIO.writeBufferAsync(myFile, buffer).then(
function () {
new Windows.UI.Popups.MessageDialog("Writing Done! ", "Zip").showAsync().done();
});
});
}
fileindex++;
});
});
});
});
});
}

function getFileAsUint8Array(file) { //convert the file to uint array
return Windows.Storage.FileIO.readBufferAsync(file).then(function (buffer) { //Read the file into a byte array
var fileContents = new Uint8Array(buffer.length);
var dataReader = Windows.Storage.Streams.DataReader.fromBuffer(buffer);
dataReader.readBytes(fileContents);
dataReader.close();
return fileContents;
});
}

@dduponchel

This comment has been minimized.

Show comment
Hide comment
@dduponchel

dduponchel May 21, 2014

Collaborator

I tested with nodejs (just to do a quick test, I will investigate further tonight) with a 300MB random file (as Uint8Array and generating the result like you).
I start with 300MB of memory used and finish with 600MB (half for the original file, half for the generated zip file).
I do get a huge memory consumption (about 3GB) when loading the file in memory and converting it to a Uint8Array (don't know why...). Maybe you have the same issue : could you check the same code with the zip.generate() line commented out ?

Collaborator

dduponchel commented May 21, 2014

I tested with nodejs (just to do a quick test, I will investigate further tonight) with a 300MB random file (as Uint8Array and generating the result like you).
I start with 300MB of memory used and finish with 600MB (half for the original file, half for the generated zip file).
I do get a huge memory consumption (about 3GB) when loading the file in memory and converting it to a Uint8Array (don't know why...). Maybe you have the same issue : could you check the same code with the zip.generate() line commented out ?

@mohamedjmustafa

This comment has been minimized.

Show comment
Hide comment
@mohamedjmustafa

mohamedjmustafa May 21, 2014

i have commented it , no RAm consumption ...
the consumption happens when generating the zip file
zip.generate()

mohamedjmustafa commented May 21, 2014

i have commented it , no RAm consumption ...
the consumption happens when generating the zip file
zip.generate()

@jamshid

This comment has been minimized.

Show comment
Hide comment
@jamshid

jamshid May 21, 2014

Hi, I'm also interested in using this library to generate a zip file dynamically that is downloaded by the user. Sorry but it's not clear from documentation -- is zip.generate() supposed to be usable (at least in modern non-IE browsers) for large zip files? Maybe http://stuk.github.io/jszip/documentation/howto/write_zip.html could clarify browser memory usage -- i.e. can it stream the contents of the zip as it's generated, or will the zip file always need to be loaded in memory?

jamshid commented May 21, 2014

Hi, I'm also interested in using this library to generate a zip file dynamically that is downloaded by the user. Sorry but it's not clear from documentation -- is zip.generate() supposed to be usable (at least in modern non-IE browsers) for large zip files? Maybe http://stuk.github.io/jszip/documentation/howto/write_zip.html could clarify browser memory usage -- i.e. can it stream the contents of the zip as it's generated, or will the zip file always need to be loaded in memory?

@dduponchel

This comment has been minimized.

Show comment
Hide comment
@dduponchel

dduponchel May 21, 2014

Collaborator

@mohamedjmustafa I tested in IE11 on Windows 8.1 (I think it's the closest I can go without installing Visual Studio in a VM) and with a 300MB file, I get an Out of Memory message from IE at about 700MB after slowly eating memory. I still don't know why JSZip takes 6GB in your case but even 700MB in this case is too much. Your issue mainly comes from JSZip which holds everything in memory and doesn't support (yet !) zip streaming.

@jamshid the current implementation loads everything in memory so no, JSZip isn't suited (yet) for large zip files (hundreds of MB).

I'm working on async and streams features and this should come "soon" (I'm bad at estimating the remaining time but I'm over 80% completion).

Collaborator

dduponchel commented May 21, 2014

@mohamedjmustafa I tested in IE11 on Windows 8.1 (I think it's the closest I can go without installing Visual Studio in a VM) and with a 300MB file, I get an Out of Memory message from IE at about 700MB after slowly eating memory. I still don't know why JSZip takes 6GB in your case but even 700MB in this case is too much. Your issue mainly comes from JSZip which holds everything in memory and doesn't support (yet !) zip streaming.

@jamshid the current implementation loads everything in memory so no, JSZip isn't suited (yet) for large zip files (hundreds of MB).

I'm working on async and streams features and this should come "soon" (I'm bad at estimating the remaining time but I'm over 80% completion).

@jamshid

This comment has been minimized.

Show comment
Hide comment
@jamshid

jamshid May 21, 2014

That's exciting, thanks for your work on this library, async will be awesome.
--Jamshid
On Wednesday, May 21, 2014 4:35 PM, David Duponchel notifications@github.com wrote:

@mohamedjmustafa I tested in IE11 on Windows 8.1 (I think it's the closest I can go without installing Visual Studio in a VM) and with a 300MB file, I get an Out of Memory message from IE at about 700MB after slowly eating memory. I still don't know why JSZip takes 6GB in your case but even 700MB in this case is too much. Your issue mainly comes from JSZip which holds everything in memory and doesn't support (yet !) zip streaming.
@jamshid the current implementation loads everything in memory so no, JSZip isn't suited (yet) for large zip files (hundreds of MB).
I'm working on async and streams features and this should come "soon" (I'm bad at estimating the remaining time but I'm over 80% completion).

Reply to this email directly or view it on GitHub.

jamshid commented May 21, 2014

That's exciting, thanks for your work on this library, async will be awesome.
--Jamshid
On Wednesday, May 21, 2014 4:35 PM, David Duponchel notifications@github.com wrote:

@mohamedjmustafa I tested in IE11 on Windows 8.1 (I think it's the closest I can go without installing Visual Studio in a VM) and with a 300MB file, I get an Out of Memory message from IE at about 700MB after slowly eating memory. I still don't know why JSZip takes 6GB in your case but even 700MB in this case is too much. Your issue mainly comes from JSZip which holds everything in memory and doesn't support (yet !) zip streaming.
@jamshid the current implementation loads everything in memory so no, JSZip isn't suited (yet) for large zip files (hundreds of MB).
I'm working on async and streams features and this should come "soon" (I'm bad at estimating the remaining time but I'm over 80% completion).

Reply to this email directly or view it on GitHub.

@mohamedjmustafa

This comment has been minimized.

Show comment
Hide comment
@mohamedjmustafa

mohamedjmustafa May 22, 2014

you get out of memory message because you are using a web browser which allow a limited RAM consumption , while the application have no limit in RAM , it can use whole the free memory until it filled totally , it will give an out of memory message and crashes

note : it also takes too much time to zip a big files

any way thank you for replying to this issue and i wish to try to fix it as soon as you can :)

mohamedjmustafa commented May 22, 2014

you get out of memory message because you are using a web browser which allow a limited RAM consumption , while the application have no limit in RAM , it can use whole the free memory until it filled totally , it will give an out of memory message and crashes

note : it also takes too much time to zip a big files

any way thank you for replying to this issue and i wish to try to fix it as soon as you can :)

@davidnormo

This comment has been minimized.

Show comment
Hide comment
@davidnormo

davidnormo Jun 17, 2014

@dduponchel any update on the streams front?

davidnormo commented Jun 17, 2014

@dduponchel any update on the streams front?

@dduponchel

This comment has been minimized.

Show comment
Hide comment
@dduponchel

dduponchel Jun 18, 2014

Collaborator

@davidnormo Not a lot, unfortunately.

Collaborator

dduponchel commented Jun 18, 2014

@davidnormo Not a lot, unfortunately.

@davidnormo

This comment has been minimized.

Show comment
Hide comment
@davidnormo

davidnormo Jun 18, 2014

That's a pity. I'm in the process of switching to zip.js because of handling larger files. JSZip has been great otherwise!

davidnormo commented Jun 18, 2014

That's a pity. I'm in the process of switching to zip.js because of handling larger files. JSZip has been great otherwise!

@amita-trantor

This comment has been minimized.

Show comment
Hide comment
@amita-trantor

amita-trantor Jun 17, 2016

I am also facing out of memory issue with jszip. Any update on this ?

amita-trantor commented Jun 17, 2016

I am also facing out of memory issue with jszip. Any update on this ?

@alvinnguyen

This comment has been minimized.

Show comment
Hide comment
@alvinnguyen

alvinnguyen Jul 11, 2016

Any updates on this front? I'm using JSZip for Chrome Extension and because of the memory usage, it constantly crashes on large file.

alvinnguyen commented Jul 11, 2016

Any updates on this front? I'm using JSZip for Chrome Extension and because of the memory usage, it constantly crashes on large file.

@SavageCore

This comment has been minimized.

Show comment
Hide comment
@SavageCore

SavageCore Oct 7, 2016

I'm working with node.js (electron) and creating a large zip 1gb+ using generateNodeStream, should memory be released once zip file is written? or have I made a mistake? I tried setting zip to null on finish

SavageCore commented Oct 7, 2016

I'm working with node.js (electron) and creating a large zip 1gb+ using generateNodeStream, should memory be released once zip file is written? or have I made a mistake? I tried setting zip to null on finish

@jimmywarting

This comment has been minimized.

Show comment
Hide comment
@jimmywarting

jimmywarting Oct 7, 2016

Contributor

There are some tricks you can make to reduce the memory
This is mostly focused on how you would do it in a modern browser...

This is all focused on streams

  • Adding all entries as a stream so data gets requested on demand
  • getting zip output as stream (generateInternalStream) for piping it to a destination instead of holding of the entiere result in memory
  • add last streamFiles option to avoid process the hole entry
const stream = require('stream')
// see https://github.com/jimmywarting/StreamSaver.js
const writeStream = streamSaver.createWriteStream('output.zip').getWriter()

// Adding everything as a stream like this means that we
// Don't have any data to complete the zip file necessary
// instead jsZip is going to pull data from a node stream
// when it needs more data - so we give it on demand instead
for (let url of urls) {
  let reader, p
  let rs = stream.Readable()

  // On the first read we open up a ReadableStream with the new fetch api
  //   this is only avalible in blink but can be adobted with ms-stream 
  //   and moz-chunked-array
  //   https://github.com/jhiesey/stream-http is a good option...
  //   
  // Doing it like this means we only going to have one open ajax request at once 
  rs._read = () => {
    let p = reader || (reader = fetch(url, {headers: {range}}).then(res => res.body.getReader()))
    p.then(reader => reader.read().then(({ value, done }) =>
      rs.push(done ? null : new Buffer(value)))
    )
  }

  zip.file(filename, rs)
}

// Using generateInternalStream instead of generateAsync means we can take
// the data and pipe it to a destination instead of waiting for jszip to
// buffer up the hole complete result.
// 
// Without the streamFiles option, jsZip must process the hole file to figure
// out the size and preappend it. with this option a you use a description instead
zip.generateInternalStream({type: 'uint8array', streamFiles: true})
.on('data', data => writeStream.write(data))
.on('error', err => writeStream.error(err))
.on('end', () => writeStream.close())
.resume()

See it in action https://jsfiddle.net/Lxwvco3u/5/

Contributor

jimmywarting commented Oct 7, 2016

There are some tricks you can make to reduce the memory
This is mostly focused on how you would do it in a modern browser...

This is all focused on streams

  • Adding all entries as a stream so data gets requested on demand
  • getting zip output as stream (generateInternalStream) for piping it to a destination instead of holding of the entiere result in memory
  • add last streamFiles option to avoid process the hole entry
const stream = require('stream')
// see https://github.com/jimmywarting/StreamSaver.js
const writeStream = streamSaver.createWriteStream('output.zip').getWriter()

// Adding everything as a stream like this means that we
// Don't have any data to complete the zip file necessary
// instead jsZip is going to pull data from a node stream
// when it needs more data - so we give it on demand instead
for (let url of urls) {
  let reader, p
  let rs = stream.Readable()

  // On the first read we open up a ReadableStream with the new fetch api
  //   this is only avalible in blink but can be adobted with ms-stream 
  //   and moz-chunked-array
  //   https://github.com/jhiesey/stream-http is a good option...
  //   
  // Doing it like this means we only going to have one open ajax request at once 
  rs._read = () => {
    let p = reader || (reader = fetch(url, {headers: {range}}).then(res => res.body.getReader()))
    p.then(reader => reader.read().then(({ value, done }) =>
      rs.push(done ? null : new Buffer(value)))
    )
  }

  zip.file(filename, rs)
}

// Using generateInternalStream instead of generateAsync means we can take
// the data and pipe it to a destination instead of waiting for jszip to
// buffer up the hole complete result.
// 
// Without the streamFiles option, jsZip must process the hole file to figure
// out the size and preappend it. with this option a you use a description instead
zip.generateInternalStream({type: 'uint8array', streamFiles: true})
.on('data', data => writeStream.write(data))
.on('error', err => writeStream.error(err))
.on('end', () => writeStream.close())
.resume()

See it in action https://jsfiddle.net/Lxwvco3u/5/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment