Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.OutOfMemoryException when tried with 70 files #412

Closed
surensaluka opened this issue Jan 13, 2020 · 3 comments
Closed

System.OutOfMemoryException when tried with 70 files #412

surensaluka opened this issue Jan 13, 2020 · 3 comments

Comments

@surensaluka
Copy link

surensaluka commented Jan 13, 2020

Steps to reproduce

  1. Use 70 different pdf files which are in various sizes. (Combined size is 165MB)
  2. Try looping through all 70 files adding to ZipOutputStream
  3. Throw OutOfMemoryException
  4. Use my sample code
public byte[] DownloadPaperList()
{
    try
    {
        PaperDAO paperDAO = new PaperDAO();
        List<PaperModel> papers = paperDAO.GetPaperListByPaperIds(paperIDs);
        using (MemoryStream outputMemoryStream = new MemoryStream())
        {
            using (var zipOutputStream = new ZipOutputStream(outputMemoryStream))
            {
                papers = papers.OrderBy(x => x.Order).ToList();
                foreach (PaperModel paper in papers)
                {
                    byte[] decryptedPaper = CryptoServices.DecryptFile(
                        paper.FileData, 
                        paperDAO.GenerateCloseOpenFile(paper, string.Empty), 
                        paper.SEVersion);

                    ZipEntry zipFileEntry = new ZipEntry(paper.DocName + ".pdf")
                    {
                        Size = decryptedPaper.Length
                    };

                    zipOutputStream.SetLevel(3);

                    //EXCEPTION THROWS HERE!!!
                    zipOutputStream.PutNextEntry(zipFileEntry); 

                    StreamUtils.Copy(new MemoryStream(decryptedPaper), 
                        zipOutputStream, 
                        new byte[4096]);
                }
                zipOutputStream.CloseEntry();
                // Stop ZipStream.Dispose() from also Closing the underlying stream.
                zipOutputStream.IsStreamOwner = false;
                outputMemoryStream.Position = 0;
            }
            return outputMemoryStream.ToArray();
        }
    }            
    catch (Exception)
    {
        throw;
    }
}

Expected behavior

Create a zip file and return the result as a byte[]

Actual behavior

Throw OutOfMemoryException

Version of SharpZipLib

ICSharpCode.SharpZipLib 0.85.4.369

Obtained from (only keep the relevant lines)

  • Package installed using NuGet
@piksel
Copy link
Member

piksel commented Jan 27, 2020

This is probably due to random allocation of additional memory. Try setting the initial size of the outputMemoryStream to something close to what the final size ought to be.
It could also be because of how the decompression is attempted inside SharpZipLib, in that case I would suggest trying "Store" instead of Deflation compression, as PDFs usually doesn't compress that well anyway (really depends on the content though).
Another thing you could try is to write the files to a temporary folder before compressing them. If the compression is done in a separate loop, chances are that the memory allocated for the decryption could all be freed by the GC, allowing the compression to allocate the necessary memory chunks.

@Numpsy
Copy link
Contributor

Numpsy commented Jan 30, 2020

Probably pretty trivial, but maybe try writing the decryptedPaper byte array straight into the output stream instead of creating temporary memory streams and buffers inside the loop?

@piksel
Copy link
Member

piksel commented Jun 22, 2020

Closing due to lack of response.

@piksel piksel closed this as completed Jun 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants