Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tar expansion should not use so much memory #77

Closed
luispedro opened this issue Jul 11, 2018 · 1 comment
Closed

tar expansion should not use so much memory #77

luispedro opened this issue Jul 11, 2018 · 1 comment
Labels
enhancement help wanted performance

Comments

@luispedro
Copy link
Member

Currently, tar expansion is done in such a way that files are expanded in memory as they are extracted. The relevant function is:

ngless/NGLess/Utils/Network.hs

Line 69 in 89309c6

expandTar :: Tar.Entries Tar.FormatError -> NGLessIO ()

The tar-conduit package should be able to stream the file correctly.
unode added a commit to unode/ngless that referenced this issue Oct 29, 2018
@unode
ENH Implement tarball extraction with tar-conduit
9d2787f 
fixes ngless-toolkit#77
@luispedro
Copy link
Member Author

Once snoyberg/tar-conduit#26 and snoyberg/tar-conduit#27 are closed (and make it into a stackage LTS), we can implement tar expansion with tar-conduit and close this for good.

luispedro added a commit that referenced this issue Feb 22, 2019
@luispedro
RLS Release 0.11.0
a665cd8 
A collection of several bugfixes and performance improvements over the
last few months.

Full ChangeLog:

    * Switch to diagrams package for plotting
    * Update minimap2 version to 2.14
    * Module samtools (version 0.1) now includes samtools_view
    * Update to LTS-13 (GHC 8.6)
    * Fix bug with orf_find & prots_out argument
    * Call bwa/minimap2 with interleaved fastq files
    * Add --verbose flag to check-install mode
    * Avoid leaving open file descriptors after FastQ encoding detection
    * Fix bug in garbage collection
    * Compress intermediate SAM files (#22)
    * Tar extraction uses much less memory (#77)
    * Add early checks for input files in more situations (#33)
    * Support compression in collect() output (#42)
    * Fix CIGAR (#92) for select() blocks
luispedro added a commit that referenced this issue Mar 15, 2019
@luispedro
RLS Release 0.11.0
88301ed 
A collection of several bugfixes and performance improvements over the
last few months.

Full ChangeLog:

    * Switch to diagrams package for plotting
    * Update minimap2 version to 2.14
    * Module samtools (version 0.1) now includes samtools_view
    * Fix bug with orf_find & prots_out argument
    * Call bwa/minimap2 with interleaved fastq files
    * Add --verbose flag to check-install mode
    * Avoid leaving open file descriptors after FastQ encoding detection
    * Fix bug in garbage collection
    * Compress intermediate SAM files (#22)
    * Tar extraction uses much less memory (#77)
    * Add early checks for input files in more situations (#33)
    * Support compression in collect() output (#42)
    * Fix CIGAR (#92) for select() blocks
    * Update to LTS-13 (GHC 8.6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@luispedro