Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tar expansion should not use so much memory #77

Closed
luispedro opened this issue Jul 11, 2018 · 1 comment
Closed

tar expansion should not use so much memory #77

luispedro opened this issue Jul 11, 2018 · 1 comment

Comments

@luispedro
Copy link
Member

@luispedro luispedro commented Jul 11, 2018

Currently, tar expansion is done in such a way that files are expanded in memory as they are extracted. The relevant function is:

expandTar :: Tar.Entries Tar.FormatError -> NGLessIO ()

The tar-conduit package should be able to stream the file correctly.

unode added a commit to unode/ngless that referenced this issue Oct 29, 2018
@luispedro
Copy link
Member Author

@luispedro luispedro commented Jan 13, 2019

Once snoyberg/tar-conduit#26 and snoyberg/tar-conduit#27 are closed (and make it into a stackage LTS), we can implement tar expansion with tar-conduit and close this for good.

@luispedro luispedro closed this in 3a157d3 Jan 16, 2019
luispedro added a commit that referenced this issue Feb 22, 2019
A collection of several bugfixes and performance improvements over the
last few months.

Full ChangeLog:

    * Switch to diagrams package for plotting
    * Update minimap2 version to 2.14
    * Module samtools (version 0.1) now includes samtools_view
    * Update to LTS-13 (GHC 8.6)
    * Fix bug with orf_find & prots_out argument
    * Call bwa/minimap2 with interleaved fastq files
    * Add --verbose flag to check-install mode
    * Avoid leaving open file descriptors after FastQ encoding detection
    * Fix bug in garbage collection
    * Compress intermediate SAM files (#22)
    * Tar extraction uses much less memory (#77)
    * Add early checks for input files in more situations (#33)
    * Support compression in collect() output (#42)
    * Fix CIGAR (#92) for select() blocks
luispedro added a commit that referenced this issue Mar 15, 2019
A collection of several bugfixes and performance improvements over the
last few months.

Full ChangeLog:

    * Switch to diagrams package for plotting
    * Update minimap2 version to 2.14
    * Module samtools (version 0.1) now includes samtools_view
    * Fix bug with orf_find & prots_out argument
    * Call bwa/minimap2 with interleaved fastq files
    * Add --verbose flag to check-install mode
    * Avoid leaving open file descriptors after FastQ encoding detection
    * Fix bug in garbage collection
    * Compress intermediate SAM files (#22)
    * Tar extraction uses much less memory (#77)
    * Add early checks for input files in more situations (#33)
    * Support compression in collect() output (#42)
    * Fix CIGAR (#92) for select() blocks
    * Update to LTS-13 (GHC 8.6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant