Too many open files #56
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…the task processing part; transfer manager is global for all tasks
laughedelic
added
status:planned
status:in_progress
and removed
status:in_progress
status:planned
labels
|
TODO:
|
laughedelic
added a commit
to ohnosequences/mg7
that referenced
this pull request
Mar 7, 2016
This milestone release is huge. It contains a lot of improvements and some important fixes. Here are some higlights: * Features and improvements: - #4, #15: The BLAST reference database and IDs mapping is now configurable - #17: Added RNACentral reference DB (as the default one now) - #18: BLAST options are now configurable - #16: A dataflow API was improved and a pipeline without the Flash step was added - ~~#20~~ #24: For pipelines without Flash you can choose between FASTA and FASTQ input format - Output tables improvements: + #10: Added CSV-headers + #19, #23, #27: Added _taxonomic rank_ and _name_ columns + #6: Split each counts table on two: direct/accumulated + #30: Zero direct counts are filtered out * Important fixes: - #28: Bio4j-NCBITaxonomy database was updated with an important fix for the taxonomic names - #26: Ongoing effort to solve the "Too many open files" exception has been moved to [ohnosequences/loquat#56](ohnosequences/loquat#56) See the full list of pull requests merged in this release in the [v1.0-M2 milestone](https://github.com/ohnosequences/mg7/issues?q=milestone%3Av1.0-M2).
…rocessing exceptions
|
@eparejatobes review, please. I've done what's written above plus some changes from our experiments. |
| @@ -212,6 +209,7 @@ class DataProcessor( | |||
| ) | |||
| } | |||
|
|
|||
| transferManager.shutdownNow(false) | |||
There was a problem hiding this comment.
Now I'm not sure this should be here.. "_
laughedelic
added a commit
that referenced
this pull request
Apr 8, 2016
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a worker processes too many tasks, it exceeds the limit of open files, because it deletes working directory for every task, but it stays in somewhere as an open descriptor or whatever. See ohnosequences/mg7#26 for details.