Large data mode #182
Closed
Large data mode #182
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This removes the warning from bedtools, 'WARNING: Genome (-g) files are ignored when BAM input is provided.' Test of `bedtools genomecov -ibam records.bam > output.tsv` creates the same file as `bedtools genomecov -ibam records.bam -g lengths.tsv > output.tsv`
Binning now uses embeddings from canonical rank or from the specific rank name within the canononical rank depending on the rank partition size
…ecified type 🎨 Add string instance check in kmers.embed(...) for pca_dimensions and attempt to convert to int if str given.
🐛 Add if statement to check whether user specified an output filepath to update logger message in parse(...)
…into large-data-mode
🎨 Add script to extract log information from recursive_dbscan 🎨 Add autometa-binning-loginfo entrypoint for extracting binning log information
🐛 Fix checkpoint file writing logic where spaces were prepended to comment (#) lines 🐛 Fix merge logic when updating a checkpoint 🐛🎨 Add gzip functionality for writing of binning checkpoints file
…zipped checkpoint files
🎨 Clean logger emit message after retrieval of checkpoint info
…ckpoint_info(...)
🎨 Move search_prot_accessions(...) method to NCBI class 🎨 Add checks for prot.accession2taxid.FULL.gz in NCBI class
LCA blast2lca method now tries to retrieve taxids from sseqid_to_taxid_output iff it is already written . This allows to skip the expensive step of parsing prot.accession2taxid databases and the blast table
🎨🐛 Account for bug where output filepath corresponds to a non-existent directory. Now creates all output directories in outpath that do not exist.
🎨 default is -1, which uses (n_cpus + 1 + core_dist_n_jobs)
…tered metagenome and is trying to retrieve gc_content and other assembly stats.
…into large-data-mode
Sidduppal
reviewed
Oct 19, 2021
autometa/binning/large_data_mode.py
Outdated
| gc_content_stddev : float, optional | ||
| cluster GC content threshold to retain cluster (the default is 5.0). | ||
|
|
||
| taxonomy : bool, optional |
There was a problem hiding this comment.
Great work on specifying whether the user is specifying a taxonomy or not, however, there is no taxonomy variable present in the function declaration and no taxonomy variable is being used in the function.
How can this entry point be useful if the user hasn't specified any taxonomy file, as the cluster_by_taxon_partitioning function is not checking for that
There was a problem hiding this comment.
see filter_taxonomy(..., rank=args.rank_name_filter) call in main().
Also def filter_taxonomy(...) in autometa/binning/utilities.py L72
Sidduppal
reviewed
Oct 19, 2021
Comment on lines
+563
to
+599
| parser.add_argument( | ||
| "--kmers", | ||
| help="Path to k-mer counts table", | ||
| metavar="filepath", | ||
| required=True, | ||
| ) | ||
| parser.add_argument( | ||
| "--coverages", | ||
| help="Path to metagenome coverages table", | ||
| metavar="filepath", | ||
| required=True, | ||
| ) | ||
| parser.add_argument( | ||
| "--gc-content", | ||
| help="Path to metagenome GC contents table", | ||
| metavar="filepath", | ||
| required=True, | ||
| ) | ||
| parser.add_argument( | ||
| "--markers", | ||
| help="Path to Autometa annotated markers table", | ||
| metavar="filepath", | ||
| required=True, | ||
| ) | ||
| parser.add_argument( | ||
| "--taxonomy", | ||
| metavar="filepath", | ||
| help="Path to Autometa assigned taxonomies table", | ||
| required=True, | ||
| ) | ||
| parser.add_argument( | ||
| "--output-binning", | ||
| help="Path to write Autometa binning results", | ||
| metavar="filepath", | ||
| required=True, | ||
| ) | ||
| parser.add_argument( |
There was a problem hiding this comment.
When asking for different input files I feel it would be nice if you can mention the specific format of the file ie. what headers should be there.
Sidduppal
reviewed
Oct 19, 2021
Comment on lines
+675
to
+681
| parser.add_argument( | ||
| "--max-partition-size", | ||
| help="Maximum number of contigs to consider for a recursive binning batch.", | ||
| default=10000, | ||
| metavar="int", | ||
| type=int, | ||
| ) |
There was a problem hiding this comment.
I feel it would be nice to include what determines the selection of max-partition-size. How did you reach the 10,000 value? It would be nice for prospective users to know how to select this value.
Basically when running this how should I user know what partition size to use.
Sidduppal
reviewed
Oct 19, 2021
Comment on lines
+600
to
+602
| "--output-main", | ||
| help="Path to write Autometa main table used during/after binning", | ||
| metavar="filepath", |
There was a problem hiding this comment.
Difference between the binning file and main file is not clear from the help text.
Sidduppal
reviewed
Oct 19, 2021
Sidduppal
reviewed
Oct 19, 2021
🔥📝 Remove unused variables in docstring
🎨🐛 Drop metric columns when adding new metrics to avoid addition of suffixes 🔥🎨 Remove dropcols variable to isolate cluster metrics columns to add_metrics(...) and apply_binning_metrics_filter(...) 🔥🎨 Only drop cluster column in run_hdbscan(...)
🎨 rename 'rank' variable to more specific 'canonical_rank' variable. 🎨 Add logic to retrieve previous canonical rank kmer embedding if the current canonical rank embedding is not possible.
This first checks if the variable has been provided a value prior to filepath and filesize checking
…-binning entrypoints Both clustering algorithms may be passed a parameter to allow them to use more cores for parallelization. e.g. HDBSCAN(..., core_dist_n_jobs) and DBSCAN(..., n_jobs) This parameter has been propagated through to these functions rather than the previously hardcoded -1 due to errors being raised from the joblib.externals library while using HDBSCAN. The errors arising from n_jobs=-1 are infrequent, but frequent enough to merit providing the user more control Some example exceptions that were raised: - `[11/17/2021 10:53:42 PM ERROR] concurrent.futures: exception calling callback for <Future at 0x7fd2083a9c10 state=finished raised BrokenProcessPool>` - `joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.`
🐛 Add input type conversion for --cpus arg in binning entrypoints
…ternals using joblib. 🔥🐛 This is a somewhat known error as similar messages have been discussed [here](scikit-learn/scikit-learn#21685) and on the [hdbscan GH pull-#495](scikit-learn-contrib/hdbscan#495). The error messages is emitted from joblib.externals.loky.process_executor._RemoteTraceback and emits a ValueError: 'ValueError: buffer source array is read-only.' So far this has not been encountered with scikit-learn version 0.24
…=Union[str,pd.DataFrame])
…into large-data-mode
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Behavior
Iterates through taxa and selects contigs to cluster based on a provided upper bound (
--max-partition-size). If the taxon set is below the upper bound, the taxon's k-mer counts are normalized and embedded prior to recursive clustering. If the taxon set is below the lower bound (min_contigs = max([pca_dimensions + 1, embed_dimensions + 1])) k-mer's are retrieved from the canonical rank embedding then the same recursive clustering implementation begins.Features/Additions
autometa-large-data-mode-binning--max-partition-size--cache--binning-checkpointsautometa-large-data-mode-binning-loginfothat will parse log file generated fromautometa-large-data-mode-binningautometa-parse-bedtoautometa-bedtools-genomecov