Major changes to plastid
are documented here. Version numbers for the
project follow the conventions described in PEP 440 and
Semantic versioning 2.0.0.
This is a maintenance release meant to leave this package in a reasonable state, as it is only sporadically maintained.
- Dockerization, to further control test environments, and make this release release more future-proof. Test environments now include Python 3.6 and 3.9.
- Bumped minimum requirements to reasonable 2022 standards.
- Upgraded embedded Kent & HTSlib source code
- Clarified licenses
- Dropped support for Python versions 2.7–3.5. These might still run, but are no longer tested.
- Deprecated classes:
BPlusTree
,RTree
- Deprecated methods: of
SegmentChain.get_length()
andSegmentChain.get_masked_length()
- Updates to package metadata and docs
Changes are predominantly for maintenance, bugfixes, and streamlining.
Iterators rewritten for compatibility with Python 3.7 and Python 3.8 (per instructions in PEP 479 )
As a result, while readers in :mod:`plastid.readers` are still generators, they are no longer their own iterators. This is a bit of an obscure point, but implications are:
they can still be cast to lists:
>>> my_list = list(GTF2_TranscriptAssembler("/path/to/foo.gtf"))
they can still be used in
for
loops:>>> for my_transcript in GTF2_TranscriptAssembler("/path/to/foo.gtf"): >>> # pass
they cannot be used like this:
>>> transcripts = GTF2_TranscriptAssembler("/path/to/foo.gtf") >>> a = next(transcripts)
# a will be None!!
instead, wrap the reader in iter if you want to use them this way:
>>> transcripts = iter(GTF2_TranscriptAssembler("/path/to/foo.gtf")) >>> a = next(transcripts) # a is now a Transcript
- BioConda support (special thanks to
@lparsons
) - Testing streamlined via
tox
, and additional test environments added
- Moved calls to
open()
intowith
context managers to avoid creation of stale filehandles - Rewrote
setup.py
to remove requirement for pre-installation ofcython
,numpy
, andpysam
, as this is now handled by :mod:`pip` - Removed references to deprecated
pandas.DataFrame.sort()
, enabling compatibility withpandas
versions above 0.2.0 - Improved compatibility with
numpy
versions above 1.31.1 VariableFivePrimeMapFactory.from_file()
andStratifiedVariableFivePrimeMapFactory.from_file()
now work on filenames as well as file handles, as they were supposed toStratifiedVariableFivePrimeMapFactory
now imported by typingfrom plastid import *
- And others as well
BigBedReader.custom_fields
was removed in favor of its non-deprecated alias,BigBedReader.extension_fields
- Fixed a change in setup.py that caused Plastid compilation to fail in Macintosh environments. Sorry Mac users!
This update is minor compared to the release 0.4.6, and was mainly motivated by
updates, bugfixes, and changes required for compatibility with new versions of
Pysam
- Support for
Pysam
>= 0.10.0 write_pl_table()
added as a convenience function--use_mean
flag added tometagene
- Warnings / better help text
- rounding error in
get_str_from_rgb()
PSL_Reader()
now capable of parsing strands from translated blat output- Fixed bug in header parsing in
PSL_reader
Highlights
- Support for `BigWig`_ files
- Reimplementation of `BigBed`_ file support
- Simplification of syntax / removal of annoyances in both command-line scripts and in infrastructure
- Support for `BigWig`_ files.
BigWigReader
reads `BigWig`_ files, andBigWigGenomeArray
handles them conveniently. BigBedReader
has been reimplemented using Jim Kent's C library, making it far faster and more memory efficient.BigBedReader.search()
created to search indexed fields included in BigBed files, e.g. to find transcripts with a given gene_id (if gene_id is included as an extension column and indexed). To see which fields are searchable, useBigBedReader.indexed_fields
Simplified file opening. All readers can now take filenames in addition to open filehandles. No need to wrap filenames in lists any more. For example:
# old way to open GTF2 file >>> data = GTF2_TranscriptAssembler(open("some_file.gtf")) # new way. Also works with BED_Reader, GTF2_Reader, GFF3_TranscriptAssembler, and others >>> data = GTF2_TranscriptAssembler("some_file.gtf") # old way to get read alignments from BAM files >>> alignments = BAMGenomeArray(["some_file.bam","some_other_file.bam"]) # new way >>> alignemnts = BAMGenomeArray("some_file.bam","some_other_file.bam") # old way to open a tabix-indexed file >>> data = BED_Reader(pysam.tabix_iterator(open("some_file.bed.gz"),pysam.asTuple()),tabix=True) # new way >>> data = BED_Reader("some_file.bed.gz",tabix=True)
To maintain backward compatibility, the old syntax still works
BAMGenomeArray
can now use mapping functions that return multidimensional arrays. As an example we addedStratifiedVariableFivePrimeMapFactory
, which produces a 2D array of counts at each position in a region (columns), stratified by read length (rows).Reformatted & colorized warning output to improve legibility
read_pl_table()
convenience function for reading tables written by command-line scripts into DataFrames, preserving headers, formatting, et c
All script output metadata now includes command as executed, for easier re-running and record keeping
Scripts using count files get
--sum
flag, enabling users to set effective sum of counts/reads used in normalization and RPKM calculationspsite
--constrain
option added topsite
to improve performance on noisy or low count data.- No longer saves intermediate count files.
--keep
option added to take care of this.
metagene
- Fixed/improved color scaling in heatmap output. Color values are now capped at the 95th percentile of nonzero values, improving contrast
- Added warnings for files that appear not to contain UTRs
- Like
psite
, no longer saves intermediate count files.--keep
option added to take care of this.
phase_by_size
can now optionally use an ROI file from themetagene generate
subprogram. This improves accuracy in higher eukaryotes by preventing double-counting of codons when more than one transcript is annotated per gene.cs chart
file containing list of genes is now optional. If not given, all genes are included in comparisonsreformat_transcripts
is now able to export extended BED columns (e.g. gene_id) if the input data has useful attributes. This particularly useful when working with large transcript annotations in GTF2/GFF3 format- they can now be exported to BED format, and converted to BigBed foramt, allowing random access and low memory usage, while preserving gene-transcript relationships.
- Version parsing bug in setup script.
@deprecated
function decorator now givesFutureWarning
instead ofDeprecationWarning
--norm_region
option ofpsite
andmetagene
has been deprecated and will be removed inplastid
v0.5. Instead, use--normalize_over
, which performs the same role, except coordinates are specified relative to the landmark of interest, rather than entire window. This change is more intuitive to many users, and saves them mental math. If both--norm_region
and--normalize_over
are specified,--normalize_over
will be used.BigBedReader.custom_fields
has been replaced withBigBedReader.extension_fields
BigBedReader.chrom_sizes
has been replaced withBigBedReader.chroms
for consistency with other data structuresBPlusTree
andRTree
classes, which will be removed inplastid
v0.5
Changes here are mostly under the hood, involving improvements in usability, speed, stability, compatibility, and error reporting. We also fixed up tools for developers and added entrypoints for custom mapping rules.
Users can now control verbosity/frequency of warnings via '-v' or '-q' options! By default there should no long screens of DataWarnings when processing Ensembl (or other) GTFs.
--aggregate
option added topsite
script to improve sensitivity for low-count data.Created entrypoints for allowing users to use custom mapping rules in the command line scripts:
plastid.mapping_rules
for specifying new mapping functionsplastid.mapping_options
for specifying any other command-line arguments they consume
Detailed instructions for use in the developer info section of plastid.readthedocs.org.
Argument parsing classes that replace methods deprecated below:
- updated plotting tools to fetch color cycles from matplotlib versions >= 1.5
as well as >= 1.3. This corrected a plotting bug in cs.
- :meth:`AnnotationParser.get_genome_hash_from_args` now internally uses
GFF3_Reader and GTF2_Reader instead of GFF3_TranscriptAssembler and GTF2_TranscriptAssembler, allowing mask files in GTF2/GFF3 foramts to be type-agnostic in command-line scripts
contig names no longer lost when using 2bit files in crossmap
updates to :mod:`~plastid.bin.psite`
- output header in metagene profiles. Sorry about that
- fix compatibility problem with new versions of matplotlib
- now catches a
ValueError
that used to be anIndexError
in earlier versions of :mod:`numpy`.
Fixed loss-of-ID bug in :meth:`Transcript.get_cds`
- :func:`~plastid.util.services.decorators.deprecated` function decorator
- now optionally takes parameters indicating the future version of plastid in which deprecated features will be removed, and what replacement to use instead
Argument parsing methods:
get_alignment_file_parser()
&get_genome_array_from_args()
. Use :class:`~plastid.util.scriptlib.argparsers.AlignmentParser` instead.get_annotation_file_parser()
&get_transcripts_from_args()
,get_segmentchain_file_parser()
&get_segmentchains_from_args()
Use :class:`~plastid.util.scriptlib.argparsers.AnnotationParser` instead.get_mask_file_parser()
&get_genome_hash_from_mask_args()
. Use :class:`~plastid.util.scriptlib.argparsers.MaskParser` instead.get_sequence_file_parser()
&get_seqdict_from_args()
. Use :class:`~plastid.util.scriptlib.argparsers.SequenceParser` insteadget_plotting_parser()
,get_figure-from_args()
, &get_colors_from_args
. Use :class:`~plastid.util.scriptlib.argparsers.PlottingParser` instead
Although the list of changes is short, this release includes dramatic reductions in memory usage and speed improvements, as well as a few bug fixes. We recommend everybody upgrade
- Fast
merge_segments()
function inroitools
module.
- 10-100 fold reduction in memory consumed by
SegmentChain
objects, GTF2_TranscriptAssembler
andGFF3_TranscriptAssembler
. All position & mask hashes now lazily evaluated
- 10-100 fold reduction in memory consumed by
- 50-fold fold Speed boosts in
SegmentChain.overlaps()
, SegmentChain.covers()
and other methods for comparingSegmentChain
andTranscript
objects
- 50-fold fold Speed boosts in
GenomicSegment
is now hashable, e.g. can be used in sets or dict keys
- Track naming bug in
make_wiggle
- init bug in
GenomeHash
- Fixed bug in
crossmap
script when run on 2bit files
No change in codebase vs 0.4.0. Updated required matplotlib version to 1.4.0. Made some changes in sphinx doc config for readthedocs.org, which is still at matplotlib 1.3.0.
This release primarily focuses on ease of use: mainly, it is a lot easier to do things with fewer lines of code. Imports have been shortened, plotting tools have been added, and scripts now produce more informative output.
Logical imports: the following commonly-used data structures can now be directly imported from the parent package
plastid
, instead of subpackages/submodules:GenomicSegment
,SegmentChain
, andTranscript
- All GenomeHashes and GenomeArrays
- All file readers
VariableFivePrimeMapFactory
can now be created from static methodfrom_file()
, so no need to manually parse text files or create dictionariesBAMGenomeArray
can now be initialized with a list of paths to BAM files, in addition or instead of a list ofpysam.AlignmentFiles
Plotting improvements
plastid.plotting
package, which includes tools for making MA plots, scatter plots with marginal histograms, metagene profiles, et c- more informative plots made in
metagene
,psite
,phase_by_size
, andcs
scripts - support for matplotlib stylesheets, colormaps, et c in all command-line scripts
add_three_for_stop_codon()
reimplemented in Cython, resulting in 2-fold speedup. Moved fromplastid.readers.common
toplastid.genomics.roitools
(though previous import path still works)
- Fixed IndexError in
psite
that arose when running with the latest release of numpy, when generating a read profile over an empty array - Legends/text no longer get cut off in plots
- Removed deprecated functions
BED_to_Transcripts()
andBED_to_SegmentChains
, for whichBED_Reader
serves as a drop-in replacement
- Important docstring updates: removed outdated warnings and descriptions
- Cython implementations of
SegmentChain
andTranscript
provide massive speedups Transcript.cds_start
,cds_genome_start
,cds_end
,cds_genome_end
are now managed properties and update each other to maintain synchronySegmentChain._segments
andSegmentChain._mask_segments
are now read-only
- Methods
SegmentChain.get_length()
andSegmentChain.get_masked_length()
are replaced by propertiesSegmentChain.length
andSegmentChain.masked_length
sort_segments_lexically()
andsort_segmentchains_lexically()
removed, becauseGenomicSegment
andSegmentChain
now sort lexically without help
- Cython implementations of BAM mapping rules now default, are 2-10x faster than Python implementations
First release under official name!
- Major algorithmic improvements to internals & command-line scripts
- Reimplemented mapping rules and some internals in Cython, giving 2-10x speedup for some operations
GenomicSegment
now sorts lexically. Properties are read-only
Note
This project was initially developed internally under the provisional name
genometools
, and then later under the codename yeti
. The current
name, plastid
will not change. Changelogs from earlier versions
appear below.
- Support for extended BED formats now in both import & export, in command-line scripts and interactively
- BED Detail format and known ENCODE BED subtypes now automatically parsed from track definition lines
- Created warning classes DataWarning, FileFormatWarning, and ArgumentWarning
- parallelized crossmap script
- command line support for more sequence file formats; and a sequence
- argparser
- speed & memory optimizations for cs generate script, resulting in 90% memory reduction on human genome annotation GrCh38.78
- ditto metagene generate script
- crossmap script does not save kmer files unless --save_kmers is given
- warnings now given at first (instead of every) occurence
- lazy imports; giving speed improvements to command-line scripts
Big changes, including some that are backwards-incompatible. We really think these are for the best, because they improve compatibility with other packages (e.g. pandas) and make the package more consistent in design & behavior
- GenomeArray __getitem__ and __setitem__ now can take SegmentChains as arguments
- Mapping functions for bowtie files now issue warnings when reads are unmappable
- support for 2bit files (via twobitreader) and for dicts of strings in SegmentChain.get_sequence
- various warnings added
- pandas compatibility: header rows in all output files no longer have starting '#. meaning UPDATE YOUR OLD POSITIONS/ROI FILES
- __getitem__ from GenomeArrays now returns vectors 5' to 3' relative to
GenomicSegment rather than to genome. This is more consistent with user expectations.
- _get_valid_X methods of SegmentChain changed to _get_masked_X for consistency with documentation and with numpy notation
- ArrayTable class & tests
- Created & backpopulated changelog
- Docstrings re-written for user rather than developer focus
- AssembledFeatureReader
- Complete first draft of user manual documentation
- Readthedocs support for documentation
- GFF3_TranscriptAssembler now also handles features whose subfeatures share ID attributes instead of Parent attributes.
- import of scientific packages now simulated using mock during documentation builds by Sphinx
- duplicated attributes in GTF2 column 9 are now catenated & returned as a list in attr dict. This is outside GTF2 spec, but a behavior used by GENCODE
- Removed bug from :func:`yeti.bin.metagene.do_generate` that extended maximal spanning windows past equivalence points in 3' directions. Added extra unit test cases to suit it.
- GenomeHash can again accept GenomicSegments as parameters to __getitem__. Added unit tests for this.
Removed deprecated functions, modules, & classes:
- GenomicFeature
- BED_to_Transcripts
- BigBed_to_Transcripts
- GTF2_to_Transcripts
- GFF3_to_Transcripts
- TagAlignReader
First internal release of project under new codename, yeti
. Reset version
number.
Added .....a
- AssembledFeatureReader, GTF2_TranscriptAssembler, GFF3_TranscriptAssembler
- GTF2/GFF3 token parsers now issue warnings on repeated keys
- GFF3 token parsers now return 'Parent', 'Alias', 'Dbxref', 'dbxref', and 'Note' fields as lists
Package renamed from
genometools
to its provisional codenameyeti
Reset version number to 0.1.0
Refactored existing readers to descent from AssembledFeatureReader
Migration from old SVN to GIT repo
Renaming & moving of functions, classes, & modules for consistency and to avoid name clashes with other packages
Old name
New Name
GenomicInterva
GenomicSegment
IVCollection
SegmentChain
NibbleMapFactory
CenterMapFactory
genometools.genomics.ivtools
yeti.genomics.roitools
genometools.genomics.readers
yeti.readers
genometools.genomics.scriptlib
yeti.util.scriptlib
- renamed suppress_stdr -> capture_stderr
- capture_stdout decorator
- All functions that used GenomicFeatures now use IVCollections instead
- GenomicFeature support from GenomeHash subclasses
- GenomicFeature support from IVCollection and GenomicInterval overlap end quality criteria
- GenomicFeature
- Included missing .positions and .sizes files into egg package
- Test data now packaged in eggs
- updated documentation
- Bug in cleanup for test_crossmap
- Bug in setup.py
- Python 3.0 support
- Support for tabix-compressed files. Creation of TabixGenomeHash
- Propagate various attributes to sub-features (utr_ivc, CDS) from Transcript
- Propagate all attributes to sub-features during GTF export from Transcript
- GTF2 export of Transcript objects now generates 'start_codon' and 'stop_codon' features
- Update of setup.py and Makefile to make dev vs distribution eggs
- 'transcript_ids' column of 'cs generate' position file now sorted before comma join.
- Merger of make_tophat_juncs, find_juncs, and merge_juncs into one script
- Standardization of column names among various output files: region, regions_counted, counts
- Standardized method names in IVCollection: get_valid_counts, get_valid_length, get_length, get_counts, et c
- IVCollection/Transcript openers/assemblers all return generators and can take multiple input files
- IVCollection/Transcript openers/assemblers return lexically-sorted objects
- Update to GFF3 escaping conventions rather than URL escaping. Also applied to GTF2 files
- Refactors to cs script, plus garbage collection to reduce memory usage
- Changelog
- Implementation of test suites
- Lazy assembly of GFF3 and GTF2 files to save memory in GTF2_TranscriptAssembler and GFF3_TranscriptAssembler
- BigBed support, creation of BigBedReader and BigBedGenomeHash. AutoSQL support
- Supported for truncated BED formats
- P-site offset script
- get_count_vectors script
- counts_in_region script
- UniqueFifo class
- Decorators: parallelize, suppress_stderr, in_separate_process
- variableStep export for BAMGenomeArray
- Support of GTF2 "frame" attribute for CDS features
- Bugfixes in minus strand offsets in crossmaps
- Fixed bug where minus strand crossmap features were ignored
- Bugfixes in CDS end export from Transcript when CDSes ended at the endpoint of internal but not terminal introns on plus-strand transcripts
- spliced_count_files
- Ingolia file tagalign import
- Deprecation of GTF2_to_Transcripts and GFF3_to_Transcripts