Dev (#186)

* Major DROP update

* removed obsolete file

* changed project folder structure for initialisation

* API change: separated config and sample annotation parsers

* fixed missin root path

* import config differently

* package finding for subpackage

* running pipeline

* added circleci config

* removed obsolete file

* changed project folder structure for initialisation

* API change: separated config and sample annotation parsers

* fixed missin root path

* import config differently

* package finding for subpackage

* using correct .travis.yml

* removed subworkflow and running pipeline for aberrantExpression; new functions introduced for Script-Rule-Html conversions; analysis script directory renamed; pipeline code copied to different destination

* removed subworkflow for AS and MAE; changed folder structure; using log for tmp snakemake objects

* fixed input flags for analysis scripts

* running pipeline

* added circleci config

* removed obsolete file

* changed project folder structure for initialisation

* API change: separated config and sample annotation parsers

* fixed missin root path

* import config differently

* package finding for subpackage

* API change: separated config and sample annotation parsers

* fixed missin root path

* import config differently

* package finding for subpackage

* removed subworkflow and running pipeline for aberrantExpression; new functions introduced for Script-Rule-Html conversions; analysis script directory renamed; pipeline code copied to different destination

* removed subworkflow for AS and MAE; changed folder structure; using log for tmp snakemake objects

* fixed input flags for analysis scripts

* set submodules to master

* applied branch-specific modifications

* removed submodules.py and refactored path setup

* fixed Readme and config copy errors

* fixed copying helpers functions

* using wbuild with fixed Readme.html functionality and saving wbuild config in DropConfig

* moved exportCounts into separate class

* parsing GENE_COUNT_FILE column in sample annotation

* removed wbuild from conda recipe

* using bioconda version of drop for dependencies, pip install drop, no conda building

* remove bioconda wbuild

* install wbuild with expl pip

* remove wbuild with pip

* get count info from AE class

* some refactoring and saving external counts IDs in separate dictionary

* removed ID renaming in merge script (naming already in counts script); fixed missing input bug in DropConfig

* add external counts to count files

* merge counts and coverage including external counts

* file checks for config and sample annotation reimplemented

* config file getters for submodules

* refactored export counts

* travis run export counts rule

* reapplied bcftools command modification

* Documentation update (#102)

* README update to include Baylor counts
* updated drop installation command
* update install command docs to include conda-forge plus better descriptions

Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Michaela Müller <mi.mueller@tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>

* updated documentation to include count import

* splicing export counts all columns added

* resolved requested formality errors

* fixed requested MAE changes

* removed scanBamParam

* create missing columns for MAE results

* Subindex (#3)

* first version of subindex implemented

* including readme and dependency graph for subindexes; removed fileRegex key from config (using default of ".*\.(R|md)")

* using different (wb1.8) config file

* separated dependency graph computation from rest of the pipeline

* using latest updated wbuild version

* use conda install for wbuild

Co-authored-by: mumichae <mi.mueller@tum.de>

* Fix version for wbuild (#108)

Co-authored-by: mumichae <mi.mueller@tum.de>

* updated version number in README and drop/cli

* Tests (#4)

Added pytest suite to project. The main things tested are:

* cli: basic drop command line functions

* all config classes

* pipeline runs for all submodules (including checking numbers of entries in output) and count export


## Commits

* first setup of pytest, pipeline runthroughs, no output checking yet

* updated version number in README and drop/cli

* fixed dependencies

* pip uninstall with -y

* changed pytest installation and error catching

* compare pipeline output and add more pipeline tests

* refactored pipeline tests

* fixed demo creation

* refactored pipeline tests again & fixed fixture scoping

* fixed minor issues in tests

* added config tests

* refactored getHtmlFromScript

Co-authored-by: mumichae <mi.mueller@tum.de>

* downloading data to temporary directory

* updated version to 1.0.0

Co-authored-by: Michaela Müller <mi.mueller@tum.de>
Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>

* updated travis to use correct R and wbuild versions

* use github release badge instead of fixing it

* Fix Issue #114 (#116)

* refactored drop demo code s.t. config and sample annotation files are in the package

* adapted pipeline to handle non-existing external counts
new group in demo to distinguish count import from no count import cases
some bugfixes regarding exportCounts parsing and import counts subsetting

* fixed AE pipeline test

* travis with latest R version 4

* revert pipeline in pytest to 2 cores

* R 4.0.2

Co-authored-by: mumichae <mi.mueller@tum.de>

* Refactoring (#118)

* refactored Submodule classes into separate files

* using MAE and AS functions from respective classes instead of Snakefile or R script header

* fixed input for MAE rules

* use chr with bam counts for ASEReadCounter

* added test for aberrant splicing count file functions

* FRASER compatibility fixes

* fixed subsetGroup API change

Co-authored-by: Michaela Müller <mi.mueller@tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>

* using wbuild 1.8.0 for pip installation

* fixed travis install error

* adapt DROP to theta usage in FRASER (#129)

* adapt DROP to theta usage in FRASER

* use travis wait

* due to new filters one donor site is deleted

* Fix import counts (#122)

* updated version number

* version checking for drop

* use defaultdict for group IDs; add testcase for removing import counts in demo input; fix count param data types

* moved no import file modifications into separate fixture

* fixed sample annotation ID mapping check

* using groups defined in config instead of inferring from ID objects

* force ignoring first column as index when reading sample annotation file

* added wait to pytest command

* drop update to overwrite files every time & shortened output for creating drop demo fixture

* fixed deprecation message

* profiling runtime for tests

Co-authored-by: mumichae <mi.mueller@tum.de>

* Offline addons (#137)

* Update gene_name_mapping.R

* add hpoFile option

* fix hg38 on FRASER counting

* doc hpoFile, files to download

* add hpoFile option

* doc hpoFile, files to download

* vcf doc fixed

* gene_biotype added

Co-authored-by: Vicente <yepez@in.tum.de>

* Offline addons (#139)

* Update gene_name_mapping.R

* add hpoFile option

* fix hg38 on FRASER counting

* doc hpoFile, files to download

* add hpoFile option

* doc hpoFile, files to download

* vcf doc fixed

* gene_biotype added

* hpo term column can be blank

Co-authored-by: Vicente <yepez@in.tum.de>

* remove file extension in outrider module (#140)

* Update splicing export (#141)

* export known junction with K and N counts

* add AnnotationDbi

* fix type

* properly load BSgenome

* correctly export splice counts (use K and N) and use GTF seqlevelsStyle

* speed up conda install

* fix package removal

* use travis python version for drop environment

Co-authored-by: mumichae <51025211+mumichae@users.noreply.github.com>

* Version check (#143)

* include version check for R packages when calling pipeline

* silence package loading when checking R installation

* hide warning messages in R install

Co-authored-by: mumichae <mi.mueller@tum.de>

* better check for chromosome style (#148)

* GitHub actions (#151)

* Create python-package-conda.yml

* install repo version of drop

* commented out travis file

* Export counts meta (#144)

* refactored writing export count meta files in separate python script rather than SampleAnnotation script

* include python scripts in non-module directories upon installation

Co-authored-by: mumichae <mi.mueller@tum.de>

* use buildins for flake8

* Install needed BSgenome on the fly (#147)

* remove assembly specific packages closes #142
* fix pipeline when no import count matrix in sample annotation #114 
* refactored config parsers by submodule
* update to FRASER 1.2.0 @c-mertes 
* update aberrant splicing export @c-mertes 
* local loading of HPO file if present @vyepez88 
* fixed buggy file extension when saving ods @jemten 
* Add files via upload (#146)
* update OUTRIDER version and bugfixes
* allow NCBI based assembly in config fixes #135
* move assembly version test to DropConf and adjust the pipeline.
* add unlock if snakemake fails
* use addAF to check installation
* Add badge to readme
Co-authored-by: mumichae <51025211+mumichae@users.noreply.github.com>
Co-authored-by: Vicente Yepez <30469316+vyepez88@users.noreply.github.com>

* Offline res (#160)

* annotate splice results using GTF (requires FRASER 1.3.0)

* added random seed for demo

Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>
Co-authored-by: Michaela Mueller <mumichae@in.tum.de>

* add bumpversion config

* Bump version: 1.0.2 → 1.0.3

* rename gene_biotype->gene_type in GTF (#162)

* Apply suggestions from code review

Co-authored-by: Christian Mertes <mertes@in.tum.de>

* FRASER version and author list

Co-authored-by: Christian Mertes <mertes@in.tum.de>

* Simplify MAE Pipeline (#158)

* removed separate QC VCF workflow, reusing MAE workflow

* Chr style (#177)

* mae chr style fixed

* bug in chr style of FRASER fixed

* add possibility of both chr styles

* ncbi fds object added

* minor fixes chr subset

* count matrices added

Co-authored-by: Vicente <yepez@in.tum.de>

* Qol changes (#179)

QOL changes are made to improve the ease and user experience. This branch contains changes to allow the following:

* safe overwriting of the local working directory with changes from the source drop directory. This should save some reprocessing time as only modified scripts will be rerun
* allowing multiple genomic reference files. allow for user to designate multiple references or a global reference from the config file and sample annotation table
* added additional checks and error messages
* use changes in the sample annotation to trigger reruns of the aberrantExpression and MAE pipelines. (aberrantSplicing is still in progress)

* Update warnings/errors
* Update tests 
* Handle GENOME column
* Backwards compatibility: ANNOTATION -> GENE_ANNOTATION
* Docs sample anno expanded including genome option
* update wildcard constraint
* Refactored genome operations to separate Genome class
* sampleParams file and class
* remove sampleParam code from Submodule class

Co-authored-by: Smith Nicholas <smith@in.tum.de>
Co-authored-by: nickhsmith <smithnickh@gmail.com>
Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>
Co-authored-by: Michaela Mueller <mumichae@in.tum.de>

* Qol changes real (#185)

* fresh QOL branch

* remove GENOME when not in SA

* configure paths, and fix tests to be more specific

* update pytest test_AS.py

* Update 07_extract_results_FraseR.R

* Update DropConfig.py

* Update SampleAnnotation.py

* Update test_AS.py

add potential future test for ncbi_fds obj

* Update test_AS.py

update assert

* additional checks and warnings for reading genome path

* fix warning trigger

* update sa-table column header. ANNOTATION -> GENE_ANNOTATION

* docs sample anno expanded

* check if sample has both BAM and extcount inputs

* update warnings

* docs updated with genome option

* force r-dpylr later version install

* correct typo

* set r-dpylr value

* update warnings

* update warnings/errors

* add comments, and changes based on code review discussion

* update wildcard constraint, and move it from AS to main template

* Refactored genome operations to separate Genome class

* rename sample sa to annotationTable

* fix reference mapping

* save local changes

* sampleParams file and class

* remove sampleParam code from Submodule class

* update errors and warnings for MAE

Co-authored-by: Smith Nicholas <smith@in.tum.de>
Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>
Co-authored-by: Michaela Mueller <mumichae@in.tum.de>

* Bump to 1.0.3 (#155) (#189)

* Fix Issue #114 import counts (#116, #122)

* Refactoring (#118)

* adapt DROP to theta usage in FRASER (#129)

* Offline runs (#137, #139, #160)

* remove file extension in outrider module (#140)

* Update splicing export (#141)

* Version check (#143)

* better check for chromosome style (#148)

* GitHub actions (#151)

* Export counts meta (#144)

* Install needed BSgenome on the fly (#147)

* rename gene_biotype->gene_type in GTF (#162)

Co-authored-by: Michaela Müller <mi.mueller@tum.de>
Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>
Co-authored-by: Anders Jemt <jemten@users.noreply.github.com>
Co-authored-by: Alex Paul <github@ajpdev.com>

Co-authored-by: Michaela Mueller <51025211+mumichae@users.noreply.github.com>
Co-authored-by: Michaela Müller <mi.mueller@tum.de>
Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>
Co-authored-by: Anders Jemt <jemten@users.noreply.github.com>
Co-authored-by: Alex Paul <github@ajpdev.com>

* Small fixes (#192)

* remove unnecessary DNA_ID for export counts

* updated bumversion config

* Bump version: 1.0.3 → 1.0.4

Co-authored-by: mumichae <51025211+mumichae@users.noreply.github.com>
Co-authored-by: Michaela Müller <mi.mueller@tum.de>
Co-authored-by: Vicente <yepez@in.tum.de>
Co-authored-by: Christian Mertes <mertes@in.tum.de>
Co-authored-by: Anders Jemt <jemten@users.noreply.github.com>
Co-authored-by: Michaela Mueller <mumichae@in.tum.de>
Co-authored-by: Alex Paul <github@ajpdev.com>
Co-authored-by: Smith Nicholas <smith@in.tum.de>
Co-authored-by: nickhsmith <smithnickh@gmail.com>

docs/source/conf.py

@@ -23,7 +23,7 @@
 author = 'Michaela Müller'
 
 # The full version, including alpha/beta/rc tags
-release_ = '1.0.3'
+release_ = '1.0.4'
 
 
 # -- General configuration ---------------------------------------------------

drop/__init__.py

@@ -4,4 +4,4 @@
 from . import utils
 from . import demo
 
-__version__ = "1.0.3"
+__version__ = "1.0.4"

drop/cli.py

@@ -17,7 +17,7 @@
 
 @click.group()
 @click_log.simple_verbosity_option(logger)
-@click.version_option('1.0.3',prog_name='drop')
+@click.version_option('1.0.4',prog_name='drop')
 def main():
     pass
 

drop/template/Scripts/Pipeline/exportCountsMeta.py

@@ -69,8 +69,7 @@ def get_pairedEnd(sa):
 
     # subset by group
     rna_ids = sampleAnnotation.rnaIDs[group]
-    dna_ids = sampleAnnotation.dnaIDs[group]
-    sa_sub = sa.loc[(sa["RNA_ID"].isin(rna_ids)) & (sa["DNA_ID"].isin(dna_ids))]
+    sa_sub = sa.loc[sa["RNA_ID"].isin(rna_ids)]
 
     # save sample annotation subset
     sa_sub.to_csv(sa_file, sep='\t', index=False, columns=sa_cols)

setup.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.0.3
+current_version = 1.0.4
 commit = True
 
 [bumpversion:file:setup.py]
@@ -15,5 +15,5 @@ search = @click.version_option('{current_version}'
 replace = @click.version_option('{new_version}'
 
 [bumpversion:file:docs/source/conf.py]
-search = release = '{current_version}'
+search = release_ = '{current_version}'
 replace = release_ = '{new_version}'

setup.py

@@ -21,7 +21,7 @@
 
 setuptools.setup(
     name="drop",
-    version="1.0.3",
+    version="1.0.4",
     author="Michaela Müller, Daniela Klaproth-Andrade, Vicente Yépez, Christian Mertes",
     author_email="mumichae@in.tum.de",
     description="Detection of RNA Outlier Pipeline",