0.112.0

Major changes are:

Added SV annotation using AnnotSV

• in addition to default genomecomb annotation
• option -extraannot AnnotSV

isoquant use

• fixed important bug in merging known and novel transcripts that could caused novel transcript counts to be overinflated sometimes

clair3 variant analysis

• new version 1.1.1
• now select model based on basecaller version info in the bam/cram (if available)

Added functionality

• cg select - added function exonlist
• cg mm2tsv (wip)
• cg tsv2mm (wip)
• cg var_apply (wip)

Full Changelog: 0.111.0...0.112.0

0.111.0

Major changes are:

Generic joint analysis support for isoform calling is now also supported for ont single cell analysis using the -iso_joint option. Isoquant has stricter requirements for the detection of novel isoforms than for known ones. This can cause novel isoforms to be missed in some samples. Joint analysis works by first analysing all samples separately, and than using the novel isoforms found in any of the samples as "known/reference" ones in a reanalysis (still per sample), causing them to be more likely found (and counted) even in these lower evidence samples. If you have many samples, the number of artefactual "novel" transcripts can become too large to use as a reference (performance wise). With the option -iso_joint_min only novel transcripts found in at least samples are used as a reference in stage2, reducing the number of artefacts used drastically.

The command sc_demultiplex was added for demultiplexing singlecell results into different samples, based on a demultiplexing file assigning each cell to sample. (Such a file can be generated based on genomic variants or feature barcode sequencing)

Some improvements in sc barcode detection were made:

• find_barcodes handles the presence of multiple adapter matches in a read better (mapquality filter, location)
• Support for v4 and 5 prime 10x barcodes has been added

Support for finding and annotating UMIs in bulk RNA-seq has been added, and can be used with the -addumis option. Analysis using isoquant will also take the UMIs into account when quantifying genes/isoforms.

An alternative cluster distribution method that limits the number of concurrently submitted jobs was added: When using -d slurm, all jobs will be submitted immediately (in a manner jobs will only be run when jobs they are dependent on are finished). This can (for large analyses) submit thousands of jobs, causing problemson clusters that limit the number of submitted jobs. You can now use the options -d -dsubmit slurm to run (distributed) on a slurm cluster, while limiting the number of concurrently submitted jobs to . The disadvantage of this method is that the submitting command has to keep running until the entire command/pipeline is finished (as it is managing the jobs partly by itself).

A custom set of regions for distribution can be added to a genomereference, and used by the option -distrreg g. This allows for optimization of parallel analysis by splitting up regions that take a longer time to process (due to e.g. size of gene richmess).

In several places support for cram has been added

• the longshot option for variant calling can now also produce a haplotyped cram
• tsv2sam and sam2tsv support cram
• cg_tsvdiff added the option -refseq needed for comparing some cram files

dirR improvements

• Added Rscript support to dirR
• make sure specific versions of packages are installed

Updates

• isoquant 3.6.3
• modkit 0.4.2
• clair3 1.0.10 (also tries to detect which model best to use)

Several new functions were added in cg select

• ljoin : combine elements of a list with a given separator
• percentsum : sum of field as percent versus total sum in given column
• gpercentsum : sum of field versus total sum in given group (row) as percent

Various other improvements

• Added ubam support in scywalker
• Added cg sc_analysis_basic: basic singlecell report even if some data is missing
• Added cg bam_index (atomic by making temp first, sets time to be same as bam file)
• Added convenience command cg rm (that properly deletes shadowdirs)
• cg tsv2fasta - added option -idfield and -sequencedfield, if not given, check for several fieldnames

Full Changelog: 0.110.0...0.111.0

0.110.0

Major changes are:

Support for analysis of PacBio RNA-seq data, especially single cell data,
easily usable via the new preset scywalker_pacbio

cg process_project and cg process_sample check a lot more parameters, and
return an error if they are wrong (e.g. given file does not exist), before
starting the actual run. (issue derijkp/scywalker#12)

Various memory/storage optimizations (to avoid slowdown due to swapping, or killed jobs for memory overuse)

• The option -dmaxmem was added to limit the total memory used (requested) when running local distribution.
• memory reservations adjusted, e.g. minimap2 request memory based on the
  index size, realign_gatk memory request was increased, and bam_clean request (enough) memory
• sorting (using gnusort8) compresses temporary files using zstd (to avoid filling up /tmp when sorting huge files).
  It also uses a larger block size for speed
• The -scratchdir option was added for when /tmp is too small on your system

Reference databases

• mm10 and mm39 have been updated, adding e.g. evaSNP
• makedbs/makedbs_tair10.sh was added to create the Arabidopsis reference
• The groupchromosomes option allows grouping of (smaller) chromosomes different from the default (chrX_XXX grouped under chrX_)
  This is usefull for genomes where naming follows a different pattern, e.g. the dual Hs and Mm genome
• minimap2 now checks the size of the genome when creating an index

Several smaller new features

• added cg bam2cram and cg_brams2crams (for bulk conversion of bam to cram)
• added cg samcat
• new options for cg qsub: -memlimit, -lang, -cores, -mem, -time, -priority, -submitoptions
• new options for cg qjobs: -s (-summary)
• cg tsv210x options: -round (to create integer matrix), -remdups (to remove duplicate lines)
• added cg download_bigbed

Many other fixes and improvements

• look for alternative fieldnames for gene in sc_filter_default and tsv210x (issue derijkp/scywalker#10)
• aligner_preset is added to analysisinfo (cg map)
• gzfiles and jobgzfile optimizations: use less globs/filesystem accesses, return values bsorted (per pattern)
• ...

Full Changelog: 0.109.0...0.110.0

0.109.0

Major changes are:

Qol improvements

• added cg project_make for creating a basic experiment/project directory based on a samplesheet
• Added option -samplesheet to cg_process_project (uses project_make internally) (issue derijkp/scywalker#3)
• Improvement help
• Added cg error_report for easier checking of errors after a distributed run (issue derijkp/scywalker#5 and derijkp/scywalker#4)

Distribution and installation

• Added cg install for easier install of supporting binaries and refdbs
• include more dependencies in distro (from testing in minimal docker distro)
  • less, which, bzcat, bzip2, dot
  • extra libs for dirR
  • semi-static compiles of samtools, bgzip, tabix

Updates of the reference databases (hg38)

• dbsnp 155
• gnomad 4.1
• gencode 45, ...

more cram support:

• cram can be used with ont presets (use -aliformat cram)
  • fix remora: modkit has problems with cram -> temp convert to bam to run
  • scywalker_report_job: support cram
• cg tsv2sam now supports -outformat cram

falco replaces fastqc
falco is a (mostly) drop-in replacement that is faster, smaller and comes with less dependencies than fastqc

Optimization cg select summaries

• by default making summaries (-g option) memory is proportional to size of output
  (other queries are independent of size)
• added option -optim memory
  • minimize memory use when making summaries with option -g
  • Using the option is slower, so do not use when (typically) small summary output is expected
  • used in scywalker for getting aggregate data from sc_gene and sc_isoforms

Optimizations (esp. for starting big runs)

• file access optimizations (check for compressed only where specified, direct deps check, use cached data)
• storage optimizations (shadowdirs)
• jobs optimizations (integrate cleanup, merge jobs, hold_jid in runfile)

Other changes

• Added extra cluster support (options -submitoptions, -dqueue, help, ...) (issue #1)
• process_project option -jobsample now allows specification of number of cores to be used by one-job sample analysis
• cg_cat: added option -fullfilename
• Changed preset_ont: removed npinv from default svcallers (as it often caused problems with newer data)
• cg fixtsv will also fix duplicate fields in tsv file

Full Changelog: 0.108.0...0.109.0

0.108.0

Major highlights of the update are:

• Addition of scywalker code for the analysis of single-cell ONT data
• added cg viz_transcripts for making a nice visualision of transcript usage
• ONT remora methylation support
• added phased output for clair3 (using longphase)
• added -iso_joint option : generic joint analysis support for iso calling
• added -preset option for easily setting multiple (default) options for specific analyses
• databases: added alphamissense annotation

Version opdates

• isoquant 3.3.0
• clair 1.0.4
• sniffles 2.2 (also added option -minsvlen defaulting to 35 (not 50) so as to be more like previous versions in this respect
• flair 2.0
• flye 2.9.2
• lumpy 0.3.1
• medaka 1.4.4
• nextflow 23.04.4

Some other changes/upgrades

• jobs report maxmem, pct_cpu, bytes_read, bytes_written
• ubam support
• added cg_10x2tsv and cg tsv210x
• html overview report plotly graphs can export svg
• cg multicount added options -limit_geneids, -clip_geneid_version, -empty
• cg keyvalue: added options -keyname and -valuename
• cg_select: added option -optim memory to run big output summaries using little memory (though much slower)
• optimization: shadowdirs
• cg checktsv -checksort option
• dirR updates/new libraries
• added loc_lt, loc_lte, loc_gt, loc_gte
• hisat2 binary

Full Changelog: 0.107.0...0.108.0

0.107.0

main changes

• improvements in ont rna-seq analysis
  • newer versions of isoquant, flames
  • fixes/improvements in additional processing (support presets, skip chrM, ambiguity calc, ..)
  • comparison of genes/isoform improvements: consistent and unique naming of novel transcripts, matching, ..
  • improved tsv2gtf conversions (more fields added)
• support for newer versions of minimap2, clair3, sniffles, ...
• updated databases (hg38)
• fixes (many for special case handling, like empty results)

Full Changelog: 0.106.0...0.107.0

0.106.0

tests fix: use mklink (to avoid ln errors)