Cluster Flow v0.4

This is a huge release that got lost on it's way. A great number of things have changed since the
v0.3 release, so please be prepared for a bit of tinkering if you're updating.

I'll try to make releases smaller and more frequently in the future to avoid this happening again!

Download

Download Cluster Flow v0.4 (.zip)
Download Cluster Flow v0.4 (.tar.gz)

Changelog

• Warning: Break of backwards compatability
  • The way that genome references are handled has been rewritten.
    • Genome references are no longer tied to specific types, they are now agnostic.
      to the type of reference, making it far easier to whatever type of reference you need.
      Additionally, the wizard to add genome paths has been written and is now largely automated,
      making it super fast to add new genomes.
    • A consequence of this change is any genomes.config files written before v0.3 of
      Cluster Flow will no longer work. Thankfully the fix is easy! Replace @bowtie_path
      with @reference bowtie. @gtf_path changes to @reference gtf and so on.
      @genome_path changes to @reference fasta.
    • If you have any custom pipelines these will also need to be updated. @require_bowtie
      changes to @require_reference bowtieand so on. See updated example module files
      for examples on how to update custom modules.
    • Apologies for any inconvenience that this change incurs. Feel free to get in touch
      if you have any problems.
  • ~/clusterflow/ directory moved to ~/.clusterflow/ to reduce home directory clutter.
    • Cluster Flow won't find your old config file - run mv ~/clusterflow/ ~/.clusterflow/ to fix.
• New Stuff
  • New website! http://clusterflow.io
    • New home of cluster flow documentation
    • Docs are now versioned along with CF version.
    • Online interactive demo and other cool stuff. Code can be found at https://github.com/ewels/clusterflow-website
  • You can now run Cluster Flow locally (new @cluster_environment local )
    • Tested on Mac OSX and Linux. Includes --qstat and --qdel functionality
    • Allows easy testing and use of pipelines for those without access to a HPC cluster.
    • New --environment command line option allows you to set this at run time.
  • The --make_config wizard has been renamed to --setup and does a lot more stuff
    • Should make first-run of Cluster Flow much easier - just download and run cf --setup
  • Support for STAR RNA-seq aligner (thanks to @stu2)
  • New ChIP-Seq analysis pipeline (thanks to @orzechoj)
    • Includes six new modules: bedToNrf, bedtools_bamToBed, deeptools_bamCoverage, deeptools_bamFingerprint, phantompeaktools_runSpp and picard_dedup
  • Modules are given more information via the run file to help
    decide the amount of memory and cores they bid for (eg. number of files, reference)
  • Running job stats (cf --qstat) now has a summary at the bottom, and counts running and completed jobs.
  • New config option @time_multiplier to globally adjust time requests for slow clusters
    • To stop things getting silly, there's also @max_time to put a top limit on what's possible..
  • Added a special case value null for the @environment_module_alias config option, to prevent Cluster Flow from loading that specific module.
  • All perl scripts now have env perl in shebang to increase portability
  • Modules can now have file extensions, as long as they have .cfmod at the end of the basename
    • This helps editing tools with syntax highlighting, amongst other things
  • Python comes to Cluster Flow! The first Python module is up and running, along with a Helpers.py module file
    • See the example_module.py file for help in writing your own modules in Python
    • The basic Perl module helpers are now available in the Python packages as well, more translation to follow
  • Now using GRIDEngine h_vmem memory option instead of vf
    • Gives a hard memory limit instead of a request limit at job submission time
    • Thanks to @stu2 and @s-andrews
  • Support for explicit GRIDEngine queue nomination on the command line
  • Modules now print their software versions to the log where possible.
  • New --merge (command line) and @merge_regex (config file) options to automatically merge input files.
    • This is implemented using a new module, cf_merge_files, which can also be used in pipelines
    • If the supplied regexes only match single files, the module can be used to simply rename files
  • New --runfile_prefix option to help avoid potential filename clashes.
  • New @cluster_project config option to specify project for cluster jobs.
  • Added compatability with GRIDEngine ~/.sge_request files (by ignoring them).
    Thanks to @s-andrews
  • New tophat module called tophat which introduces a workaround for buggy MAPQ
    reporting by tophat whilst keeping unique alignments. Thanks to @FelixKrueger.
    • The previous tophat module is still available if you're not interested in MAPQ scores and
      would like slightly faster processing. This is now called tophat_broken_MAPQ.cfmod.
  • Pipeline completion e-mails are now written to disk as well (HTML and plain text)
  • New log file containing the job submission commands as well as the output received from the cluster at submission (usually numeric job identifiers)
  • Removed the --qstatcols command line option and added the @colourful config option to replace it
    • The config wizard is also updated to add this to your personal config
  • Added checks to make sure that we have at least one config file, and that the cluster environment is set
  • Added new @environment_module_always config option to always load certain environment modules at run time.
  • Added new @require_python_package pipeline option to check that a Python pacakge is installed before pipeline launch
• Bugs Squashed
  • Fixed output filename problem in tophat with output cleaning
  • Fixed bugs causing minimum memory allocation regardless of availability
  • Fixed bug causing Bowtie2 to fail if Bowtie1 index absent
  • Cleaned up some unrecognised output that always made it into the log file
  • Allow multiple merge regexes