Fix inclusivity test

.circleci/config.yml

@@ -0,0 +1,142 @@
+version: 2.1
+
+commands:
+  # Run tests under BovTB-nf/tests/jobs/ and store artifacts
+  run-test:
+    parameters:
+      script:
+        description: This test calls 'bash -e tests/jobs/<< script >>'
+        type: string
+
+    steps:
+      - run: |
+          cd /BovTB-nf/
+          mkdir /reads/ /results/ /artifacts/
+          bash -e tests/jobs/<< parameters.script >>
+
+      - store_artifacts:
+          path: /artifacts/
+
+  build-and-push:
+    parameters:
+      tag:
+        description: Tag to push built docker image to
+        type: string
+
+    steps:
+      - checkout
+      - setup_remote_docker
+      - run: |
+          TAG=<< parameters.tag >>
+          docker build -t aaronsfishman/bov-tb:$TAG -f ./docker/Dockerfile .
+          echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
+          docker push aaronsfishman/bov-tb:$TAG
+
+# Run tests under BovTB-nf/tests/jobs/ and store artifacts
+jobs:
+  # Docker image containing the nextflow pipeline $CIRCLE_BRANCH
+  build:
+    docker:
+      # Circleci base ubuntu image
+      - image: &build_img cimg/base:2020.01
+
+    steps:
+      - build-and-push:
+          tag: $CIRCLE_BRANCH
+
+  deploy: 
+    docker:
+      # Circleci base ubuntu image
+      - image: *build_img
+
+    steps:
+      - build-and-push:
+          tag: latest
+
+  unittests:
+    executor: nf-pipeline
+    steps:
+      - run: |
+          cd /BovTB-nf/
+          python tests/jobs/unit_tests.py
+
+  tinyreads:
+    executor: nf-pipeline
+    steps:
+      - run-test:
+          script: "tinyreads.bash"
+
+  quality:
+    executor: nf-pipeline
+    parameters:
+      case:
+        type: string
+    steps:
+      - run-test:
+          script: "quality.bash << parameters.case >>"
+
+  lod:
+    executor: nf-pipeline
+    parameters:
+      case:
+        type: string
+    steps:
+      - run-test:
+          script: "lod.bash << parameters.case >>"
+
+  inclusivity:
+    executor: nf-pipeline
+    parameters:
+      case:
+        type: string
+    steps:
+      - run-test:
+          script: "inclusivity.bash << parameters.case >>"
+
+# Run a job on the current branch's Dockerfile
+executors:
+  nf-pipeline:
+    docker:
+      - image: "aaronsfishman/bov-tb:$CIRCLE_BRANCH"
+
+# Orchestrates the validation tests 
+workflows:
+  validation:
+    jobs:
+      - build
+
+      - deploy:
+          requires:
+            - build
+          filters:
+            branches:
+              only: master
+
+      - unittests:
+          requires:
+            - build
+
+      - tinyreads:
+          requires:
+            - build
+
+      - quality:
+          requires:
+            - tinyreads
+          matrix:
+            parameters:
+              case: ["low", "adequate"]
+
+      - lod:
+          requires:
+            - tinyreads
+          matrix:
+            parameters:
+              case: ["0", "1", "2", "3"]
+
+      - inclusivity:
+          requires:
+            - tinyreads
+          matrix:
+            parameters:
+              case: ["0", "1", "2", "3"]

.gitignore

@@ -0,0 +1,6 @@
+.nextflow.log*
+.nextflow/
+.DS_Store
+temp/
+work/
+Results*

Readme.md

@@ -1,10 +1,10 @@
 # **BovTB-nf**
 
-------------
+[![APHA-CSU](https://circleci.com/gh/APHA-CSU/BovTB-nf.svg?style=svg)](https://app.circleci.com/pipelines/github/APHA-CSU)
 
 This is the updated pipeline for APHA's processing of *Mycobacterium bovis* WGS data. BovTB-nf is designed to process a batch (1 or more samples) of paired-end fastq files generated on an Illumina sequencer. It will first remove duplicate reads from the dataset (FastUniq) and then trim the unique reads based on base-call quality and the presence of adapters (Trimmomatic). Reads are then mapped to the *M. bovis* AF2122 reference genome and variants called (bwa/samtools/bcftools).
 
-It has been built to run using nextflow, using standard bioinformatic tools for the most part. The external dependancies are:
+It has been built to run using [nextflow](https://www.nextflow.io/docs/latest/getstarted.html), using standard bioinformatic tools for the most part. The external dependancies are:
 -	FastUniq
 -	Trimmomatic
 -	bwa
@@ -13,34 +13,51 @@ It has been built to run using nextflow, using standard bioinformatic tools for
 -	Kraken2 (and database)
 -	Bracken
 
-## Installation
 
-Of course Nextflow itself is a prerequisite and should be installed as described in the [Nextflow Documentation](https://www.nextflow.io/docs/latest/getstarted.html)
+## Run pipeline in Docker
 
-If you have the dependancies installed the pipeline can run by simply typing: 
+We recommend running the pipeline with our docker image. 
 
-	nextflow run ellisrichardj/BovTB-nf
+To pull the latest image (if it's not already fetched) and run the nextflow container on data:
+```
+./bov-tb /PATH/TO/READS/ /PATH/TO/OUTPUT/RESULTS/
+```
 
-Alternatively, clone the repository:
+The `/PATH/TO/READS/` directory should contain fastq files named with the `*_{S*_R1,S*_R2}*.fastq.gz` pattern. For example, a directory with `bovis_S1_R1.fastq.gz` and `bovis_S1_R2.fastq.gz` contains a single pair of reads
 
-	git clone https://github.com/ellisrichardj/BovTB-nf.git
 
-If required, there is simple script for installing the dependancies (helpfully called Install_dependancies.sh), which will also update the nextflow config file with their locations.
+### Build image from source 
 
-# Docker build
+You can also build and run the image directly from source
+```
+docker build /PATH/TO/REPO/ -t my-bov-tb
+./bov-tb /PATH/TO/READS/ /PATH/TO/OUTPUT/RESULTS/ my-bov-tb
+```
 
-Alternatively, the pipeline can run in an ubuntu image on docker. 
-To build the image:
-`docker build /PATH/TO/REPO/ -t bov-tb`
 
-Run a docker container in bash:
-`docker run --rm -it bov-tb`
+## Validation
 
--------------
+The pipeline is validated against real-world biological samples sequenced with Illumina NextSeq machines at APHA. The test code for the validation tests is stored under `tests/jobs/`. A summary of each test is described below
 
-## Examples
 
-In its simplest form just run the Nextflow process from the directory containing the fastq files:
+### Quality Test
+
+The quality test ensures that low quality reads (<20) are not considered for variant calling and genotyping. This is performed by setting uniform quality values to a real-world *M. bovis* sample and asserting output. Low quality bases are removed from the sequence using `Trimmomatic`, which uses a sliding window that deletes reads when the average base quality drops below 20. A table of expected results is shown below.
+
+| Base Quality | Outcome | flag | group |
+| ------------- | ------------- | ------------- | ------------- | 
+| 19   | CheckRequired | LowCoverage | NA |
+| 20   | Pass | BritishbTB | B6-16 |
+
+### Limit of Detection (LoD)
+
+The limit of detection test ensures mixtures of M. Avium and M. Bovis at varying proportions give the correct Outcome. This is performed by taking random reads from reference samples of M. Bovis and M. Avium.
+
+
+| M. Bovis (%) | M. Avium (%) | Outcome |
+| ------------- | ------------- | ------------- | 
+| 100%   | 0% | Pass | 
+| 65%   | 35% | BritishbTB | 
+| 60%   | 40% | CheckRequired | 
+| 0%   | 100% | Comtaminated | 
 
-	cd /path/to/Data
-	nextflow run BovTB-nf