Skip to content

Compare: Input data formats

New page
Showing with 212 additions and 74 deletions.
  1. +47 −0 Changelog.md
  2. +0 −15 Development.md
  3. +29 −0 FAQ.md
  4. +43 −8 Home.md
  5. +66 −0 How-To-Use.md
  6. +27 −38 Input-data-formats.md
  7. +0 −13 Roadmap.md
65 changes: 27 additions & 38 deletions Input-data-formats.md
View file
Original file line number Diff line number Diff line change
@@ -1,40 +1,32 @@
1. <a href="#trees">Trees</a>
1. <a href="#netadata">Metadata</a>
1. <a href="#annotations">Genome Annotations</a>
1. <a href="#blocks">Genomic data (blocks)</a>
1. <a href="#gwas">GWAS data</a>
Data formats in bioinformatics can be problematic so I have tried to make this detailed enough. While we try to identify errors upon parsing, this is hard to do. Please get in touch if your data doesn't work.

Data formats in bioinformatics can be problematic so I have tried to make this detailed enough. While we try to identify errors upon parsing, it's not perfect. Yet.
The files mentioned in this page are taken from the examples on [phandango.net](http://phandango.net). Please note that in most cases one cannot load individual files - for instance, metadata cannot be loaded without a phylogeny. **Individual files are referenced here for the sole purposes of understanding the required file formats.** The example datasets in their entirety are available [here](https://github.com/jameshadfield/phandangoExampleData).

All of the raw files which you can use as the online examples are available [here](https://github.com/jameshadfield/phandangoExampleData).


<a name="trees" />
### Phylogenies (Trees)
Phylogenies form the backbone of the visualization as they link together all the other data. While it is possible to use phandango without them for GWAS-type graphs, all metadata, recombination blocks and pan-genome content relies on them.
Phylogenies form the backbone of the visualisation as they link together all the other data. While it is possible to use Phandango without them for GWAS-type graphs, all metadata, recombination blocks and pan-genome content relies on them.

Trees must be in [Newick format](http://evolution.genetics.washington.edu/phylip/newicktree.html) and must end in *.tre* or *.tree*. Newick is the standard output from most tree drawing software (e.g. [RAxML](http://sco.h-its.org/exelixis/software.html)), but not all. If you need to convert your tree to a different format try using [FigTree](http://tree.bio.ed.ac.uk/software/figtree/) but watch out - often single quotations are added around taxon names which must be manually removed! Notably Nexus files are not supported.
Trees must be in [Newick format](http://evolution.genetics.washington.edu/phylip/newicktree.html) and must end in *.tre* or *.tree* ([example here](https://github.com/jameshadfield/phandangoExampleData/tree/master/PMEN1_recombination/gubbins.tre)). Newick is the standard output from most tree drawing software (e.g. [RAxML](http://sco.h-its.org/exelixis/software.html)), but not all. If you need to convert your tree to a different format try using [FigTree](http://tree.bio.ed.ac.uk/software/figtree/) but watch out - often single quotations are added around taxon names which must be manually removed! Regrettably Nexus files are not currently supported.

<a name="metadata" />
### Metadata
Metadata is displayed to the right of the tree (so a tree must exist!) and the taxon names here must match those in the tree. Which columns are displayed can be controlled in the settings menu, and a key can be displayed by pressing *k*.
Metadata is displayed to the right of the tree. A corresponding tree with matching taxon names must be loaded for the metadata to be displayed. Which columns are displayed can be controlled in the settings menu, and a key can be toggled by pressing `k`.

Format:

* comma seperated values (CSV) file ([example here](https://github.com/jameshadfield/phandangoExampleData/blob/master/gubbinsGono/ESC_resistance_groups.csv))
* comma separated values (CSV) file ([example here](https://github.com/jameshadfield/phandangoExampleData/blob/master/gubbinsGono/ESC_resistance_groups.csv))
* File ending in *.csv*
* The first line is used for the column headers
* The first column contains the taxon names, which must match those in the tree

Colour selection:

* The colour scale depends on the type of data in each column (binary, ordinal or continuous), which is inferred from the data, but this is far from perfect!
* Adding on *:o* or *:c* to the end of the name (in the first row) forces the choice to be ordinal or continuous, respectively.
* If you want multiple columns to use the same colours for the same values (e.g. so that the value *42* is the same colour in each column), then group these columns by adding an integer to the suffix - e.g. *:o1*. You can have as many groups as you like.
* Adding on `:o` or `:c` to the end of the name (in the first row) forces the choice to be ordinal or continuous, respectively. E.g. a header named `year:c` forces the colours to be drawn from a continuous scale.
* If you want multiple columns to use the same colours for the same values (e.g. so that the value `42` is the same colour in each column), then group these columns by adding an integer to the suffix - e.g. `:o1`. You can have as many groups as you like.
* You can specify your own colours as hex values in a separate column. The column header for the hex values must be the same as for the data with `:colour` attached, and to come after the data column. E.g. if you have a column named `year` you can add a second column titled `year:colour` containing hex values to use as colours.

<a name="annotations" />
### Genome Annotations
Annotations appear in the top right of the display and are nearly essential for interpreting recombination / GWAS results. They must be in [GFF3](http://gmod.org/wiki/GFF3#GFF3_Format) format and end in *.gff* or *.gff3*. Parsing GFF files is error prone so it's worth looking at an [example file](https://raw.githubusercontent.com/jameshadfield/phandangoExampleData/master/gubbinsNAR/Spn23f.gff), especially the first two lines:
Annotations appear in the top right of the display and are required for visualising recombination / GWAS results. They must be in [GFF3](http://gmod.org/wiki/GFF3#GFF3_Format) format and end in *.gff* or *.gff3*. Parsing GFF files is error prone so it's worth looking at an [example file](https://raw.githubusercontent.com/jameshadfield/phandangoExampleData/master/gubbinsNAR/Spn23f.gff), especially the first two lines:
```
##gff-version 3
##sequence-region <chromosome name> 1 <chromosome length>
Expand All @@ -48,42 +40,39 @@ Converting to GFF3:
Display:

* All of the semi-colon separated fields are read and displayed when you hover over a gene / region.
* If *colour* appears in the info field then genes are coloured similarly to Artemis.
* If `colour` appears in the info field then genes are coloured similarly to Artemis.

<a name="blocks" />
### Genomic data (recombination blocks, pan genome output)
Currently three different file types are parsed, but it shouldn't be hard to convert any block-like data into one of these formats.
Currently three different file types are parsed, but it shouldn't be too hard to convert any block-like data into one of these formats.

#### [Gubbins](https://github.com/sanger-pathogens/gubbins)
Gubbins output is in GFF3 format and must end in *.gff* or *.gff3*, similar to the genome annotation ([example here](https://github.com/jameshadfield/phandangoExampleData/blob/master/gubbinsNAR/EVAL.PMEN1.recombination_predictions.gff)). If you have an old gubbins output file (e.g. _*rec.tab_) then there is a simple python script [here](https://github.com/jameshadfield/phandango/blob/master/scripts/gubbins_tab2gff.py) which will convert it for you.
#### Gubbins
Gubbins output is in GFF3 format and must end in `.gff` or `.gff3`, similar to the genome annotation ([example here](https://github.com/jameshadfield/phandangoExampleData/blob/master/gubbinsNAR/EVAL.PMEN1.recombination_predictions.gff)). If you have an old gubbins output file (e.g. `*rec.tab`) then there is a simple python script [here](https://github.com/jameshadfield/phandango/blob/master/scripts/gubbins_tab2gff.py) which will convert it for you.
The Gubbins software is available [here](https://github.com/sanger-pathogens/gubbins)

Essential fields:

* The second field of each line (except the headers) must be *GUBBINS*, to distinguish these files from annotation GFFs.
* The semi-colon selerated info string (field 9) must contain the following strings *neg\_log\_likelihood*, *taxa* and *snp_count*
* Values are surrounded with double quotes, e.g. *snp_count="7";*
* The *taxa* field is a list of whitespace separated taxon names which must match taxa in the tree in order to be displayed.
* The second field of each line (except the headers) must be `GUBBINS`, to distinguish these files from annotation GFFs.
* The semi-colon separated info string (field 9) must contain the following strings `neg_log_likelihood`, `taxa` and `snp_count`
* Values are surrounded with double quotes, e.g. `snp_count="7";`
* The `taxa` field is a list of whitespace separated taxon names which must match taxa in the tree in order to be displayed.

#### [BRAT NextGen](http://www.helsinki.fi/bsg/software/BRAT-NextGen/)
A tab seperated txt file (i.e. ending in *.txt*) - this has a default file name like *segments\_tabular.txt* ([example here](https://github.com/jameshadfield/phandangoExampleData/blob/master/PMEN1_recombination/pmen1_segments_tabular.txt)).
#### BRAT NextGen
A tab separated txt file (i.e. ending in `.txt`) - this has a default file name like `segments_tabular.txt` ([example here](https://github.com/jameshadfield/phandangoExampleData/blob/master/PMEN1_recombination/pmen1_segments_tabular.txt)). BRATNextGen software is available [here](http://www.helsinki.fi/bsg/software/BRAT-NextGen/)

Format:

* The first line must be *
LIST OF FOREIGN GENOMIC SEGMENTS:*
* The first line must be `LIST OF FOREIGN GENOMIC SEGMENTS:`
* The second line (the header) is not used
* Subsequent lines have 6 fields corresponding to (1) block start co-ordinate (integer), (2) block end co-ordinate (integer), (3) origin cluster (integer), (4) home cluster (integer), (5) not used, (6) taxon name (string).

#### ROARY pan genome
The output file `gene_presence_absence.csv` is used and this contributes both the annotation data and the block data ([example file](https://github.com/jameshadfield/phandangoExampleData/blob/master/roary_S.Weltevreden/gene_presence_absence.csv)). The ROARY software is available [here](https://github.com/sanger-pathogens/Roary).

#### [ROARY pan genome](https://github.com/sanger-pathogens/Roary)
The output file *gene\_presence_absence.csv* is used and this contributes both the annotation data and the block data ([example file](https://github.com/jameshadfield/phandangoExampleData/blob/master/roary_S.Weltevreden/gene_presence_absence.csv)).

This CSV file is often huge and can cause browsers to crash. There is a simple python script [here](https://github.com/jameshadfield/phandango/blob/master/scripts/minimiseROARY.py) which minimises this file.
This CSV file is often huge and can cause browsers to crash. There is a simple python script [here](https://github.com/jameshadfield/phandango/blob/master/scripts/minimiseROARY.py) which minimises this file. It also seems to cause the output to SVG to crash - if it's this big then consider a screenshot instead!

<a name="gwas" />
### Scatterplots (Manhattan plots)
### Manhattan plots

GWAS results are in plink format, i.e. a tab deliminated file with header line similar to
GWAS results are in plink format, i.e. a tab delimited file with header line similar to
`#CHR SNP BP minLOG10(P) log10(p) r^2`

* The 3rd column is as the genome co-ordinate
Expand Down
Back to top