Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Welcome to the Introduction to Data Formats in Computationaly Biology exercises page!

These materials are part of the COST Project Epichembio Bioinformatics workshop: Introduction to NGS data analysis held in Badalona the 13th to 15th of March 2019.


  • EU Horizon 2020 COST Project Epichembio
  • IJC Carreras Foundation
  • Organizers: Sarah, Marguerite-Marie, David, Roberto
  • SIB Swiss Institute of Bioinformatics and University of Zurich


The course slides cover first why do we need scripted workflows with data stored using standards, in context of the best practices for scientific reproducibility; and after that discuss sequence formats (fasta, fastq), coordinate-based formats (BED, SAM, GTF, VCF..) and finally presents some tricks in efficience and speed-up (Wiggle files, indexed binary formats).

Slides are linked to exercises (see below) that largely differ in complexity.

Why? Good practices in computational biology

Computational biology needs high standards to make results reproducible, track work, organize projects and collaborate with colleagues.

To accomplish this we rely on sharing data and results using data standards (lingua franca) dealing with them using powerful and open tools (UNIX).

Recommended reads:


These exercises are meant to be run under a commodity computer (i.e. old laptop) running GNU/Linux or MacOS. To refresh your UNIX skills please run the tutorial at SIB Course on UNIX.

As a backup, the repository data folder has a backup of all datasets used, and the script to retrieve them.

Table of contents

Exercises complexity goes from low to challenging. Namely, from counting the number of lines of a fasta file or using the modulus operator to do so in FASTQ to dissecting chromatin compartmentalization trends in 1000 genomes project transposon variants. Please note gluttony is encouraged but exercises listing exceeds the workshop's workload.


For extra/advanced reading, please check the following tutorials on bash and awk:

Set up


Setting up a working directory with folders

cd ~                             # goes to the home directory
mkdir -p course
ls -l course

mkdir -p course/data course/soft # multiple directories can be created at once

ls -l course

Retrieving data

Downloading text files and checking their details.

cd  ~/course/data


file example.bed     # which kind of file is it, is it text?
ls -lah example.bed  # how big?
head example.bed     # how do the first lines look like?

file hg19.genome
ls -lah hg19.genome
head hg19.genome

Retrieving software/compiling (bedtools)

Retrieving source code and using a Makefile to compile exactly the software version required is a standard task in computational biology. Importantly, this can be scripted to make sure of dependencies/version. In this example we'll install bedtools (if not yet installed, which can be tested typing bedtools --help in a terminal).

mkdir -p course/soft

cd course/soft


tar zxvf bedtools-2.25.0.tar.gz
cd bedtools2
make  # will take time! you could read about Makefiles during compiling time
alias bedtools='~/course/soft/bedtools2/bin/bedtools'

bedtools --help


Exercise 1

Find the number of lines of the file ~/course/data/example.bed


Note the -l flag

wc -l ~/course/data/example.bed

Exercise 2

Get the manual page of head, what is this command for?


man head

Exercise 3

Get the first 5 lines of the file ~/course/data/example.bed


head -5 ~/course/data/example.bed

Exercise 4

Using pipes (|): chain the head -5 command before with wc -l to make sure the number of lines was as expected (5).

Seems a stupid thing to do? Actually it's a good idea to check some results during routine computational biology runs, i.e. whether the number of genes checked remains the same across samples and similar things. Negative and positive controls are easily transferable from wetlab to drylab read more about unit testing in Wikipedia.


head -5 ~/course/data/example.bed | wc -l

FASTQ/A Exercises

DISCLAIMER: data is based upon the beautiful Genome Analysis Workshop (MOLB 7621) Course on genome analysis at the University of Colorado SOM at

Exercise 5

Retrieve the data for these exercises (mind the path you're working at, should be ~/course/data).

The path for URL for the data is


cd ~/course/data

Exercise 6

(Source: MOLB7621)

Inspect the file from the previous exercise (file size, number of lines, and visualize its head)


cd  ~/course/data
ls -lh SP1.fq
wc -l SP1.fq

head SP1.fq

Exercise 7

(Source: MOLB7621)

The FASTQ file stored at ~/course/data/SP1.fq is in FASTQ format, meaning it contains 4 lines per read sequence.

head -n 4 SP1.fq


  • So Line 1 contains a sequence identifier that begins with an @
  • Line 2 contains the read sequence (A,T,C,G,N)
  • Line 3 is a comment line, often unused and only contains a +
  • Line 4 is the Phred quality score for each sequence encoded in ASCII format

First use wc and awk to determine the number of sequences in the fastq


cd ~/course/data/
wc -l SP1.fq  # total number of lines
wc -l SP1.fq | awk '{print $1 / 4}' # total number of fastq records

Exercise 8

(Source: MOLB7621)

A common mistake is to use grep to pattern match the @ in the sequence identifier. Why doesn't this work?

wc -l SP1.fq | awk '{print $1 / 4}'

renders 250

grep -c "@" SP1.fq

renders 459.


`@` is a valid quality score, so lines of phred scores will be counted as well when using grep

Exercise 9

(Source: MOLB7621)

Next, extract out the read sequences from the fastq. This is a bit complicated as we need to only pull out the second line from each record.


One approach to this problem is to use the % modulo operator (Wikipedia), which returns the remainder after division of two integers. For example using awk

awk 'BEGIN { {print 4 % 2}}'
awk 'BEGIN { {print 4 % 3}}'
awk 'BEGIN { {print 5 % 3}}'
awk 'BEGIN { {print 1 % 4}}'

In awk there is a special variable NR which is equal to the current line number.

So let's extract the sequences of the fasta file SP1.fq

awk 'NR%4==2'   ~/course/data/SP1.fq

Exercise 10

Print the line numbers of the file SP1.fq


awk '{print NR}' ~/course/data/SP1.fq

We can also prepend the line number to the FASTQ file (please note we asume SP1.fq is on the working directory)

Tip: awk '{print $0 }' filename prints the whole line, to which you can add the NR.

awk '{print NR, $0 }' SP1.fq | head # note output in first column

Exercise 11

(Source: MOLB7621)

We can use the modulo operator with NR to only capture specific records from the fastq

awk 'NR % 4 == 1' SP1.fq | head  # get header line
awk 'NR % 4 == 2' SP1.fq | head  # get sequence line
awk 'NR % 4 == 3' SP1.fq | head  # get comment line
awk 'NR % 4 == 0' SP1.fq | head  # get quality line

Exercise 12

(Source: MOLB7621)

FASTA format is a more compact sequence format than FASTQ


Convert fastq to fasta (tip: use lines 1 and 2)


awk 'NR % 4 == 1 {print ">"$1}; 
     NR % 4 == 2 {print}' SP1.fq \
     | head 

Exercise 13

Save the SP1 sequences as a file in fasta format (named ~/course/data/example.fa)


awk 'NR % 4 == 1 {print ">"$1}; 
     NR % 4 == 2 {print}' SP1.fq \
     > ~/course/data/example.fa

or, equivalently (data is one-column),

awk 'NR % 4 == 1 {print ">"$0}; 
     NR % 4 == 2 {print}' SP1.fq \
     > ~/course/data/example.fa

Exercise 14

Make sure the number of sequences is the same in both fastq and fasta files


awk 'NR % 4 == 1' SP1.fq | wc -l
awk 'NR % 2 == 1' example.fa | wc -l

SAM format

Exercise 15

Download the compressed SAM data from, uncompress (.gz files need to be uncompressed with gunzip) and inspect it.


cd ~/course/data


head chr22_with_ERCC92.gtf

wc -l chr22_with_ERCC92.gtf

BED format

Exercise 16

Transform the file ~/course/soft/bedtools2/test/intersect/a.bed (BED6) to BED3 (tip: use awk to extract the first three columns).

Tip: OFS='\t' in awk specifies the output field separator: a tab (\t).


## to go to the a.bed file directory
cd ~/course/soft/bedtools2/test/intersect/

awk -v OFS='\t' '{print $1,$2,$3}' a.bed

cd - ## to go back to the previous directory

Exercise 17

Transform the BED3 file ~/course/soft/bedtools2/test/intersect/recordsOutOfOrder.bed to BED6 (unspecified strand, 0 score)

Tip: check the BED6 file specification


## to go to the a.bed file directory
cd ~/course/soft/bedtools2/test/intersect/

awk -v OFS='\t' '{print $1,$2,$3,".",0,"."}' \

cd - ## to go back to the previous directory

Exercise 18

Add a nucleotide to the start and subtract a nucleotide to the end to all records, regardless of the strand, to the file ~/course/soft/bedtools2/test/intersect/a.bed.

Tip: use awk.


## to go to the a.bed file directory
cd ~/course/soft/bedtools2/test/intersect/

awk -v OFS='\t' '{print $1,$2-1,$3+1,$4,$5,$6}' a.bed

cd - ## to go back to the previous directory

Exercise 19

Use bedtools intersect to find overlapping genomic features. The BEDtools paper is also very helpful to understand the genomic arithmetics analysis (without sequences, using coordinates).

For this and the next exercises we use some example datasets from the bedtools sourcecode we compiled in one of the first activities. For instance, you could check them i.e. at ~/course/soft/bedtools2/test/intersect/.

Check the ~/course/soft/bedtools2/test/intersect/a.bed and ~/course/soft/bedtools2/test/intersect/b.bed files content. Are they BED3, BED6 or BED12 files?


cat ~/course/soft/bedtools2/test/intersect/a.bed
cat ~/course/soft/bedtools2/test/intersect/b.bed

Exercise 20

What will happen if you intersect those files? For example, the a.bed region chr1:100-200 overlaps with b.bed

alias bedtools='~/course/soft/bedtools2/bin/bedtools'

bedtools intersect \
  -a  ~/course/soft/bedtools2/test/intersect/a.bed \
  -b  ~/course/soft/bedtools2/test/intersect/b.bed

Which is the format of the output? How are results interpreted?


The output is a direct intersect:

chr1	100	101	a2	2	-
chr1	100	110	a2	2	-

Starting from the original interval from a.bed:

chr1        100     200     a2      2       -

And the overlapping intervals from b.bed:

chr1        90      101     b2      2       -
chr1        100     110     b3      3       +

Exercise 21

What does it happen if running bedtools intersect with the a.bed file as -b and the b.bed file as -a?


bedtools intersect \
  -b  ~/course/soft/bedtools2/test/intersect/a.bed \
  -a  ~/course/soft/bedtools2/test/intersect/b.bed

Tip: check the strands

Exercise 22

Run bedtools intersect with the a.bed file as -a, the b.bed file as -b but forcing strandness, i.e. reporting hits in B that overlap A on the same strand


bedtools intersect \
  -s \
  -a  ~/course/soft/bedtools2/test/intersect/a.bed \
  -b  ~/course/soft/bedtools2/test/intersect/b.bed

Exercise 23

Use the -v to report those intervals in a.bed that do not overlap any of the intervals in b.bed.


bedtools intersect \
  -v \
  -a  ~/course/soft/bedtools2/test/intersect/a.bed \
  -b  ~/course/soft/bedtools2/test/intersect/b.bed

You can explore what happens if inverting the -a and -b flags; for this the -wa and -wb flags might be helpful

bedtools intersect \
  -v \
  -wa -wb \
  -b  ~/course/soft/bedtools2/test/intersect/a.bed \
  -a  ~/course/soft/bedtools2/test/intersect/b.bed

Exercise 24

Use the -wao flag to report the amounts of overlap for all features when comparing a.bed and b.bed, including those that do not overlap. How are non overlaps encoded? Which strand are they on?


bedtools intersect \
  -wao \
  -a  ~/course/soft/bedtools2/test/intersect/a.bed \
  -b  ~/course/soft/bedtools2/test/intersect/b.bed

Exercise 26

Retrieve the details of transcript ENST00000342247 (tip: use grep) from the chr22_with_ERCC92.gtf file. Then, retrieve the details of the exons of transcript ENST00000342247 (tip: use grep after the grep). How many exons are they?


cd ~/course/data

grep ENST00000342247 chr22_with_ERCC92.gtf | grep "exon\s"

grep ENST00000342247 chr22_with_ERCC92.gtf | grep "exon\s" | wc -l

Exercise 27

How many start codons and stop codons does the chr22 have? Tip: use the gtf and grep for start_codon and stop_codon.


cd ~/course/data

grep start_codon chr22_with_ERCC92.gtf | wc -l

grep stop_codon chr22_with_ERCC92.gtf | wc -l


Exercise 28

Install an old version of VCFtools (disclaimer, this is old! this version is ok for teaching purposes, but please consider installing an up-to-date verson for handling your data in the future) you can download from using make. The path you can choose, but you can use for instance ~/course/soft/.


cd ~/course/soft/


tar xzvf vcftools.tar.gz

cd vcftools_0.1.13/


Exercise 29

Check which is the current version of VCFtools? How should installation be carried out? (tip: Git repositories release tagged versions).


Git repository at

There is a tag named v0.1.16 (as for the 26th Sept 2018)

Exercise 30

VCFtools has some data for testing purposes; find all the VCF files (filenames similar to *vcf*) that you downloaded during installation and inspect them using head.


find ~/course/soft/vcftools_0.1.13 -name "*vcf" -type f

Exercise 31

Alias the vcftool binary (full path) to vcftools and then run it with no parameters


alias vcftools='~/course/soft/vcftools_0.1.13/bin/vcftools'

Should render

VCFtools (v0.1.13)
© Adam Auton and Anthony Marcketta 2009

Process Variant Call Format files

For a list of options, please go to:

Questions, comments, and suggestions should be emailed to:

Exercise 32

How many variants are kept after filtering at ~/course/soft/vcftools_0.1.13/examples/merge-test-a.vcf? Tip: use the --vcf flag to vcftools. What does this result mean?


vcftools --vcf ~/course/soft/vcftools_0.1.13/examples/merge-test-a.vcf

Real data integration

Exercise 33

Explore human variation data for mobile elements insertion using the 1000genomes data. More precisely, download the pilot pilot data named CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf.gz (the full path to the remote FTP server is

Then, explore the number of variants (tip: use vcftools), the file contents etc.

Tip: remember where the bedtools and vcftools binaries are and/or use an alias to them.

For further details on mobile elements insertion as a source of human variation please check the paper Stewart et al 2011.


cd ~/course/data
curl -L > CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf.gz

gunzip CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf.gz 

wc -l CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf 

vcftools --vcf CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf

Exercise 34

As a first exercise, how many of these intertions overlap human exons? Do you expect it to be a low or high proportion?

To do so download an exons bedfile from and then use bedtools intersect. Tip: it won't work because of the chromosome numbering (tip: check how chromosome numbers are encoded in both exons.bed and the vcf file, i.e. whether starting with a chr or not, and harmonize them using awk).


cd ~/course/data

curl -O

bedtools intersect -a CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf \
  -b exons.bed

## this does not work as intended! or, rather, gives some warnings.
## Why? The chr strings at the coordinates are present in one bedfile but not in the other

tail CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf

tail exons.bed

## removing the chr string from the exons bedfile

sed 's/^chr//g' exons.bed > exons_nochr.bed

head exons_nochr.bed

## this should work (chr naming conventions are equivalent)
bedtools intersect -a CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf \
  -b exons_nochr.bed

Are these few or lots? What would you expect?

bedtools intersect -a CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf   -b exons_nochr.bed | wc -l

Exercise 35

Continuing with the mobile elements insertions, where do they insert preferentially? In active or inactive regions? To briefly explore this get the chromatin states with the highest number of insertions using the CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf and the Ernst's genome segmentation (more info here) as downloaded from Tip: then select the fourth column of the ChromHMM bedfile, sort it and count with uniq -c.

For further details on what is a ChromHMM segmentation (i.e. assigning functions to genomic ranges) please read Nature Methods volume 9, pages 215–216 (2012).


cd ~/course/data

curl -O

# first removing the chrstrings again

sed 's/^chr//g' hesc.chromHmm.bed > hesc.chromHmm_nochr.bed

bedtools intersect\
  -b CEU.low_coverage.2010_10.MobileElementInsertions.sites.vcf \
  -a  hesc.chromHmm_nochr.bed | awk '{print $4}' | sort | uniq -c

So apparently there is plenty of heterochromatin/low CNV regions, but is there an overrepresentation? How many heterochromatin/low CNV regions are there in the whole genome?

awk '{print $4}' hesc.chromHmm_nochr.bed | sort | uniq -c

Exercises block 36

Play with other real genomic data using the BEDtools tutorial which explores the Maurano et al paper Systematic Localization of Common Disease-Associated Variation in Regulatory DNA published in Science, 2012.

Mind that the tutorial recommends creating a folder with mkdir -p ~/workspace/monday/bedtools: if you do so and move (mv) there, your path (the one you can get using pwd) won't be at the standard ~/course we used till now.

Remember that the bedtools binary can be aliased using alias bedtools='~/course/soft/bedtools2/bin/bedtools'.

Some of the puzzles include:

  • Count the number of exons and CpG islands

Count the number of lines using `wc -l exons.bed` etc.

  • How many CpG islands overlap exons in the genome?

Intersect and count

bedtools intersect -a cpg.bed -b exons.bed   | wc -l

  • Create a BED file representing all of the intervals in the genome that are NOT exonic.

Use intersect with the -v flag

  • What is the average distance from GWAS SNPs to the closest exon?

(Hint - have a look at the closest tool.)

  • What fraction of the GWAS SNPs are exonic?

Exercises block 37

Explore (not necessarily run) more usage examples with biological meaning using UNIX and BEDTools

Exercise 38 Sum up

Read an overview of all the file formats available at UCSC. You can upload some of the files you used during the course to the UCSC genome browser to see how do they look like. Anyway you can download, read, edit them using UNIX (i.e. awk, grep etc). And ask biological questions using bedtools/vcftools.

In this course we used plain text data standards. But faster, indexed versions of such formats exist: i.e. bigWig (for wiggle), BCF (for VCF), BAM (for SAM). For instance, you can read how indexing is carried out in bigWigs.


Hands-on training on data formats in bioinformatics




No releases published


No packages published