Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Fetching, parsing, and analyzing data from the 1000 genomes project.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Type||Name||Latest commit message||Commit time|
|Failed to load latest commit information.|
Last updated: 29 April 2016 The purpose of this code is to fetch, parse, and analyze data from the 1000 genomes project. Because the files are stored on the 1000genomes server, it does not make sense to store them locally as well (unless you have limited internet access). So this code fetches and unzips the files, extracts the necessary information, and gets rid of the local copy. It then cross-matches the extracted data (which person has which mutations), with the mutation's sift score (how bad it is). Then it performs a few analyses, including plotting. A more detailed description and a how-to is below. _________________________ Code in the repository: --- README (this document, right here) individuals.bash sifting.bash pair_overlap_mutations.py frequency_overlap_mutations.py Extra data in repository: --- populations/1kg_POP.txt for POP = ALL, AFR, AMR, EAS, EUR, SAS, GBR for definitions of each population, see: http://www.1000genomes.org/category/frequently-asked-questions/population each txt file has a list of individual names/codes that are in that population, with ALL including all of them. columns.txt with all the indivuals ids. Important information: 1. Four directories needs to be created: - data - code - analysis_1 - analysis_2 2. Inside 'data' directory, the '20130502' directory needs to be created: - data/20130502 3. Inside 'analysis_1' and 'analysis_2' directories, 'output_no_sift' and 'plots_no_sift' directories needs to be created: - analysis_1/output_no_sift analysis_1/plots_no_sift - analysis_2/output_no_sift analysis_2/plots_no_sift 3. All the scripts explained bellow need to be stored in the code directory 4. columns.txt needs to be stored in data and code directories _________________________ Step 1: ./download_individuals.bash -c <Number Chromosome> Step 2: ./individuals.bash -c <Number Chromosome> Step 3: ./download_sifting.bash -c <Number Chromosome> Step 4 ./sifting.bash -c <Number Chromosome> Step 5: python pair_overlap_mutations.py -c <Number Chromsome> -pop <POP: 0:ALL, 1:AFR, 2:AMR, 3:EAS, 4:EUR, 5:SAS, 6:GBR> e.g. python pair_overlap_mutations.py -c 1 -pop 6 Step 6: python frequency_overlap_mutations.py -c <Chromosome> -pop <POP: 0:ALL, 1:AFR, 2:AMR, 3:EAS, 4:EUR, 5:SAS, 6:GBR> e.g. python frequency_overlap_mutations.py -c 1 -pop 6 _________________________ download_individuals.bash --- To use: ./download_individuals.bash -c <Number Chromosome> download_individuals.bash downloads comprossed files from 1000 geneome (Phase3) by chromosome. _________________________ individuals.bash --- To use: ./individuals.bash -c <Number Chromosome> --- individuals.bash decompresses, fetches and parses the files previously downloaded by download_individuals.bash script. These files list all of the variants in that chromosome and which individuals have each one. Each individual is tested twice. If both tests have values greater than 0, then the individual is said to have that variant. The indices of these variants are then copied into files for each individual. This has to be done for each chromosome, which can individually take a few hours to half a day (as run on an 8GB core). To run them all in succession, use run_individuals.bash. It takes no arguments and runs chromosomes 1-22. _________________________ downlod_sifting.bash --- To use: ./download_sifiting.bash -c <Number Chromosome> --- download_sifting.bash downloads all the SIFT scores of all the variants as caluclated by the VEP in compressed files. _________________________ sifting.bash --- To use: ./sifting.bash <Number Chromosome> --- sifting.bash finds the sift scores of all of the variants, as calculated by the Variant Effect Predictor (more information: www.ensembl.org/info/docs/tools/vep/). The VEP has been run on the 1000 genomes data, but these files do not have the results linked to the individual patients. sifting.bash decompresses the download_sifting.bash files, fetches the release data for each chromosome. Not every variant has a sift score, so this script pulls out the variants that do. It copies the variant ids (the index for this code set, rs number, and within ENSEMBL), the sift score, and the phenotype. _________________________ pair_overlap_mutations.py --- To use: python pair_overlap_mutations.py -c <Number Chromosome> -pop POP --- The script measures overlap in mutations among pairs of individuals by population. Considering two people, if both people have a mutation in a given variant then they are considered to have an overlap. Individuals are paired randomly without replacement, giving N/2 pairs, with N being the total number of individuals. The script performs several analyses by using different configurations: a) modifying the number of pair of individuals: a.1 )all individuals a.2) half of individuals a.3) 100 random individuals) b) modifying the number of variants: b.1) only the variants with a score less than 0.05 b.2) all the variants. This analysis is run 100 times to reduce any signal from accidental pairings between family members. These results are then plotted as a histogram of the number of pairs (y-axis) and the pairs are also output as text files named. As an output, this script generates: a) Text files (stored in the 'analysis_1/output_no_sift' directory) a.1)individual_half_pairs_overlap_chr<NumChromosome>_s<SIFT_Level>_<POP>.txt a.2)total_individual_pairs_overlap_chr<NumChromosome>_s<SIFT_Level>_<POP>.txt a.3)100_individual_overlap_chr<NumChromosome>_s<SIFT_Level>_<POP>.txt a.4)random_mutations_individual<NumChromosome>_s<SIFT_Level>_<POP>_run_<NumRun>.txt b) Plot figures (stored in the 'analysis_1/plots_no_sift' directory) b.1)half_distribution_c<NumChromosome>_s<SIFT_Level>_<POP>.png b.2)total_distribution_c<NumChromosome>_s<SIFT_Level>_<POP>.png b.3)100_distribution_c<NumChromosome>_s<SIFT_Level>_<POP>.png b.4)colormap_distribution_c<NumChromosome>_s<SIFT_Level>_<POP>.png c) Another interesting files stored 'analysis_1/output_no_sift' directory c.1) map_variations<NumChromosome>_s<SIFT_Level>_<POP>.txt c.2) mutation_index_array<NumChromosome>_s<SIFT_Level>_<POP>.txt Note: This script works with the SNPs variations (rs numbers), and not with the gens ids (ENSEMBL), since we are interested to measure the overlapping of mutations (variations) of the genes. _________________________ frequency_overlap_mutations.py --- To use: python frequency_overlap_mutations.py -c <Number Chromosome> -pop POP The script asks measures the frequency of overlappings in mutations by selecting a number of random individuals and selecting all variants without taking into account their SIFT scores. For example, if variant 1, variant 20, variant 42, and variant 80 are the only variants which presents 3 overlappings among individuals, we could say that the frequency of 3 overlappings among that group of individuals is 4 mutations. This analysis has been repeated 100 times. As an output, this script generates: a) Text files (stored in the 'analysis_2/output_no_sift' directory) a.1)Histogram_mutation_overlap_chr<NumChromosome>_s<SIFT_Level>_<POP>_<NumRun>.txt b) Plot figures (stored in the 'analysis_2/plots_no_sift' directory) b.1)Frequency_mutations<NumChromosome>_s<SIFT_Level>_<POP><NumRun>.png c) Another interesting files stored in 'analysis_2/output_no_sift' directory c.1)map_variations<NumChromosome>_s<SIFT_Level>_<POP>.txt c.2)mutation_index_array<NumChromosome>_s<SIFT_Level>_<POP>.txt c.3)Mutation_overlap_chr<NumChromosome>_s<SIFT_Level>_<POP>_<NumRun>.txt Note: This script works with the SNPs variations (rs numbers), and not with the gens ids (ENSEMBL), since we are interested to measure the frequency in mutations (varaiations) of the genes. ---