Skip to content

Topic 5 assignment

Jump to bottom
davidaray edited this page Oct 23, 2024 · 5 revisions

This is the most difficult assignment yet.

You have two options:

  1. Work solo.
  2. Work as a pair with a fellow student.

You will also have two weeks to work on this one with a troubleshooting class happening next week.

You will be provided with TE annotation files from multiple species' genome assemblies. Create a python script that will:

  1. Import each TE annotation file and convert it to a DataFrame

Input file format.

   SW  perc perc perc  query      position in query           matching       repeat              position in  repeat
score  div. del. ins.  sequence    begin     end    (left)    repeat         class/family         begin  end (left)   ID

  256  16.7  0.0  0.0  scaffold_m19_p_1       1      42 (223357944) +  MurFea-1.301   LTR                   2302 2343  (887)      1
 1705  12.5  0.8  5.0  scaffold_m19_p_1     287     344 (223357642) +  MurFea-1.321_mMyoMys1.1.hap1-2 LTR/ERVK                 1  334    (1)      2
  684  16.9  2.2 12.0  scaffold_m19_p_1     345     754 (223357232) +  sBil-1.315     LTR/ERV               2359 2732    (3)      3
19548   5.3  0.4  1.1  scaffold_m19_p_1     761    3748 (223354238) C  eFus.6.1557-   SINE/tRNA           (4985) 3923    956      4
 1010  10.0  6.1  0.5  scaffold_m19_p_1    3717    3743 (223354243) C  LasBor-1.248_mPleAur1.1.pri-2 DNA/TcMar-Mariner     (93) 1436   1246      5
 1015   4.6  6.5  0.6  scaffold_m19_p_1    3744    3893 (223354093) C  LasBor-5.32_mMyoMyo1.6.pri-7 SINE/tRNA             (15)  473    315      6
  714   2.3  0.0  0.0  scaffold_m19_p_1    3894    3979 (223354007) C  MurFea-5.1448  LTR/ERV1               (0) 3586   3501      7
  252  10.3  0.0  0.0  scaffold_m19_p_1    3950    3988 (223353998) C  LasBor-5.2021  LTR/ERV1              (61)  478    440      8
  441  21.4  0.8  5.7  scaffold_m19_p_1    4151    4279 (223353707) C  aPal.6.3571-   LTR/ERV1             (488) 6628   6506      9
  1. Replace the column headers provided with these: Score Pct_div Pct_del Pct_ins Query Start_in_query End_in_query (left_in_scaffold) Orientation TE_ID Class/Family Start_in_TE End_in_TE (left_in_TE) RM_ID
  2. create several new DataFrames consisting of insertion counts and mean divergences summarizing the data for all of them. They will eventually be output as .csv files.
  3. There should be one new .csv file. The final output should be .csv files with this format. Column 1, the species IDs (or filenames), should be in organized alphabetically.

image

Tip

Investigate the use of 'glob'. https://www.geeksforgeeks.org/how-to-use-glob-function-to-find-files-recursively-in-python/#

Also investigate the use of 'split()' to deal with the combined Class/Family column. https://www.w3schools.com/python/ref_string_split.asp

In a related manner, consider what to do if there is no value for 'Family' on any given row. Look at example 7 here: https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.Series.str.split.html

You might make use of 'skiprows' when importing your files. https://www.geeksforgeeks.org/how-to-skip-rows-while-reading-csv-file-using-pandas/

Consider whether the columns in the input file are delimited by tabs or something else. https://www.geeksforgeeks.org/how-to-read-space-delimited-files-in-pandas/

Clone this wiki locally