-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dnaPipeTE estimates #22
Comments
This is how we compute % of annotated TEs from
i.e. the initial sampling is based on estimated genome size, but the % of TEs is estimated as "proportion of reads annotated as TEs", hence it's ploid/genome size agnostic. |
Re: Apis melifera: https://www.nature.com/articles/nature05260 Yes, it seems that there are 6 described transposons in Apis genome (< than 0.5% of the genome). Also verified with newer Apis melifera assembly: https://www.ncbi.nlm.nih.gov/assembly/GCA_003254395.2 (0.47%). And finally "Most of the groups of retrotransposable elements were detected in the genome of the honey bee. In comparison to many other organisms, the most striking difference is the extremely low diversity and abundance of these elements." from https://bmcgenomics.biomedcentral.com/articles/10.1186/1471-2164-15-86 As a sanity check I am rerunning some of the previous estimates (Anan1); I am also recalculating again A. ricciae with 1.3.1, P. fallax and M. belhari. (follow up on https://docs.google.com/document/d/1a8ER3bvDBn4wpehsAUN-s3QIEUwGSqlEYs_2ksWOkNA/edit?disco=AAAADaT02Ig) |
Yes, the read percentage TE caclulation is genome size independent. |
Patrick managed to run dnaPipeTE v1.2 that produces consistent estimates with the previous version. Thank you, Patrick! We got estimates for all three honey bees and Mbel, now rerunning A ricciae. If we get similar estimates we need to discuss the fragility of these results. |
Sorry folks for the delay. I updated the table with newer estimates of TEs using dnaPipeTE. The difference now is that I have used the "diploid" genomescope model to specify the genome size. The A. ricciae is still the same as before (0.7%), but A. vaga is now more similar (2.2%). I don't think there is anything we could do besides disclaiming that identification of TEs might be dependent on their frequency in the genome and phylogenetic distance to known TEs which is possibly causing this approach to identify the low copy-number TEs that are found in the assembly and previous literature. We also have there three new honey bees. The third one is a really strange sample too. The estimated genome size is substantially smaller than in the case of the other two bee strains. It does not seem like inference problem, it looks like there is something weird with the genome. I mean, we might remove this value for TEs. |
Hi Kamil,
No problem man!
Where is the table? in the Gdoc or somewhere on Github? Would be interesting to see the details. Also what was the Genomescope estimates for diploid genome size for the bdelloids? I have a feeling that this is at the root of this funny differences between Ar/Av... caused by something weird going on in the A. ricciae data, most likely
Apologies if I'm missing something obvious, but what is the denominator in this calculation? a few comments up you say it's "proportion of reads annotated as TE", but in the ms it seems to be related to assembly span.
I'm sure this is true, but then it is likely to be true for many of these species, not just the rotifers: Repbase is totally biased towards a select few model species (and is now behind a paywall.. 😮). I guess A. vaga is a bit of an outlier in that a huge effort has gone into manual curation of TEs found there - even A. ricciae, in the same genus (kinda), will have a whole bunch of stuff unique to it, but this doesn't explain why the discrepancy relative to the other Rotaria species. What do the raw TE counts look like, for your re-analyses? I can compare them to the data I have and see if anything jumps out. |
Also, @jensbast, I see now you asked a question on the comment that I only just saw - sorry!
I used a custom TE library from A. vaga (I guess this is what you worked on in 2013?), from Irina Arkhipova, combined with all known protostome TEs pulled from Repbase. Apparently, the Av TE library only partially made it into Repbase (the DNA TEs where never submitted). This could be a partial reason for the difference between Av and Ar, but again doesn't really explain why the Rotaria species are also quite high. I don't think that's the root cause here.
Perhaps... it's still a puzzle though. It's the difference between Ar and Av (.7 vs 2.2%) that bothers me the most. Maybe a supplementary that basically says "bdelloids are weird" will do the trick 😄 |
Here is the table: Within the table you also have the latest (diploid) genomescope estimates of haploid size (and if you would browse history you could dig out the previous models). |
dnaPipeTE outputs reads that are TEs and reads that are not. Therefore you divide by the total number of sampled reads. |
is that what is on the x-axis of Fig. 4? |
Yes. This is a
I process this file as described in this previous post: #22 (comment) And that's x-axis on the Figure 4. --- edit --- This is |
Hi both,
We exclusively based all TE identification on the DNApipeTE standard database, which is the repbase one (Yes, it is really annoying that it is behind a paywall now).
I would not. We can just say "rotifers are weird and say that these estimates depend on genome structure and so on, which is tricky (What Reuben said). Also sorry that I reads like we use the assembly span. I have to recheck, this is not true. |
during a meeting with Marc and Tanja we agreed that the best would be just to explain, why we think there is this difference between our and known estimates in Adinita. |
Hey, I updated the figure 4 in the google docs so I belive that everything is ready for finally fixing this paragraph. Is there a volunteer who will take it? |
Hi Kamil, I had a busy day today (we are moving the lab to Oxford!) but tomorrow I will have a bit of time to look at this - still a bit unsure about the dnaPipeTE estimates but I'll have a dig around tomorrow |
I can have a look, but I am still on holidays so I am a bit slow. Cheers Jens
|
I understand, but I am not sure what else we can do about it. I am open to suggestions though :-) |
Hi guys, Sorry for the delay. Here are a couple of thoughts!
These values in the
Which would suggest it is a sum of base pairs attributed to each TE class, rather than number of reads. I wasn't sure so I had another look at the relevant snippet of code in Then the last row ("Total") is computed as the total span of the fasta file at Thinking about it, it kinda shouldn't make a big difference to the overall results (i.e. the %'s you compute from Counts file) because the number of sampled bases should be proportional to the number of sampled reads (I think? unless crazy things like biases in read trimming might affect that), for given a genome size and coverage thresholds. As Jens already pointed out, probably more important is the effect of the |
Related to this point, I found an old
This gives the results (from R script above):
Thus the TE estimate for Ar is similar with your recent result for A. vaga, at ~2/3%. In this case, I simply used assembly span, ~175 Mb, for the I am basically wondering if the extreme peculiarity of the A. ricciae genome is throwing off the haploid genome size from Genomescope, which in turn is affecting the sampling regime in dnaPipeTE, which is leading to an underestimate of TE % in the Ar data. But I agree with what you said above Kamil: the best way forward would be to simply explain that somewhere in the ms, without changing Fig 4, which should remain as it is because it is consistent across the taxa. But better if we can have some evidence that we know Ar is more fragile to the |
I will read this in detail soon. For now:
Talk to you latter. |
I might have mixed up what counts mean for the DNApipeTE because I worked with PopoolationTE2 counts files as well (and other measurements of counts). I agree on rerunning with different genome sizes, but conceptually this is a bit tricky to explain, because for the other analyses we always used the genome size from kmer estimates (but maybe we can explain why it is different for Ar). Sorry, I am not at work this week, so I am a bit slow to respond and check stuff. |
No problem. Yes bases seemed more logical but I'm glad I looked at it because I was never exactly sure what was in the Counts.txt file.
I'll set this analysis running now, varying
I guess the point is that perhaps the dnaPipeTE approach might be more sensitive to this than I realised, if there is a correlation between PS I'm on holiday next week, but back the week after. It will take a few days to re-run dnaPipe anyway. |
Thinking more about it, it's wise to do the various `-genome_size. So, did it work @reubwn ? I don't mean to pressure you, just if dnaPipeTE would misbehave (again?) I could also try to rerun it on our cluster (I am waiting for P. fallax results anyway). |
Hey man, indeed it has finished, summary results below and also in full here in Google Sheet. I ran 3 genome sizes (87.5,175,262.5 Mb) each with 3 coverages (0.3,0.5,0.7). I based the genome sizes on the idea that 175 Mb (the assembly span) represents at least two subgenomes, so "true" genome size might vary in steps of ~87.5 Mb. The results show that TE "amount" (expressed either as absolute count or % total sampled bp) does systematically vary both across That's a bit worrying, I think? i.e. that the result should be so dependent on the specified input genome_size/coverage parameters. Hmmmmm... so it seems:
What do you guys think? Maybe this is just for Ar, I haven't looked at any other species.
|
Phew, that's really worrying... It's seems that it's simply a function of sampled nucleotides, look at this plot... I wonder if this is something related to the low frequency of individual TEs. And I wonder if it would get saturated. If we set genome size to 262.5Mbp and genome sampling to 3, 4, 5 or something that crazy. With other genomes we know that our estimates correlated with the published numbers (although they are bit higher), hence I don't think all dnaPipeTE estimates are bs... |
Yup.
I have wondered about that. But, I can't see how low copy number of genuine TEs (as is the case in A. vaga anyway) would lead to this correlation? I would have expected it to lead to systematic underrepresentation, maybe, but the correlation is weird. I also wondered if the ploidy is somehow messing with dnaPipeTE, e.g., if just normal tetraploid genes are being flagged as TEs somehow. It might be something like this, but the plots are from TEs annotated as LINES, LTRs etc - so they must have some basis of similarity to a TE in Repbase (I used Metazoa TEs as the
Doesn't look that way from the shape of the graphs! If anything it shows an exponential increase in TE discovery...
I can set this running, will takes ~3 days to run a full pipeline x3 (maybe more if coverage is so high).
That is good to know. But what is correlated exactly? The absolute counts compared to RepeatMasker estimates, or proportions? I can run the same experiment on a non-bdelloid, to test if just A. ricciae is totally whack - which would be the best one do you think? |
dnaPipeTE subsamples reads and assembles it with Trinity. If there is one TE per family in the genome, you would increase fraction of "assemblable" TEs by increasing coverage. (unlike in the case of abbundant TEs in the genome, when 0.5x should be sufficient to sample enough coverage to assemble them). At least this is how I understand dnaPipeTE and this is why I expect it to get saturated as some point. I do agree that these plots do not suggest that there will be any sarturation, but there should be.
That depends on studies. We correlated what was in papers with what we have measured. We did not check how exactly they estimated TEs in every study (because that varied a lot). It was just a sanity check.
I hoped to wrap it up with minimum number of computing hours, but it sounds like a reasonable thing to do. What about Leptopilina clavipes? Relatively small genome, sane TE content but not too small. |
Yes I see your point. But, then we should expect a linear increase with increased coverage, but instead we see an exponential increase. I wonder, is there a selection step in dnaPipeTE pipeline, where only reads above a certain abundance threshold are carried forward to assembly stage? If yes, then single-copy TEs might be "lost" because they never occur in the high-abundance selected reads. I looked again at the paper but it was unclear. Maybe tetra/polyploidy makes a difference too, because it will raise the genome-average abundance threshold even higher (i.e. even 'normal' genes are in 4x). Below are the 3 graphs for "Bases_per_component" for
Is there an easy way you can share the QC reads? I will set the program running on this, using same RM library etc. Let's hope it's just Ar that is the weird one. |
Unfortunately, I can not share them on the ftp anymore (my data were clansed and I archived only non-easily-recomputable bits). However, dl and clenaing should be rather straightforward.
|
I am not sure if I understand right the graphs, but maybe this is already good enough to open a ticket at their GitHub. This behaviour really needs some explanation. |
Me neither, but all I see is that the pink bit gets bigger when I don't think it should do that, as the x-axis is a proportion. Possibly I am just misunderstanding something. OK I will try and get it running on SRR7028347 over weekend so we can chat about it next week maybe. Yeah it's a bit worrying, but testing on a non-bdelloid is a good idea first imo. Bdelloids are weird, not really a good benchmark |
This looks so weird and it contradict benchmark they talk about it in their paper (in the supplement they say that the increasing coverage did not yield in higher fractions of annotated TEs, but it increased the computational time). i think it's the time to talk to developers of dnaPipeTE. I wonder if there is a way how to ratain sanity of the analysis. I will also once again do the figure of our dnaPipeTE estimates vs published values and post it here for completeness. |
Well, Clém said that our sampling is too high, not too low and he speculated that the saturation should happen only on the level of annotated TEs and that this was observed before. Could we check this out? If we see the pattern even if we exclude the unknown TEs. I would be happy to do that if you share me the Count file @reubwn . If that's the case, we could either exclude unknown from all the estimates and claim to report the "more active part of transposons" or we could rerun the estimates with smaller sampling values to avoid this strange behaviour (as also suggested by Clém). Not sure what is the right course of action here, but I feel we need to decide fast as this is kind of blocking the resubmission. |
The graphs I made above are already only for annotated TEs; unannotated, "na" and "Other" etc have already been excluded (see the data at the Google sheet, which has all the Counts.txt data in there too). I can run Lcla1 and Ar at lower coverage values to see if we can observe any saturation, this will be much quicker than before (it can be done by the end of the week). I would expect not however, because of above. If we do see it, we could quite easily re-run the analysis on the samples (e.g. I can help with the compute to speed things up). Alternative ideas:
Overall, my understanding is that we do not make a big conclusion from the TEs results, therefore I think it is perfectly OK to just explain the limitations here. Estimating TE content in these non-model organisms is (bloody) difficult and the results are a hypothesis; we do not claim to annotate every bp with complete accuracy. In that sense, so long as we are clear with our limitations and explain the caveats that we do already know about, I don't think this needs to be a major issue or stand in the way of resubmission for very long! (but I'm still interested to get to the bottom of it because I've been using dnaPipeTE a lot recently.) Just let me know how I can help. |
I agree, good to rerun with low coverage value. |
Cool, I'll run Aric1 and Lcla1 at 0.05, 0.1, 0.15, 0.2 and 0.25 now, at least that will give us a better idea of what's going on across the range of coverage values.
Yep, I think the approach is the right one for sure. I've got loads of data for a bunch of bdelloids on RM and dnaPipeTE, eventually I'll do some tests to see how much they correlate. But for this I think we stick with dnaPipeTE - it removes entirely the question of how each genome was assembled, which (hopefully) is a real strength of the comparative analysis. |
Alright. Then unknown do not explain (and sorry for not checking the sheet, I was a bit in rush when I tried to respond the last time).
This would be an easy way how if we would manage to phrase it well. |
Ha @reubwn, I found one little detail in the spreadsheet you shared. You used Ar total for Lcla fractions. When corrected, it looks like this. There is still the pattern, but the numbers are not THAT worrisome. P.S. The graphs you posted here are correct. Just the numbers in the Lcla1_simple in the sheet I shared are now corresponding with the figure. |
Weird... well spotted! But yeah, the graphs are correct - for Lcla1, genome size = 274 MB, we see an increase from 15.7% to 16.4% to 18.0% TEs across the coverage values 0.3, 0.5, 0.7. It's true that this is not such a huge increase. It's more the trend that is worrying - I guess there would always be a bit of variation (and we have only 3 data points). We'll know more when the lower coverage counts are ready. The new Google sheet seems to be different - can you give us edit access and I can put the new dnaPipeTE values into your version when the program has finished? |
Alright, one more question. Do you think we should the put investigation of this thread in supplementary materials? I guess it would not harm and show that we actually looked into it. However, it's yet another section. |
Hi Kamil, here are the results from the lower coverage runs:
Counts.txt data and code are attached below. |
It does not appear to me there either of the species is converging around any value. :-( -> I still have hard times figuring out how exactly to phrase it. I will try to get it done over the weekend. Thanks @reubwn for all this analyses! Although a bit frustrating, they were very helpful. |
Yes - although the blue one for Lcla1 seems to level off from 0.1 to 0.25, but then it climbs again.
No worries, and sorry that it's been frustrating (I can understand that!). I don't really know what to think about it - happy to Skype and chat more next week if you want. |
Alright, closing this issue for now |
A. melifera estimated with dnaPipeTE 1.3.1 leads to pracically no TEs. Is honey bee genome known to be TE-less?
M. belhari estimates I got have 0's everywhere
The text was updated successfully, but these errors were encountered: