-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement import of trait names from ClinVar #14
Comments
To clarify the scope here: as we discussed yesterday, this issue is specifically about backend logic to do the import. Import automation & manual triggering will be implemented separately. |
So just to clear up the process of selecting trait names and calculating the number of source records:
Also, we need to see what the behavior will be for already existing trait names whenever a new import cycle begins. Do we calculate the source records again for already existing trait names, and just insert the new ones as usual? |
Another thing, is that I am excluding "not provided" values for both trait name imports and source record number calculation, should I treat those records differently? |
That's correct, provided that the PhenotypeList column values are exploded prior to that—i.e., split by
That's correct. Just to clarify, after calculating tuples per each AlleleID, it's also important to combine all of them together and then do the deduplication (remember, AlleleID is not unique per row). To expand a bit on why we do this: the central object which is in the end submitted to Open Targets is an "evidence string". Each evidence string is defined by a tuple (trait, variant, ClinVar record). So the number of evidence strings generated by any given trait will depend on the total number of (variant, ClinVar record) tuples associated with it. This corresponds to (AlleleID, RCV) tuples in the source data.
Yeah, that's a good question. ClinVar has a number of trait "names" which cannot be mapped to any ontology term. Most notably "not provided", but there are also things like "see cases", "other" and a couple of others. In this ticket you don't need to address those situations in any special way. In future, we will need a possibility for curators to mark the trait as "invalid", probably necessitating an additional status. This is not a high priority issue. I've added it to backlog: #37 |
Regarding this, what I am doing right now is that If a record already exists in the database I leave it as is, and I only insert new ones with their source record numbers. Let me know if I should change this. Other than that, the script is ready. Should I add a button somewhere to trigger the import and submit it in a PR? |
Oh, I missed that question, sorry. The correct behaviour in this case is what you described originally: insert new trait names with their record counts + also recalculate number of linked records for all existing traits.
Yes, please do. I'm not sure where to put this button, though. There is an issue #32 for designing the page for buttons triggering various processes, but it's a separate one. So for now, I guess you could just add the button anywhere where it is convenient. And then, once #32 and subsequent implementation issues are done, we'll move it there. Another issue is logging. In case the import doesn't go through for some reason, we need a way to access the logs of the script. Do you have any ideas on how to do that best? For example, regarding the Heroku instances where all of this is running, do you have any way to access them (semi-) directly? Or could we make the script, once triggered, output its logs to the Javascript console or something? |
The easiest way to view logs would be to output them to the server console. They can then be accessed via the Heroku CLI or simply by going to the 'logs' page in the Heroku webpage. For example https://dashboard.heroku.com/apps/clinvar-trai-prototype-pmia54t/logs |
OK, that's great, let's use this way for now. Maybe in the future we'll implement some user friendly logging, a status page, email notifications, or something similar. For now Heroku server logs will do just fine |
Information about traits can be retrieved from a static, periodically updated endpoint https://ftp.ncbi.nlm.nih.gov/pub/clinvar/tab_delimited/variant_summary.txt.gz. There is an accompanying checksum file https://ftp.ncbi.nlm.nih.gov/pub/clinvar/tab_delimited/variant_summary.txt.gz.md5.
The main file is a gzipped TSV containing about two dozen columns (first line, starting with
#
, is a header). There are three columns of interest:These columns follow this list of constraints:
Trait names imported into the database should represent a unique set of all trait names present in this file.
To calculate the number of records assigned to each trait, we need to consider all tuples of (AlleleID, RCVaccession, PhenotypeList). For example, exploding along RCVaccession and PhenotypeList, the sample record described above would generate four tuples:
When we do this for all rows in the file, we need to combine all tuples together and deduplicate them (leave only unique tuples). In this final set, the number of tuples which mention a given trait name is the number of records associated with this trait, which should also be imported into the database and stored in a separate field.
The reason there can be non-unique tuples is because of different reference genomes. Sometimes there will be two records with the same AlleleID and with most columns containing identical or very similar values; the difference will only be in the reference genome (GRCh37/GRCh38) and chromosomal coordinates. This is why we do deduplication to count the number of records correctly.
Some considerations about the import:
Action checklist
The text was updated successfully, but these errors were encountered: