redraft clinvar write

[MattWellie]

Fixes

• Closes Change Clinvar Hail Table generation to per-line JSON #174

Proposed Changes

• The initial version of the Clinvar summarising script created all the ~800k entries as a single JSON, converted that to a DataFrame, then to a HailTable. Somehow, converting 200MB of JSON into a Hail Table consumed 80GB+ of memory, and required a massive resource allocation
• Switching this to syntax already in use by @vladsavelyev (modelled on the original pipeline from Broad?), the data is written & ingested line-by-line on a fixed schema basis (See code here)
• This whole process now takes 3 mins, a trivial resource allocation, and is light enough that it could be run with each AIP run (i.e. pull down the latest clinvar data at runtime, and regenerate annotations on the fly). For reference this can be run on my local machine, whereas before it could barely run in GCP.

Notes

Conflicting and Unknown entries were previously removed, as they are not relevant to any AIP category, and removing them from consideration was the only way to get the process to complete. These could now be re-introduced if we feel like keeping all that data.

I'd also like to find a good place to stash this new table in cpg-reference-main so that the same data is available to all project buckets. I can see the following process working:

• Add this script in analysis-runner's scripts folder
• Copy latest the data from the NCBI FTP into the CPG-reference bucket
• Run to generate the new data, and write back into cpg-reference-main/test
• Re-summarised data is then available to all projects

Checklist

• Related Issue created
• Tests covering new change
• Linting checks pass