v3 subset script #566
v3 subset script #566
Conversation
| vd.row_key, variant_qc_annotations | ||
| ) | ||
| ) | ||
| logger.info("Dropping VRS as it is VCF incompatible as structured.") |
There was a problem hiding this comment.
| logger.info("Dropping VRS as it is VCF incompatible as structured.") | |
| logger.info("Dropping VRS because its current structure is incompatible with VCF.") |
| vd = vd.annotate_rows( | ||
| n_unsplit_alleles=hl.len(vd.alleles), | ||
| mixed_site=(hl.len(vd.alleles) > 2) | ||
| & hl.any(lambda a: hl.is_indel(vd.alleles[0], a), vd.alleles[1:]) | ||
| & hl.any(lambda a: hl.is_snp(vd.alleles[0], a), vd.alleles[1:]), | ||
| ) |
There was a problem hiding this comment.
Why this extra step? Could you add a comment?
| if vds_output: | ||
| vds.write(f"{output_path}/subset.vds", overwrite=args.overwrite) | ||
|
|
||
| if vcf: # TODO: Do we want to add in sharded vcf delivery using n_partitions? |
There was a problem hiding this comment.
Maybe add an argument as an option?
But aren't we getting a VCF per Chr when subsetting?
| "--variant-qc-annotations", | ||
| nargs="+", | ||
| type=str, | ||
| choices=VARIANT_QC_ANNOTATIONS, |
There was a problem hiding this comment.
Is choices necessary here?
|
|
||
| def compute_partitions( | ||
| mt, entry_size=3.5, partition_size=128000000, min_partitions=20 | ||
| ) -> int: # TODO: move to gnomad_methods? |
There was a problem hiding this comment.
Yes, I think this function should be in gnomad_methods/utils.py. It's very useful.
| vds_output = args.vds | ||
| dense_mt = args.dense_mt | ||
| split_multi = args.split_multi | ||
| subset_call_stats = args.subset_call_stats |
There was a problem hiding this comment.
How is this used? It seems that we are still copying all info fields?
| if add_variant_qc: | ||
| logger.info("Adding variant QC annotations.") | ||
| vd = vd.annotate_rows( | ||
| **make_variant_qc_annotations_dict( |
There was a problem hiding this comment.
This function from v4 is using v4 release_sites HT, but raw_qual_hists is now under histograms, maybe copy that function here, and use v3 release_sites HT?
It will run into error otherwise.
This is the most updated subset script. I went ahead and removed subset specific callstats. We've deprioritized the old subset script for years and eventually moved it into a draft after it became outdated but we should get this in. This is the script, with the exception of the subset callstats removal, that was last run on v3 subset requests.