Annotate a VCF

Upload a VCF, have VariantGrid import and annotate any novel variants, then download the cohort-level annotated export (all samples, single-sample VCFs included).

Annotation can take a while - minutes to hours, depending on how many novel variants there are - so the pattern we recommend is submit now, download later: fire off your VCF(s), then come back and download once they're ready, rather than holding a process open the whole time. Uploads are keyed on the file's SHA-256, so the VCF itself is the receipt - there's no upload id or state file to keep, and you can submit and fetch from different machines.

From the command line

vg_api annotate_vcf is the quickest way in. The first call uploads the VCF; running the same command again downloads the annotated result once it's ready, or tells you it isn't yet. Submit now, come back later - from a terminal, a cron job, another machine, whatever:

$ export VARIANTGRID_API_TOKEN=YOUR_API_TOKEN         # or pass --token
$ vg_api annotate_vcf input.vcf.gz -o results/
Uploaded input.vcf.gz (id=13256).
Annotating input.vcf.gz - run the same command again later to download.

$ vg_api annotate_vcf input.vcf.gz -o results/        # a while later
input.vcf.gz: not ready yet (progress 40.0%) - run the same command again later.

$ vg_api annotate_vcf input.vcf.gz -o results/        # once it's done
Annotated vcf written to results/input.vcf_annotated_v254_GRCh38.vcf.gz

Exit codes make it scriptable: 0 downloaded, 3 pending, 1 error.

Options: --export-type vcf|csv, -o/--dest, --server (or $VARIANTGRID_API_SERVER), --token (or $VARIANTGRID_API_TOKEN). If you'd rather block until it's done, see Wait in one call.

From Python

Same idea from the library: upload each VCF, then later re-hash the same files and fetch whatever's ready. The server dedups on the file's SHA-256, so you store nothing. Here it is over a whole directory of VCFs:

import hashlib
from pathlib import Path
from variantgrid_api.api_client import VariantGridAPI

api = VariantGridAPI(server="https://variantgrid.com", api_token="YOUR_API_TOKEN")
vcf_dir = Path("/data/vcfs")


def sha256(path):
    """Content hash the server dedups on - same value as `sha256sum <file>`."""
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


# --- Submit: fire off every VCF in the directory, then walk away. Store nothing. ---
for vcf in sorted(vcf_dir.glob("*.vcf.gz")):
    api.upload_file(str(vcf), path=None)
    print(f"submitted {vcf.name}")

# --- Later (even on another machine): re-hash the same files, fetch whatever's ready ---
for vcf in sorted(vcf_dir.glob("*.vcf.gz")):
    status = api.poll_upload_status(sha256=sha256(vcf))
    if status.get("error"):
        print(f"{vcf.name}: ERROR - {status['error']}")
    elif status["annotation_complete"]:
        api.download_annotated(sha256=sha256(vcf), export_type="vcf", dest_path="/data/results/")
        print(f"{vcf.name}: downloaded")
    else:
        print(f"{vcf.name}: not ready yet ({status.get('progress_percent')}%)")

Re-run the second loop whenever you like - already-downloaded files just get fetched again, and anything still annotating reports its progress. Because the hash comes from the file bytes, sha256sum *.vcf.gz on the command line gives you the very same handles if you'd rather look them up by hand.

poll_upload_status() returns the raw status dict (annotation_complete, progress_percent, error, vcf_id, samples, ...) if you want to inspect progress directly. It accepts either sha256=<hash> or uploaded_file_id=<id> (the id is returned from upload_file()); the hash is the one that's stable across machines.

Wait in one call

For a quick interactive job you can just block until annotation finishes and download in one go - but remember this can take hours, so prefer the submit-now/download-later pattern above for anything large or batched.

From Python, annotate_vcf() chains upload → wait → download (it raises TimeoutError if it isn't done within timeout, default 3600s, so bump that for slow jobs):

path = api.annotate_vcf("input.vcf.gz", export_type="vcf", dest_path="/data/results/")

From the command line, add --wait:

$ vg_api annotate_vcf input.vcf.gz --wait -o results/
Uploaded input.vcf.gz (id=13256).
Waiting for annotation to finish - this can take a while...
Annotated vcf written to results/input.vcf_annotated_v254_GRCh38.vcf.gz

--wait uploads only if the server hasn't seen the file, then polls (--poll-interval, --timeout) and downloads.

Notes

• The annotate_vcf wrapper and vg_api already upload with path=None. Only pass a path to upload_file for SeqAuto uploads that link to a registered JointCalledVCF / SingleSampleVCF (that is what path is for - it is ignored on non-SeqAuto deployments).
• Include a full ##contig header in the VCF so the server can detect the genome build - a partial contig set can leave the build undetectable.
• export_type is vcf (gzipped *.vcf.gz) or csv (zipped *.csv.zip); the export covers all samples in the uploaded VCF.
• Downloads require the target deployment to have the cohort export analysis templates configured (ANALYSIS_TEMPLATES_AUTO_COHORT_EXPORT) - otherwise a clear error is returned.