Handling large inputs
The Variant Transforms pipeline can process hundreds of thousands of files, millions of samples, and billions of records. There are a few settings that may need to be adjusted depending on the size of the input files. Each of these settings are explained in the sections below.
/opt/gcp_variant_transforms/bin/vcf_to_bq ... \ --max_num_workers <default is automatically determined> \ --worker_machine_type <default n1-standard-1> \ --disk_size_gb <default 250> \ --worker_disk_type <default PD> \ --keep_intermediate_avro_files \ --sharding_config_path <default gcp_variant_transforms/data/ sharding_configs/homo_sapiens_default.yaml> \
By default, Dataflow uses its autoscaling algorithm to adjust the number of
workers assigned to each job (limited by your Compute Engine quota).
You may adjust the maximum number of workers using
You may also use
--num_workers to specify the initial number of workers
to assign to the job.
By default, Dataflow uses the
n1-standard-1 machine, which has 1 vCPU
and 3.75GB of RAM. You may need to request a larger machine for large
datasets. Please see this page
for a list of available machine types.
We have observed that Dataflow performs more efficiently when running
with a large number of small machines rather than a small number of
large machines (e.g. 200
n1-standard-4 workers instead of 25
n1-standard-32 workers). This is due to disk/network IOPS
(input/output operations per second) being limited for each machine,
especially if merging is enabled.
Using a large number of workers may not always be possible due to disk
and IP quotas. As a result, we recommend using SSDs
when choosing a large (
n1-standard-16 or larger) machine,
which yields higher disk IOPS and can avoid idle CPU cycles.
Note that disk is significantly cheaper than CPU, so always try to optimize
for high CPU utilization rather than disk usage.
By default, each worker has 250GB of disk. The aggregate amount of disk space from all workers must be at least as large as the uncompressed size of all VCF files being processed. However, to accommodate for intermediate stages of the pipeline and also to account for the additional overhead introduced by the transforms (e.g. the sample name is repeated in every record in the BigQuery output rather than just being specified once as in the VCF header).
In addition, if merging is enabled, you may need more disk space per worker (e.g. 500GB), as the same variants need to be aggregated together on one machine.
SSDs provide significantly more IOPS than standard persistent disks,
but are more expensive. However, when choosing a large machine
n1-standard-16), they can reduce cost as they can avoid idle
CPU cycles due to disk IOPS limitations.
As a result, we recommend using SSDs if merging is enabled: these operations require "shuffling" the data (i.e. redistributing the data among workers), which require significant disk I/O. Add the following flag to use SSDs:
Adjusting Quotas and Limits
Compute Engine enforces quota on the maximum amount of resources that can
be used at any time, please see
for more details. As a result, you may need to adjust your quota to satisfy
the job requirements. The flags mentioned above will not be effective if you
do not have enough quota. In other words, Dataflow autoscaling will not be
able to raise the number of workers to reach the target number if you don't
have enough quota for one of the required resources. One way to confirm this
is to check the current usage of your quotas. The following image shows a
Persistent Disk SSD in
us-central1 region has reached its
To resolve situations like this, increase the following Compute Engine quotas:
In-use IP addresses: One per worker. If you set
--use_public_ips falsethen Dataflow workers use private IP addresses for all communication.
CPUs: At least one per worker. More if a larger machine type is used.
Persistent Disk Standard (GB): At least 250GB per worker. More if a larger disk is used.
Persistent Disk SSD (GB): Only needed if
--worker_disk_typeis set to SSD. Required quota size is the same as
Persistent Disk Standard.
For more information please refer to
Dataflow quotas guidelines.
Values assigned to these quotas are the upper limit of available resources for
your job. For example, if the quota for
In-use IP addresses is 10, but you try
to run with
--max_num_workers 20, your job will be running with at most 10
workers because that's all your GCP project is allowed to use.
Please note you need to set quotas for the region that your Dataflow pipeline is running. For more information related to regions please refer to our region documentation.
Other options to consider
Running preprocessor/validator tool
Because processing large inputs can take a long time and can be costly,
we highly recommend running the
prior to loading the full VCF to BigQuery pipeline to check for any
invalid/inconsistent records. Doing so can avoid failures due to invalid
records and can save you time and money. Depending on the quality of the
input files, you may consider running with
get the full report. Running with this flag will take longer, but it is
more accurate and is highly recommended when you're not sure about the
quality of the input files.
Sharding the output significantly reduces the query costs once the data is queried in BigQuery. It also optimizes the cost and time of the pipeline. As a result, we enforce sharding for all runs of Variant Transforms, please see the documentation for more details.
For very large inputs, you can use
--sharding_config_path to only
process and import a small region of genomes into BigQuery. For example,
the following sharding config file produces an output table that only
contains variants of chromosome 1 in the range of
- output_table: table_name_suffix: "chr1_1M_2M" regions: - "chr1:1,000,000-2,000,000" - "1" partition_range_end: 2,000,000
Saving AVRO files
If you are processing large inputs, you can set the
--keep_intermediate_avro_files as a safety measure to ensure that the
result of your Dataflow pipeline is stored in Google Cloud Storage in
case something goes wrong while the AVRO files are copied into BigQuery.
Doing so will not increase your compute cost, because most of the cost
of running Variant Transforms is due to resources used in the Dataflow
pipeline, and loading AVRO files to BigQuery is free. Storing the
intermediate AVRO files avoids wasting the output of Dataflow. For more
information about this flag, please refer to the
importing VCF files docs.
The downside of this approach is the extra cost of storing AVRO files in a Google Cloud Storage bucket. To avoid this cost, we recommend deleting the AVRO files after they have been loaded into BigQuery. If your import job failed and you need help with loading AVRO files into BigQuery, please let us know by submitting an issue.