Skip to content

--num_shards and --postprocess_cpus do not limit CPU usage during the later stages of DeepVariant #1098

Description

@Roick-Leo

Question: CPU usage exceeds --num_shards and --postprocess_cpus in the later stage

Hi DeepVariant developers,

I am running DeepVariant and have a question regarding CPU resource usage.

I have explicitly set both --num_shards and --postprocess_cpus when running either run_deepvariant or run_pangenome_aware_deepvariant. However, during the later stage of the pipeline, I still observe that DeepVariant consumes almost all CPU threads available on the machine.

For example, I use parameters similar to:

run_deepvariant \
    --model_type=WGS \
    --ref=reference.fa \
    --reads=input.bam \
    --output_vcf=output.vcf.gz \
    --output_gvcf=output.g.vcf.gz \
    --num_shards=32 \
    --postprocess_cpus=8

Similarly, I observe the same behavior when using:

run_pangenome_aware_deepvariant

with --num_shards and --postprocess_cpus explicitly specified.

Expected behavior

I expected:

  • --num_shards to limit the parallelism of the main DeepVariant processing stage.
  • --postprocess_cpus to limit the number of CPUs used by the postprocessing stage.
  • The total CPU consumption to remain within the resource limits specified by these parameters.

Actual behavior

The initial stages appear to respect the configured number of shards. However, during the later stage of the pipeline, CPU utilization increases substantially and appears to use all available CPU threads on the machine, rather than being limited by --postprocess_cpus.

For example, on a machine with 256 CPU threads, I may observe close to 128 threads being actively used during the later stage, even though:

--num_shards=64
--postprocess_cpus=32

are specified.

I would like to understand:

  1. Which DeepVariant stage/process is responsible for this CPU usage?
  2. Does --postprocess_cpus actually limit all CPU-intensive operations in the postprocessing stage, or only specific operations?
  3. Are there other parameters or environment variables that need to be set to prevent subprocesses or libraries (e.g. OpenMP, Eigen, BLAS, etc.) from using all available CPU threads?
  4. Is this expected behavior in the current DeepVariant version?

Environment

  • DeepVariant version: 1.0.9
  • run_deepvariant or run_pangenome_aware_deepvariant
  • Reference: GRCh38 / CHM13 / pangenome reference
  • Input: WGS BAM
  • CPU: 256 threads
  • RAM: 1 TB
  • Container: Docker
  • Docker CPU limit: not set / --cpus XXX

I can provide the complete command line, Docker command, process/thread information, and logs if needed.

Thanks for your help!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions