Bazel runfiles run out space

[awgymer]

Describe the issue:
When running make_examples on an HPC with limited space in the TMPDIR there appears to be issues with the size of the Bazel runfiles

Setup

• DeepVariant version: 1.8
• Installation method (Docker, built from source, etc.): Singularity via the Google GPU docker
• Type of data: WGS

Steps to reproduce:

• Command:

time seq 0 35 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "hg38.fa" --reads "SAMPLE1.bam" --examples "tmp/make_examples.tfrecord@36.gz" --checkpoint "/opt/models/ont_r104" --alt_aligned_pileup "diff_channels" --call_small_model_examples --gvcf "tmp/gvcf.tfrecord@36.gz" --max_reads_per_partition "600" --min_mapping_quality "5" --parse_sam_aux_fields --partition_size "25000" --phase_reads --pileup_image_width "99" --norealign_reads --sample_name "SAMPLE1" --small_model_indel_gq_threshold "25" --small_model_snp_gq_threshold "20" --small_model_vaf_context_window_size "51" --sort_by_haplotypes --track_ref_reads --trained_small_model_path "/opt/smallmodels/ont_r104" --trim_reads_for_pileup --vsc_min_fraction_indels "0.12" --vsc_min_fraction_snps "0.08" --task {}

• Error trace:

checkdir error:  cannot create /mnt/tmp_local/lls_4772490/Bazel.runfiles_b6itfhwc/runfiles/com_google_protobuf/python/compatibility_tests/v2.5.0
                 No space left on device
                 unable to process runfiles/com_google_protobuf/python/compatibility_tests/v2.5.0/tests/__init__.py.
symlink error: No space left on device
/mnt/tmp_local/lls_4772490/Bazel.runfiles_rgk2ay_z/runfiles/com_google_protobuf/python/google/protobuf/pyext/_message.so:  write error (disk full?).  Continue? (y/n/^C) 
warning:  /mnt/tmp_local/lls_4772490/Bazel.runfiles_rgk2ay_z/runfiles/com_google_protobuf/python/google/protobuf/pyext/_message.so is probably truncated
symlink error: No space left on device

I checked the code but AFAICT the only reference in Python to Bazel is within call_variants and not make_examples

Is it possible to have Bazel write these files elsewhere? Is there any idea of how much space in TMPDIR these runfiles require?

[kishwarshafin]

@awgymer run your docker like this:

docker run -v /path/to/large/disk/docker_tmp:/mnt/tmp_local google/deepvariant:1.8.0 <your_command>

This problem is specific to docker mounting which by default mounts to tmpfs.

[awgymer]

I'm not running on docker, it's singularity but from the original Google docker image.

The problem is rather that /mnt/tmp_local/ is the local nodes TMPDIR and is correctly bound into the image but has only 1GB of usable space.

I could possibly override TMPDIR with some more permanent and larger space but then I would need to manage cleanup and it might also pose issues for other tools.

If I know how much space these runfiles can run to I might be able to increase on-node tmp space to accommodate this.

[kishwarshafin]

@awgymer can you paste the singularity command here?

[pichuan]

Hi @awgymer ,
See if this is helpful for you: https://github.com/google/deepvariant/blob/r1.9/docs/FAQ.md#tmpdir

[awgymer]

The command is the following, where TMPDIR is /mnt/tmp_local/lls_4790128/:

singularity exec --no-home --pid -B $TMPDIR --nv /<path to images>/docker.io-google-deepvariant-1.8.0-gpu.img

So I don't think this is an issue with the existing /mnt in the container as it is being overwritten by our $TMPDIR.

As I say, the main issue is that my provisioned to space under TMPDIR is too small but I don't know how large it would need to be as I'm not really sure what Bazel is really doing here.

[kishwarshafin]

@awgymer please see the text @pichuan shared https://github.com/google/deepvariant/blob/r1.9/docs/FAQ.md#singularity-related-questions

[kishwarshafin]

I am going to close this issue because there has been no activity. Please feel free to reopen if you have further questions.

[awgymer]

To follow-up:

I can obviously solve the problem by setting a custom TMPDIR which is mounted in using -B:

TMPDIR=/path/with/more/space singularity exec -B $TMPDIR ...

but since bazel cleans up its temp files after the process ends I wasn't able to ever catch how big they grew so I am still none-the-wiser about how much free space it actually needs.