Terraform: concurrent scans race on the module cache, so a module can be dropped or read half-written

[wils0ns]

Description

remoteResolver.download (pkg/iac/scanners/terraform/parser/resolvers/remote.go) writes a remote Terraform module directly into its cache path, <tmp>/.aqua/cache/<key>, after os.RemoveAll(dst), with no lock and no rename. Two Trivy processes that need the same uncached module work on the same directory at once, and at least one usually loses the module. The cache is under the system temporary directory, so any two scans on one machine share it, such as parallel jobs on a CI runner.

The scan still exits 0 and reports nothing from the module. Most times it logs Failed to load module. Sometimes a process finds the directory while the other is still writing it: its debug log says Module resolved from cache, no error is logged, and the module contributes no findings. That happened in 4 of 40 runs of a wrapper tool that starts both processes, and it cannot be told from a clean result.

Desired Behavior

Each process gets the whole module, or a clear failure. Downloading into a temporary directory beside the cache path and renaming it into place when complete would do it: a reader then sees a finished module or none, and two downloaders cannot clobber each other.

Actual Behavior

Findings per directory over ten rounds, each with a new empty TMPDIR:

1 a=0 not-loaded b=48
2 a=0 not-loaded b=48
3 a=0 not-loaded b=48
4 a=48 b=0 not-loaded
5 a=48 b=0 not-loaded
6 a=48 b=0 not-loaded
7 a=0 not-loaded b=48
8 a=0 not-loaded b=0 not-loaded
9 a=0 not-loaded b=48
10 a=0 not-loaded b=0 not-loaded

Run one after the other, each directory reports 48 every time.

Reproduction Steps

mkdir a b
for d in a b; do
  cat > $d/main.tf <<'TF'
module "consul" {
  source  = "hashicorp/consul/aws"
  version = "0.11.0"
  ami_id  = "ami-00000000000000000"
}
TF
done
export TMPDIR=$(mktemp -d -p "$PWD")
trivy config --debug --format json -o a.json a > a.log 2>&1 &
trivy config --debug --format json -o b.json b > b.log 2>&1 &
wait
grep -c "Failed to load module" a.log b.log
grep -c '"ID"' a.json b.json

Target

Filesystem

Scanner

Misconfiguration

Output Format

JSON

Mode

Standalone

Debug Output

DEBUG	[module resolver] Resolving module	module="root" name="module.consul" source="hashicorp/consul/aws"
DEBUG	[module resolver] Trying to resolve module via cache	module="root" key="52124e60f968d234a454aacad26e607c"
DEBUG	[module resolver] Requesting module versions from registry using	module="root" url="https://registry.terraform.io/v1/modules/hashicorp/consul/aws/versions"
DEBUG	[module resolver] Found module version	module="root" version="0.11.0" constraint="0.11.0"
DEBUG	[module resolver] Requesting module source from registry	module="root" url="https://registry.terraform.io/v1/modules/hashicorp/consul/aws/0.11.0/download"
DEBUG	[module resolver] Module resolved via registry to new source	module="root" source="git::https://github.com/hashicorp/terraform-aws-consul?ref=e9ceb573687c3d28516c9e3714caca84db64a766" name="hashicorp/consul/aws"
DEBUG	[module resolver] Caching module	module="root" key="52124e60f968d234a454aacad26e607c"
DEBUG	[module resolver] Downloading module	module="root" source="git::https://github.com/hashicorp/terraform-aws-consul?ref=e9ceb573687c3d28516c9e3714caca84db64a766"
ERROR	[terraform evaluator] Failed to load module. Maybe try 'terraform init'?	module="root" loc="main.tf:1-5" err="failed to download: error downloading 'https://github.com/hashicorp/terraform-aws-consul?ref=e9ceb573687c3d28516c9e3714caca84db64a766': failed to remove .git entry in the destination directory <dir>/tmp.wSRVzCJj4X/.aqua/cache/52124e60f968d234a454aacad26e607c during git update"

Operating System

Ubuntu 24.04.5 LTS

Version

Version: 0.76.1

Checklist

The reproduction starts from an empty TMPDIR, where the module cache lives, so trivy clean --all does not bear on it, and it was not run.

[nikpivkin]

Hi @wils0ns , thanks for the report!

Track #11595