No space left on device when running atlantis plan

[jessebye]

We are running Atlantis on Fargate via this module. Recently, we started to get this error on running an atlantis plan:

running git clone --branch patch/rds_alerts_raw_delivery --depth=1 --single-branch https://atlantis-bot:<redacted>@github.com/<redacted>/terraform.git /home/atlantis/.atlantis/repos/<redacted>/terraform/293/default: Cloning into '/home/atlantis/.atlantis/repos/<redacted>/terraform/293/default'...
error: unable to write file _legacy/permissions/policies/README.md
error: unable to write file _legacy/permissions/policies/step_functions_start_stop_execution.tf
error: unable to write file _legacy/permissions/role_policy_attachments.tf
fatal: cannot create directory at '_legacy/permissions/roles': No space left on device
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

: exit status 128

This indicates that the instance is out of disk space, but I don't see any way to increase the disk size in the module inputs. What should we do to address this?

[gonzaclavell]

Hi, we are experimenting the same issue.
Is there any workaround like cleaning all plans which applies were just executed ?
Thx,

[alemairebe]

I believe this is more an atlantis question rather than linked to this terraform module.
The way we work around it is to define a custom workflow in your atlantis.yaml or repos_config.yaml.
We do this before any plan :
'find $(git rev-parse --show-toplevel) -type d -name '.terragrunt-cache' | xargs rm -rf'
It needs to be adapted to your use case as we use terragrunt.
see https://www.runatlantis.io/docs/custom-workflows.html#usage

[dmattia]

I solved this by adding the following to my Dockerfile:

ENV HOME "/home/atlantis"
ENV TF_PLUGIN_CACHE "$HOME/.terraform.d/plugins/linux_amd64"

# The providers/plugins you need may be different
RUN mkdir -p ${TF_PLUGIN_CACHE}
RUN curl https://releases.hashicorp.com/terraform-provider-local/1.4.0/terraform-provider-local_1.4.0_linux_amd64.zip \
  --output local.zip && \
  unzip local.zip -d ${TF_PLUGIN_CACHE}
RUN curl https://releases.hashicorp.com/terraform-provider-null/2.1.2/terraform-provider-null_2.1.2_linux_amd64.zip \
  --output null-provider.zip && \
  unzip null-provider.zip -d ${TF_PLUGIN_CACHE} 
RUN curl https://releases.hashicorp.com/terraform-provider-template/2.1.2/terraform-provider-template_2.1.2_linux_amd64.zip \
  --output template.zip && \
  unzip template.zip -d ${TF_PLUGIN_CACHE} 
RUN curl https://releases.hashicorp.com/terraform-provider-vault/2.9.0/terraform-provider-vault_2.9.0_linux_amd64.zip \
  --output vault.zip && \
  unzip vault.zip -d ${TF_PLUGIN_CACHE} 
RUN curl https://releases.hashicorp.com/terraform-provider-aws/2.53.0/terraform-provider-aws_2.53.0_linux_amd64.zip \
  --output aws.zip && \
  unzip aws.zip -d ${TF_PLUGIN_CACHE}

This solves the problem by making the .terragrunt-cache directories significantly smaller since they do not need to re-download providers all over the place

[bryantbiggs]

FYI - you can kill the Fargate task and that frees up the local 20gb that Fargate provisions. otherwise, this isn't something we are able to solve here in this module unfortunately

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.