Error unsafe repository using EFS via terraform-aws-atlantis pre-v3.17.0

[chtakahashi]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I haven't seen this issue noted in here, but I am reporting this in case anybody else has experienced this.

After recently restarting our Atlantis task, each atlantis plan results in a failure to download every single module. One such failure message looks like this:

╷
│ Error: Failed to download module
│ 
│ Could not download module "cloudtrail" (cloudtrail.tf:12) source code from
│ "git::https://github.com/trussworks/terraform-aws-cloudtrail?ref=v4.3.0":
│ error downloading
│ 'https://github.com/trussworks/terraform-aws-cloudtrail?ref=v4.3.0':
│ /usr/bin/git exited with 128: fatal: unsafe repository
│ ('/******/.terraform/modules/cloudtrail'
│ is owned by someone else)
│ To add an exception for this directory, call:
│ 
│ 	git config --global --add safe.directory
│ /******/.terraform/modules/cloudtrail
│ .
╵

We understand this is related to the recent Git security vulnerability. We've tried adding the suggested git config --global --add safe.directory <repo> or to via * as documented here but have not had success.

Reproduction Steps

This problem began after we restarted our atlantis task in ECS. The git version running on the previous task was from a git version before the security patch.

Logs

Environment details

Atlantis version:

• terraform-aws-modules/atlantis/aws v3.14.0
• runatlantis/atlantis:v0.18.1 - https://github.com/transcom/circleci-docker/blob/master/milmove-atlantis/Dockerfile
  If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: No

Additional Context

[jamengual]

https://communities.sas.com/t5/SAS-Communities-Library/What-to-do-when-Git-reports-Fatal-Unsafe-Repository/ta-p/808910

[magistersart]

Experiencing the same issue with the same setup, but latest version of Atlantis

[rogeruiz]

Interesting @magistersart i can write up a bit of how we got this bug. We restarted our Atlantis instance hoping to speed up some slowness we were experiencing. I’m the update of Atlantis, we noticed that the latest Git version was getting updated through the package manager. This caused us to need the variable or configurations set but ultimately we had zero success there.

I looked through the Atlantis codebase to see where we can introduce a configuration check for the variables or the Git config. But I didn’t find anything at my first glance. But ultimately I think it’s something that’s solvable with a patch modifying how Atlantis runs Git commands.

[magistersart]

I had to reinstall everything with a rollback to v0.19.2 to make it working.
Adding git config --global --replace-all safe.directory '*' to workflows didn't help either.

[AlecBruns]

I am also having this issue. However it only occurs on atlantis apply. The plan command works as expected.

[mrmcgrew]

Experiencing this same issue with atlantis apply's. Tried using the git config --global --replace-all safe.directory '*' command in the workflow as well but currently having no success. I think at this point we may have to rollback versions, but continuing to investigate.

[AlecBruns]

I was able to fix this by rolling back to 3.0.0 for the server version

[andyshinn]

For those running https://github.com/terraform-aws-modules/terraform-aws-atlantis the fix for me was to set the user = "100:1000" which is the atlantis user in the official image. You will also need to mount the EFS directory externally and sudo chown -R 100:1000 <your_efs_mount>. By default, the EFS mount is root:root.

[mrmcgrew]

I was able to fix this by rolling back to 3.0.0 for the server version

I ended up doing this as well to get things working.

[biancarosa]

I was able to workaround this issue setting the user too. The EFS storage wasn't much of a need to me, and I couldn't easily mount it externally, so I ended up adding the following lines on my terraform resource (using terraform-aws-atlantis):

  # User, needed because of https://github.com/runatlantis/atlantis/issues/2221
  # This is atlantis user per the official docker image
  user = "100:1000"

  # Ephemeral Storage, needed because the EFS storage
  # gets created with root-owned directories
  enable_ephemeral_storage = true

[zepeng811]

I had the same issue happened in the latest Atlantis version v0.19.6 (that uses git version 2.34.2), but this issue did not happen when using Atlantis version v0.18.2 (that uses git version 2.34.1).

Although this post indicate the security vulnerability fix is added in git version 2.25.2, but from their release log it seems like it was addressed in 2.24.2 instead.

A temporary fix for this issue might be to downgrade the git version to 2.34.1 for the latest Atlantis image.

Another solution is to upgrade the git version to >= 2.35.2 so people can use git config --global --add safe.directory <repo>, as suggested here this command is only available after 2.35.2

[hungran]

thanks @andyshinn & @biancarosa
it was related to terraform-aws-modules/terraform-aws-atlantis#287
from the initial setup terraform-aws-atlantis
if you was not create user atlantis as uid = 100 & gid = 1000 by user = "100:1000"
EFS will mount with root user 1000:1000
we could use as below

  # User, needed because of https://github.com/runatlantis/atlantis/issues/2221
  # This is atlantis user per the official docker image
  user = "100:1000"

  # Ephemeral Storage, needed because the EFS storage
  # gets created with root-owned directories
  enable_ephemeral_storage = true

or

For those running https://github.com/terraform-aws-modules/terraform-aws-atlantis the fix for me was to set the user = "100:1000" which is the atlantis user in the official image. You will also need to mount the EFS directory externally and sudo chown -R 100:1000 <your_efs_mount>. By default, the EFS mount is root:root.

[nitrocode]

By the way the current git version is pinned here. We will most likely not downgrade.

atlantis/docker-base/Dockerfile

Line 31 in ad79d07

git=2.36.3-r0 \

The error seems to be resolved in terraform-aws-atlantis module's 3.17.0 release.

[jamesburriss]

thanks @andyshinn & @biancarosa it was related to terraform-aws-modules/terraform-aws-atlantis#287 from the initial setup terraform-aws-atlantis if you was not create user atlantis as uid = 100 & gid = 1000 by user = "100:1000" EFS will mount with root user 1000:1000 we could use as below

  # User, needed because of https://github.com/runatlantis/atlantis/issues/2221
  # This is atlantis user per the official docker image
  user = "100:1000"

  # Ephemeral Storage, needed because the EFS storage
  # gets created with root-owned directories
  enable_ephemeral_storage = true

or

For those running https://github.com/terraform-aws-modules/terraform-aws-atlantis the fix for me was to set the user = "100:1000" which is the atlantis user in the official image. You will also need to mount the EFS directory externally and sudo chown -R 100:1000 <your_efs_mount>. By default, the EFS mount is root:root.

I believe I need to mount my EFS externally, would you be able to provide me steps as to how to do this? Thanks

[andyshinn]

Sorry, I don't remember the exact steps. What I did was something like:

1. Launch EC2 instance in same subnets as EFS
2. Install needed NFS utilities and tools
3. Mount the volume using the commands found in the EFS console
4. Run sudo chown -R 100:1000 against the mount

[nitrocode]

Is this something that can be automated and/or documented in the upstream terraform module?

https://github.com/terraform-aws-modules/terraform-aws-atlantis

[BenoitLF]

Not sure if applicable but the issue is still present on latest version @nitrocode

[nitrocode]

note that switching to the Atlantis non root user may impact the above workaround

• fix: use the non-root atlantis user instead of root #3886