git rev-parse HEAD^2 unknown revision or path not in the working tree

[krrrr38]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

When enable --checkout-strategy=merge, will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD^2: exit status 128: HEAD^2\nfatal: ambiguous argument 'HEAD^2': unknown revision or path not in the working tree.\nUse '--' to separate paths from revisions, like this:\n'git <command> error raised.

Reproduction Steps

• enable --checkout-strategy=merge
• create PR and run plan

Logs

Provide log files from Atlantis server

Logs

{
  "level": "warn",
  "ts": "2023-01-09T02:22:25.765Z",
  "caller": "events/working_dir.go:108",
  "msg": "will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD^2: exit status 128: HEAD^2\nfatal: ambiguous argument 'HEAD^2': unknown revision or path not in the working tree.\nUse '--' to separate paths from revisions, like this:\n'git <command> [<revision>...] -- [<file>...]'\n",
  "json": {
    "repo": "something/something",
    "pull": "18189"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:108\ngithub.com/runatlantis/atlantis/server/events.(*GithubAppWorkingDir).Clone\n\tgithub.com/runatlantis/atlantis/server/events/github_app_working_dir.go:54\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).buildAllCommandsByCfg\n\tgithub.com/runatlantis/atlantis/server/events/project_command_builder.go:308\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/project_command_builder.go:202\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands.func1\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:29\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).buildAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:62\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:26\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:82\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:256\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:174"
}

Environment details

If not already included, please provide the following:

• Atlantis version: v0.22.2
• Atlantis flags: --checkout-strategy=merge

Additional Context

HEAD^^ might be helpful.

> git rev-parse HEAD^^
179a12460a1bcd37aacbd30e0ec681eee91419de

> git rev-parse HEAD^2
HEAD^2
fatal: ambiguous argument 'HEAD^2': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

atlantis/server/events/working_dir.go

Lines 100 to 103 in e4473ce

	pullHead := "HEAD"
	if w.CheckoutMerge {
	pullHead = "HEAD^2"
	}

[nitrocode]

Why would HEAD^^ return an error and HEAD^2 run successfully?

[krrrr38]

Sorry, I investigate it and found that generally it doesn't have a problem. This issue is raised by following steps.

1. enable --checkout-strategy=branch
2. Create PR, then direcotry cloned
3. enable --checkout-strategy=merge
4. push the same branch for the PR
   1. At that time HEAD^2 cannot be resolved. Cause existed directory doesn't have merge commit.

I think it's ok to close this issue for a edge case.

(I'm trying --checkout-strategy=merge feature)

---

• HEAD^2 is a second parent (first generation)
  • if HEAD is a merge commit, there are multiple parents.
  • if it is not a merge commit, the second parent is not exist
• HEAD^^ is parent of parent

In this case HEAD^2 is correct.

[renescheepers]

Running into same errors on our configuration. Running multiple instances of Atlantis using Redis locking with a shared data directory. Seems to also happen when creating a pull request and quickly pushing a commit to the pull request.

I think its caused by a race condition where mutliple instances are trying to clone the repository into the same directory, before the PR has been locked.

[nitrocode]

@renescheepers the redis locking feature is a brand new feature and has limited documentation. Could you expand on your setup? Version, repo config, server config, reproducible steps, do you use workspaces?

Perhaps the lock isn't checked until after the clone?

[renescheepers]

@nitrocode we are currently using Atlantis 0.21.0. Everything is in workspace default. I haven't been able to consistently reproduce this yet in our testing environment. It does seem to happen on production environment when the GitHub hooks don't seem to be responding very fast.

What I think happens is that a PR is opened and is quickly followed up by another commit, firing two events both arriving at different instances where the PR hasn't been locked yet. Causing a race condition on the directory containing the repository.

Atlantis deployed as 3 instances running.
/mnt/atlantis is mounted to an NFS server.
ATLANTIS_REDIS_HOST is set as an environment variable when running the Kubernetes container.

Server config

port: 18000
gh-org: X
gh-app-id: X
gh-app-slug: X
gh-app-key-file: /var/run/atlantis/gh-app-key/atlantis-app-key.pem
gh-allow-mergeable-bypass-apply: true
write-git-creds: true
repo-allowlist: github.com/X
atlantis-url: https://atlantis.X.com
repo-config: /var/run/atlantis-repo-config/repos.yaml
data-dir: /mnt/atlantis # Shared between instances
enable-regexp-cmd: true
disable-apply-all: true
default-tf-version: v0.15.4
hide-prev-plan-comments: true
skip-clone-no-changes: true
checkout-strategy: merge
enable-policy-checks: true
quiet-policy-checks: true
locking-db-type: redis
markdown-template-overrides-dir: /templates

Repo config

Removed about 1500 projects but it mostly looks like the one below.

---
version: 3
projects:
... # 4000 lines of projects here
- dir: terraform/gcp/projects/...REMOVED
  terraform_version: v1.3.7

Exception 1

running git clone --branch master --single-branch https://x-access-token:<redacted>@github.com/REMOVED.git /mnt/atlantis/repos/REMOVED/15594/default: Cloning into '/mnt/atlantis/repos/REMOVED/15594/default'...
fatal: unable to open /mnt/atlantis/repos/REMOVED/15594/default/.git/objects/pack/tmp_pack_XaLWyU: No such file or directory
fatal: index-pack failed\n:

Exception 2

"will re-clone repo, could not determine if was at correct commit: 
git rev-parse HEAD^2: exit status 128: HEAD^2
fatal: ambiguous argument 'HEAD^2': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:\n'

Exception 3

msg":"will re-clone repo, could not determine if was at correct commit: 
git rev-parse HEAD^2: exit status 128: fatal: not a git repository (or any parent up to mount point /mnt)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
","json":{"repo":"REMOVED","pull":"15581"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone
\t/home/runner/work/atlantis/atlantis/server/events/working_dir.go:108\ngithub.com/runatlantis/atlantis/server/events.(*GithubAppWorkingDir).Clone
\t/home/runner/work/atlantis/atlantis/server/events/github_app_working_dir.go:54\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandRunner).doPlan
\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:374\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandRunner).Plan\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:208\ngithub.com/runatlantis/atlantis/server/events.(*ProjectOutputWrapper).updateProjectPRStatus\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:169\ngithub.com/runatlantis/atlantis/server/events.(*ProjectOutputWrapper).Plan\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:149\ngithub.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:38\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:13\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\t/home/runner/work/atlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\t/home/runner/work/atlantis/atlantis/server/events/plan_command_runner.go:127\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\t/home/runner/work/atlantis/atlantis/server/events/plan_command_runner.go:245\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\t/home/runner/work/atlantis/atlantis/server/events/command_runner.go:174"}

[nitrocode]

Sounds like the root issue is pushing 1 commit and then the next one too quickly and Atlantis not checking if there is an already an existing run.

There is a related issue here

• New commit or "plan comment" to cancel the currently running plan and rerun #305

[renescheepers]

@nitrocode yes, I think thats the case. When trying to reproduce in a testing environment I was also able to get a Terraform locking error. Which also shouldn't happen if the locking works correctly. I'm still trying to setup an environment in which I can consistently reproduce this and hopefully fix it.
Atlantis should first create the lock before doing anything else, don't know if this the case. If it does then there is a bug there.

[nitrocode]

yes, if you can reproduce this, it would be great to have a fix and a test for this! Please feel free to propose a fix.

[renescheepers]

Ok, did a little bit more digging into this. I think the code below is causing this, the sync map is of course not across multiple instances. We are also running into some other performance issues when using an NFS server. When using the merge checkout strategy it won't do a shallow clone of the repository this in combination with the size of our repository results in the clone taking quite a while. This problem is even worsened quite a bit by the use of an NFS server, which has absolutely terrible performance when there are a lot of small files.

atlantis/server/events/working_dir.go

Lines 205 to 212 in 27b9897

	value, _ := cloneLocks.LoadOrStore(cloneDir, new(sync.Mutex))
	mutex := value.(*sync.Mutex)
	defer mutex.Unlock()
	if locked := mutex.TryLock(); !locked {
	mutex.Lock()
	return nil
	}

[nitrocode]

@renescheepers do you think this pr would resolve it?

• feat: shallow repo clone in merge checkout strategy #2758

Besides shallow clones for the merge strategy, what else could be proposed to speed up your performance?

[ygg-drop]

I ran into the original error in a situation where the merge strategy and a fixed checkout-depth was used. When a PR with more commits than checkout-depth was raised, Atlantis would fail with the above error.

I had a look at the code and I think the issue is that when Atlantis does a git fetch --unshallow here, this changes FETCH_HEAD to point to the base branch. The merge succeeds because there is nothing to merge, but the repo is left on the base branch instead of the feature branch.

[jamengual]

@nitrocode @krrrr38