(fix) Git Repo Cloning Error #4223

abmussani · 2025-06-13T17:06:30Z

First of all special thanks to @dylanTruffle for the research and guidance how to solve this issue.

Description:

TL:DR; This PR uses --mirror flag to clone git repo to avoid error due to similar names of refs (tags, branches, pull requests etc). Also, This PR removes SkipAdditionalRefs flag which become unnecessary, since by default, using --mirror, will bring all the refs.

Multiple instances of git cloning issues were reported, if tags, branches, pull requests or releases in that repo have conflicting names.

$ go run . git https://github.com/airbnb/gem-aws-creds.git
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2025-06-13T21:26:29+05:00	error	trufflehog	error running scan	{"error": "failed to scan Git: error preparing repo: failed to clone unauthenticated Git repo (https://github.com/airbnb/gem-aws-creds.git): could not clone repo: https://github.com/airbnb/gem-aws-creds.git, error executing git clone: exit status 128, fatal: multiple updates for ref 'refs/remotes/origin/users/pierre/for/1' not allowed\n"}
exit status 1

The actual problem is how git refs (includes branches, tags, releases etc) are being cloned using parameter -c remote.origin.fetch=+refs/*:refs/remotes/origin/*. Which says that,

👉 "Fetch ALL refs, whatever they are (branches, tags, pull requests,etc)**
👉 and map them into the local namespace refs/remotes/origin/*."

When two refs have similar name, Cloning get halt and error appears. This can be avoided if refs are being cloned under different head based on the type. e.g:

-c 'remote.origin.fetch=^refs/heads/*' \
  -c 'remote.origin.fetch=+refs/reviewable/*:refs/remotes/origin/reviewable/*' \
  -c 'remote.origin.fetch=+refs/pull/*:refs/remotes/origin/pull/*' \
  -c 'remote.origin.fetch=+refs/heads/*:refs/remotes/origin/heads/*' \
  -c 'remote.origin.fetch=+refs/release/*:refs/remotes/origin/releases/*' \
  -c 'remote.origin.fetch=+refs/tags/*:refs/remotes/origin/tags/*'

This solution does works but not auto-scalable if a new type of refs is added in future. Someone have to modify the above list of parameters to support it.

Another solution is the flag --mirror which clones all the available refs in respected directory instead of at origin. It is lightweight as compare to -c remote.origin.fetch=+refs/*:refs/remotes/origin/* because it does not bring the latest code to cloning directory which is actually not needed. Which is a win-win. Also, This PR remove SkipAdditionalRefs flag which become unnecessary, since by default, using --mirror, will bring all the refs.

here are the successful cloning examples using --mirror:

$ go run . git https://github.com/dylanTruffle/refconflicts
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2025-06-13T22:01:07+05:00	info-0	trufflehog	running source	{"source_manager_worker_id": "iJ6ut", "with_units": true}
2025-06-13T22:01:07+05:00	info-0	trufflehog	scanning repo	{"source_manager_worker_id": "iJ6ut", "unit_kind": "dir", "unit": "/var/folders/50/890wpbn158nc4dczq8mkwt3h0000gp/T/trufflehog-48566-747569857", "repo": "https://github.com/dylanTruffle/refconflicts"}
2025-06-13T22:01:07+05:00	info-0	trufflehog	finished scanning	{"chunks": 4, "bytes": 208, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "5.539373167s", "trufflehog_version": "dev", "verification_caching": {"Hits":0,"Misses":0,"HitsWasted":0,"AttemptsSaved":0,"VerificationTimeSpentMS":0}}

$ go run . git https://github.com/microsoft/adaptive-testing
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2025-06-13T22:02:50+05:00	info-0	trufflehog	running source	{"source_manager_worker_id": "NSGo6", "with_units": true}
2025-06-13T22:02:50+05:00	info-0	trufflehog	scanning repo	{"source_manager_worker_id": "NSGo6", "unit_kind": "dir", "unit": "/var/folders/50/890wpbn158nc4dczq8mkwt3h0000gp/T/trufflehog-49158-461213315", "repo": "https://github.com/microsoft/adaptive-testing"}
Found unverified result 🐷🔑❓
Detector Type: TomorrowIO
Decoder Type: PLAIN
Raw result: 23978ef1a2e6465e96b328a489a7711a
Commit: d8994352cfa5e04c38e9091ebe8d227fec3805be
Email: charvir <t-crastogi@microsoft.com>
File: bias_besa_thinkaloud_midway.csv
Line: 10
Repository: https://github.com/microsoft/adaptive-testing
Timestamp: 2022-08-10 00:58:53 +0000

2025-06-13T22:02:59+05:00	info-0	trufflehog	finished scanning	{"chunks": 4891, "bytes": 18626749, "verified_secrets": 0, "unverified_secrets": 1, "scan_duration": "21.841504791s", "trufflehog_version": "dev", "verification_caching": {"Hits":0,"Misses":1,"HitsWasted":0,"AttemptsSaved":0,"VerificationTimeSpentMS":954}}

Checklist:

Tests passing (make test-community)?
Lint passing (make lint this requires golangci-lint)?

…ameter . removed SkipAdditionalRefs flag, cloning using --mirror will include all the refs. put the check if userinfo is nil to avoid any crash due to invalid memory access.

dylanTruffle · 2025-06-13T17:32:39Z

Hey there, awesome. Did we check to see if it works well with the CI/CD flags?

https://github.com/trufflesecurity/trufflehog?tab=readme-ov-file#12-scan-in-ci

abmussani · 2025-06-16T11:41:52Z

Hey there, awesome. Did we check to see if it works well with the CI/CD flags?

https://github.com/trufflesecurity/trufflehog?tab=readme-ov-file#12-scan-in-ci

I did some testing around these flags and scanning is working. Do you see anyother area need to be tested out ?

abmussani · 2025-06-16T11:50:38Z

Moreover, With the help of my colleague @mustansir14, He created a testing public repo on bitbucket with tags branch name and a test tag.
https://mustansir14@bitbucket.org/mustansir14/reftest.git

$ go run . git https://bitbucket.org/mustansir14/reftest.git
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2025-06-16T16:49:16+05:00       info-0  trufflehog      running source  {"source_manager_worker_id": "0xBLT", "with_units": true}
2025-06-16T16:49:16+05:00       info-0  trufflehog      scanning repo   {"source_manager_worker_id": "0xBLT", "unit_kind": "dir", "unit": "/var/folders/50/890wpbn158nc4dczq8mkwt3h0000gp/T/trufflehog-33444-3013740438", "repo": "https://bitbucket.org/mustansir14/reftest.git"}
2025-06-16T16:49:17+05:00       info-0  trufflehog      finished scanning       {"chunks": 2, "bytes": 697, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "3.136132667s", "trufflehog_version": "dev", "verification_caching": {"Hits":0,"Misses":0,"HitsWasted":0,"AttemptsSaved":0,"VerificationTimeSpentMS":0}}

rosecodym · 2025-06-16T13:10:54Z

pkg/sources/git/git.go

+		if params.userInfo == nil {
+			if pass, ok := params.userInfo.Password(); ok {
+				/*
+					Sources:
+						- https://medium.com/%40szpytfire/authenticating-with-github-via-a-personal-access-token-7c639a979eb3
+						- https://trinhngocthuyen.com/posts/tech/50-shades-of-git-remotes-and-authentication/#using-httpextraheader-config
+				*/
+				authHeader := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", params.userInfo.Username(), pass)))
+				gitArgs = append(gitArgs, "-c", fmt.Sprintf("http.extraHeader=Authorization: Basic %s", authHeader))
+			}


Why does this block have changes?

rosecodym · 2025-06-16T13:34:20Z

#1918 tries to do this same thing, but uses the --no-replace-objects flag in addition to find some commits that are apparently missed if you don't use it (see this discussion).

rosecodym

requesting changes pending the outcome of an internal conversation about configurability

dustin-decker · 2025-06-18T13:33:36Z

main.go

+	forceSkipBinaries  = cli.Flag("force-skip-binaries", "Force skipping binaries.").Bool()
+	forceSkipArchives  = cli.Flag("force-skip-archives", "Force skipping archives.").Bool()
+	skipAdditionalRefs = cli.Flag("skip-additional-refs", "Skip additional references.").Bool()
+	useGitMirror       = cli.Flag("use-git-mirror", "Use git mirror for git scans.").Bool()


Shouldn't we make this on by default for OSS?

This is how we'd do that while still having it off by default when imported, btw:

trufflehog/main.go

Lines 454 to 455 in d3eb6c4

// OSS Default APK handling on

feature.EnableAPKHandler.Store(true)

instead of providing list of refs we want to scan, using --mirror par…

08e20af

…ameter . removed SkipAdditionalRefs flag, cloning using --mirror will include all the refs. put the check if userinfo is nil to avoid any crash due to invalid memory access.

abmussani requested review from a team as code owners June 13, 2025 17:06

rosecodym reviewed Jun 16, 2025

View reviewed changes

rosecodym requested changes Jun 16, 2025

View reviewed changes

dustin-decker mentioned this pull request Jun 16, 2025

fixing clone refs overlap error #4215

Closed

2 tasks

keep both -c and --mirror

d3eb6c4

rosecodym approved these changes Jun 17, 2025

View reviewed changes

shahzadhaider1 approved these changes Jun 18, 2025

View reviewed changes

dustin-decker reviewed Jun 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

(fix) Git Repo Cloning Error #4223

(fix) Git Repo Cloning Error #4223

Uh oh!

abmussani commented Jun 13, 2025

Uh oh!

dylanTruffle commented Jun 13, 2025

Uh oh!

abmussani commented Jun 16, 2025

Uh oh!

abmussani commented Jun 16, 2025

Uh oh!

rosecodym Jun 16, 2025

Uh oh!

rosecodym commented Jun 16, 2025

Uh oh!

rosecodym left a comment

Uh oh!

dustin-decker Jun 18, 2025

Uh oh!

dustin-decker Jun 18, 2025

Uh oh!

Uh oh!

	// OSS Default APK handling on
	feature.EnableAPKHandler.Store(true)

(fix) Git Repo Cloning Error #4223

Are you sure you want to change the base?

(fix) Git Repo Cloning Error #4223

Uh oh!

Conversation

abmussani commented Jun 13, 2025

Description:

Checklist:

Uh oh!

dylanTruffle commented Jun 13, 2025

Uh oh!

abmussani commented Jun 16, 2025

Uh oh!

abmussani commented Jun 16, 2025

Uh oh!

rosecodym Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

rosecodym commented Jun 16, 2025

Uh oh!

rosecodym left a comment

Choose a reason for hiding this comment

Uh oh!

dustin-decker Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

dustin-decker Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!