Scan commit metadata

[rgmz]

Description:

This fixes #2683. It scans the commit author, committer (which is typically GitHub <noreply@github.com> for GitHub, but can be different), and message.

commit 8104611d6e2e3993c104974ea4c507faced7e847
Author:     Richard Gomez <32133502+rgmz@users.noreply.github.com>
AuthorDate: Tue Feb 6 10:13:53 2024 -0500
Commit:     GitHub <noreply@github.com>
CommitDate: Tue Feb 6 10:13:53 2024 -0500

    fix: case-insensitive ext check (#2383)

Admittedly, this implementation is a bit of guess-work and may not be the best way.

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[rosecodym]

This is cool! I never would have guessed that secrets could leak out in the author email.

[rosecodym]

Can we use a magic constant filename to signal to users that the chunk came from commit metadata? I'm a little worried that a missing filename might confuse people.

[rgmz]

We could do something obvious like COMMIT. I will mention that empty filename is currently how this is handled elsewhere, e.g., comments

[rosecodym]

Ah, if this isn't a new approach I'm less concerned about it, but it still seems like if it's low-hanging fruit we might as well grab it while we're here. I'll leave the decision on fruit height up to you.

[rgmz]

I pushed some changes to handle Git Notes as well. However, these may not be pulled by default and would need to be added as an additional ref to checkout in #1918.

[rgmz]

I believe this is a bug as git log appears to use d and not dd formatting. It's unclear if or how frequently this failure occurred, as the logging in isDateLine/isAuthorDateLine was level 2.

[rosecodym]

cool stuff!

[rosecodym]

Ah, if this isn't a new approach I'm less concerned about it, but it still seems like if it's low-hanging fruit we might as well grab it while we're here. I'll leave the decision on fruit height up to you.

[rgmz]

Linting issues seem to be carried over from #2643.

https://github.com/trufflesecurity/trufflehog/actions/runs/8802614804

[rgmz]

@rosecodym Any idea what might be causing the test failure?

=== Failed
=== FAIL: pkg/sources/git TestSource_Chunks_Integration/remote_repo,_unauthenticated (0.23s)
2024-04-25T14:15:48Z	error	context	error chunking	{"error": "context canceled"}
2024-04-25T14:15:48Z	info-0	context	error scanning repository	{"repo": "../../../", "error": "context canceled"}
2024-04-25T14:15:48Z	error	context	error chunking	{"error": "context canceled"}
2024-04-25T14:15:48Z	info-0	context	error scanning repository	{"repo": "https://github.com/dustin-decker/secretsandstuff.git", "error": "context canceled"}
2024-04-25T14:15:48Z	error	context	error chunking	{"error": "context canceled"}
2024-04-25T14:15:48Z	info-0	context	error scanning repository	{"repo": "https://github.com/dustin-decker/secretsandstuff.git", "error": "context canceled"}
2024-04-25T14:15:48Z	error	context	error chunking	{"error": "context canceled"}
2024-04-25T14:15:48Z	info-0	context	error scanning repository	{"repo": "https://github.com/dustin-decker/secretsandstuff.git", "error": "context canceled"}
    git_test.go:254: A chunk exists that was not expected with key "70001020fab32b1fcf2f1f0e5c66424eae649826-"
    git_test.go:254: A chunk exists that was not expected with key "a6f8aa55736d4a85be31a0048a4607396898647a-"
    git_test.go:254: A chunk exists that was not expected with key "73ab4713057944753f1bdeb80e757380e64c6b5b-"
    git_test.go:254: A chunk exists that was not expected with key "2f251b8c1e72135a375b659951097ec7749d4af9-"
    git_test.go:254: A chunk exists that was not expected with key "84e9c75e388ae3e866e121087ea2dd45a71068f2-"
    git_test.go:254: A chunk exists that was not expected with key "e6c8bbabd8796ea3cd85bfc2e55b27e0a491747f-"
    git_test.go:254: A chunk exists that was not expected with key "735b52b0eb40610002bb1088e902bd61824eb305-"
    git_test.go:254: A chunk exists that was not expected with key "ce62d79908803153ef6e145e042d3e80488ef747-"
    git_test.go:254: A chunk exists that was not expected with key "27fbead3bf883cdb7de9d7825ed401f28f9398f1-"
    git_test.go:254: A chunk exists that was not expected with key "8afb0ecd4998b1179e428db5ebbcdc8221214432-"
    git_test.go:254: A chunk exists that was not expected with key "8fe6f04ef1839e3fc54b5147e3d0e0b7ab971bd5-"

=== FAIL: pkg/sources/git TestSource_Chunks_Integration/remote_repo,_limited (0.19s)
    git_test.go:254: A chunk exists that was not expected with key "70001020fab32b1fcf2f1f0e5c66424eae649826-"

=== FAIL: pkg/sources/git TestSource_Chunks_Integration/remote_repo,_main_ahead_of_branch (0.20s)
    git_test.go:254: A chunk exists that was not expected with key "547865c6cc0da46622306902b1b66f7e25dd0412-"

Edit: perhaps a slight mishap causing the commit to have a - at the end? Idk, I'll look into it.

[rosecodym]

@rgmz should we revert this until we can figure out what's going on?

[rgmz]

Perhaps. The specific error seems to be caused by the test logic (commit + - + file); it's unclear whether that's indicative of an actual issue.

trufflehog/pkg/sources/git/git_test.go

Lines 249 to 251 in 81a9c81

	case *source_metadatapb.MetaData_Git:
	key = strings.TrimRight(meta.Git.Commit+"-"+meta.Git.File, "\n")
	}

I think it's caused by the commit metadata being sent, which is introducing new chunks that aren't accounted for.

trufflehog/pkg/sources/git/git_test.go

Lines 182 to 184 in 81a9c81

	expectedChunkData: map[string]*byteCompare{
	"70001020fab32b1fcf2f1f0e5c66424eae649826-aws": {B: []byte("[default]\naws_access_key_id = AKIAXYZDQCEN4B6JSJQI\naws_secret_access_key = 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\noutput = json\nregion = us-east-2\n")},
	"a6f8aa55736d4a85be31a0048a4607396898647a-bump": {B: []byte("\n\nf\n")},

[rosecodym]

Ok, I'm going to revert. Thanks for catching this so early!