File-based Rules + Deduplication logic removal in v7.0.0

[ramimac]

Describe the bug

When using gitleaks v7.0.0, the removal of deduplication logic means that filename/path rules result in noisy, unnecessary) alerts (1 per line in file).

To Reproduce

1. Create a rule like:

[[rules]]
	description = "flag txt"
	file = '''(.*?)(txt)$'''
	tags = ["key", "extensions"]

2. Create a file "trigger.txt" that is multiple lines in length.
3. Run gitleaks, not that you will see duplicate alerts on "Filename/path offender: trigger.txt" , one per line in file

Expected behavior
I would expect file/path rules to only alert once per file.

Screenshots
If applicable, add screenshots to help explain your problem.

Basic Info (please complete the following information):

• OS: Using docker
• Gitleaks Version: 7.0.0

Additional context

cc @zricethezav

[zricethezav]

Most definitely a bug. Thanks for raising this. Will fix asap

[zricethezav]

@ramimac I was unable to recreate this bug

➜  ~/Go/src/github.com/zricethezav/gitleaks (tmp) ./gitleaks --config-path=test_data/test_configs/onlyFiles.toml -p . -v
INFO[0000] opening .
{
        "line": "",
        "lineNumber": 0,
        "offender": "Filename/path offender: trigger.txt",
        "commit": "20eaeb9aafb98e975b9b0fe70b6fc72df8ec13cf",
        "repo": ".",
        "repoURL": "",
        "leakURL": "",
        "rule": "flag txt",
        "commitMessage": "tst\n",
        "author": "zricethezav",
        "email": "zricer@protonmail.com",
        "file": "trigger.txt",
        "date": "2020-12-04T13:34:12-05:00",
        "tags": "key, extensions"
}
INFO[0009] scan time: 9 seconds 523 milliseconds 883 microseconds
INFO[0009] commits scanned: 712
WARN[0009] leaks found: 1

Is what I'm getting.

onlyFiles.toml:

[[rules]]
	description = "flag txt"
	file = '''trigger.txt'''
	tags = ["key", "extensions"]

Could you show share some additional details on how to recreate this?

[ramimac]

@zricethezav Sorry for the incomplete reproduction. This is actually an artifact of no-git as well:

gitleaks --config-path=onlyFiles.toml -p vol/ --no-git --verbose

onlyFiles.toml

[[rules]]
	description = "flag txt"
	file = '''(.*?)(txt)$'''
	tags = ["key", "extensions"]

trigger.txt

test

results:

{
	"line": "",
	"lineNumber": 1,
	"offender": "Filename/path offender: trigger.txt",
	"commit": "0000000000000000000000000000000000000000",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "flag txt",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": "vol/trigger.txt",
	"date": "1970-01-01T00:00:00Z",
	"tags": "key, extensions"
}
{
	"line": "",
	"lineNumber": 2,
	"offender": "Filename/path offender: trigger.txt",
	"commit": "0000000000000000000000000000000000000000",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "flag txt",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": "vol/trigger.txt",
	"date": "1970-01-01T00:00:00Z",
	"tags": "key, extensions"
}
{
	"line": "",
	"lineNumber": 3,
	"offender": "Filename/path offender: trigger.txt",
	"commit": "0000000000000000000000000000000000000000",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "flag txt",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": "vol/trigger.txt",
	"date": "1970-01-01T00:00:00Z",
	"tags": "key, extensions"
}
{
	"line": "",
	"lineNumber": 4,
	"offender": "Filename/path offender: trigger.txt",
	"commit": "0000000000000000000000000000000000000000",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "flag txt",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": "vol/trigger.txt",
	"date": "1970-01-01T00:00:00Z",
	"tags": "key, extensions"
}
{
	"line": "",
	"lineNumber": 5,
	"offender": "Filename/path offender: trigger.txt",
	"commit": "0000000000000000000000000000000000000000",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "flag txt",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": "vol/trigger.txt",
	"date": "1970-01-01T00:00:00Z",
	"tags": "key, extensions"
}
{
	"line": "",
	"lineNumber": 6,
	"offender": "Filename/path offender: trigger.txt",
	"commit": "0000000000000000000000000000000000000000",
	"repo": "",
	"repoURL": "",
	"leakURL": "",
	"rule": "flag txt",
	"commitMessage": "",
	"author": "",
	"email": "",
	"file": "vol/trigger.txt",
	"date": "1970-01-01T00:00:00Z",
	"tags": "key, extensions"
}
INFO[0000] scan time: 512 microseconds                  
WARN[0000] leaks found: 6                 

[zricethezav]

Perfect! Thank you.

[zricethezav]

@ramimac fixed in https://github.com/zricethezav/gitleaks/releases/tag/v7.0.2

[ramimac]

@zricethezav Thanks for the quick fix, much appreciated! Do you have plans to push 7.0.2 to dockerhub?

[zricethezav]

@ramimac just pushed! Thanks for the reminder