Document and test against known true & false positive patterns

[rgmz]

Describe the solution you'd like

Gitleaks relies on a core set of regex patterns that are mysterious and seem to work well enough, but may in fact have some blind spots (as mentioned in this comment). I believe that we can significantly increase the quality of detections (more true positives and fewer false positives) by creating a set of known/realistic true & false positive, and testing each rule against them, rather than manually creating one or a handful of test cases for each rule.

gitleaks/cmd/generate/config/rules/adobe.go

Lines 19 to 22 in 06db3b9

	tps := []string{
	generateSampleSecret("adobe", secrets.NewSecret(hex("32"))),
	}
	return validate(r, tps, nil)

Below, I've created the skeleton of what that list may look like. Realistically it may not be possible to match all of these scenarios, or do so without creating significant false positives.

Configuration Formats

HCL

INI (https://quickref.me/ini.html)

Quoted

value = "{secret}"
value="{secret}"

Unquoted

value = {secret}
value={secret}

Comments

;{secret}
; {secret}
;value={secret}
; value={secret}

#{secret}
# {secret}
# ; value={secret}

Comments after line

value = {secret} ; This is a comment
value = {secret} # This is a comment

JSON (https://quickref.me/json.html)

String value

{
    "value": "{secret}"
}

Separate key/value

{
    "name": "secret",
    "value": "{secret}"
}

Array value

{
  "values": ["{secret}", "otherstuff"]
}
{
  "values": [
    "{secret}", 
    "otherstuff"
  ]
}

Escaped

"{\"value\": \"{secret}\"}"
"{\\\\\"value\\\\\":\\\\\"{secret}\\\\\"}"

"{\"name\": \"key\", \"value\": \"{secret}\"}"
"{\"name\": \"key\",\r\n\"value\": \"{secret}\"}"
"{\"name\": \"key\",\n\"value\": \"{secret}\"}"

"{\\\\\"name\\\\\": \\\\\"key\\\\\", \\\\\"value\\\\\": \\\\\"{secret}\\\\\"}"

Properties

Equals sign

value={secret}
value = {secret}

Colon

value:{secret}
value: {secret}

TOML

XML

Element

<value>{secret}</value>
<value>
    {secret}
</value>

Attribute

<entry name="value" value="{secret}" />

Separate key/value nodes

<entry>
  <name="value" />
  <value="{secret}" />
</entry>

YAML (https://yaml-multiline.info/)

Single line

value: {secret}
value: "{secret}"
value: '{secret}'

Multiline (literal style)

value: |
  {secret}
value: |-
  {secret}
value: |+2
  {secret}

Multiline (folding style)

value: >
  {secret}
value: >-
  {secret}
value: >+2
  {secret}

Programming Languages

C# / .NET

Variable

string value = "{secret}";
var value = "{secret}";

Go (https://quickref.me/go)

Variable

var value string = "{secret}"
var value string = `{secret}`

Short variable

value := "{secret}"
value := `{secret}`

Java / JVM

Java

String value = "{secret}";

Kotlin

var value: string = "{secret}"
var value = "{secret}"
val value: string = "{secret}"
val value = "{secret}"

Node.js / JavaScript / TypeScript

PHP

Python

Simple variable

value = "{secret}"
value = '{secret}'

Ruby

Miscellaneous Tools & File Formats

CSV

...

Logstash

  "value" => "{secret}"

Makefile

VALUE = {secret}
VALUE := {secret}
VALUE ?= {secret}
VALUE += {secret}

cc @zricethezav