Skip to content

Commit 897b201

Browse files
Merge branch 'main' into feat/oss-95-launchdarkly-analyzer
2 parents ca501a0 + 8765cc6 commit 897b201

File tree

4 files changed

+224
-116
lines changed

4 files changed

+224
-116
lines changed

README.md

Lines changed: 2 additions & 91 deletions
Original file line numberDiff line numberDiff line change
@@ -662,97 +662,8 @@ your custom detector has multiple `regex` set (in this example `hogID`, and `hog
662662

663663
**NB:** This feature is alpha and subject to change.
664664

665-
## Regex Detector Example
666-
667-
```yaml
668-
# config.yaml
669-
detectors:
670-
- name: HogTokenDetector
671-
keywords:
672-
- hog
673-
regex:
674-
hogID: '\b(HOG[0-9A-Z]{17})\b'
675-
hogToken: '[^A-Za-z0-9+\/]{0,1}([A-Za-z0-9+\/]{40})[^A-Za-z0-9+\/]{0,1}'
676-
verify:
677-
- endpoint: http://localhost:8000/
678-
# unsafe must be set if the endpoint is HTTP
679-
unsafe: true
680-
headers:
681-
- "Authorization: super secret authorization header"
682-
```
683-
684-
```
685-
$ trufflehog filesystem /tmp --config config.yaml --results=verified,unknown
686-
🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷
687-
688-
Found verified result 🐷🔑
689-
Detector Type: CustomRegex
690-
Decoder Type: PLAIN
691-
Raw result: HOGAAIUNNWHAHJJWUQYR
692-
File: /tmp/hog-facts.txt
693-
```
694-
695-
Data structure sent to the custom verification server:
696-
697-
```
698-
{
699-
"HogTokenDetector": {
700-
"HogID": ["HOGAAIUNNWHAHJJWUQYR"],
701-
"HogSecret": ["sD9vzqdSsAOxntjAJ/qZ9sw+8PvEYg0r7D1Hhh0C"],
702-
}
703-
}
704-
```
705-
706-
## Verification Server Example (Python)
707-
708-
Unless you run a verification server, secrets found by the custom regex
709-
detector will be unverified. Here is an example Python implementation of a
710-
verification server for the above `config.yaml` file.
711-
712-
```python
713-
import json
714-
from http.server import BaseHTTPRequestHandler, HTTPServer
715-
716-
AUTH_HEADER = 'super secret authorization header'
717-
718-
719-
class Verifier(BaseHTTPRequestHandler):
720-
def do_GET(self):
721-
self.send_response(405)
722-
self.end_headers()
723-
724-
def do_POST(self):
725-
try:
726-
if self.headers['Authorization'] != AUTH_HEADER:
727-
self.send_response(401)
728-
self.end_headers()
729-
return
730-
731-
# read the body
732-
length = int(self.headers['Content-Length'])
733-
request = json.loads(self.rfile.read(length))
734-
self.log_message("%s", request)
735-
736-
# check the match, you'll need to implement validateToken, which takes an array of ID's and Secrets
737-
if not validateTokens(request['HogTokenDetector']['hogID'], request['HogTokenDetector']['hogSecret']):
738-
self.send_response(200)
739-
self.end_headers()
740-
else:
741-
# any other response besides 200
742-
self.send_response(406)
743-
self.end_headers()
744-
except Exception:
745-
self.send_response(400)
746-
self.end_headers()
747-
748-
749-
with HTTPServer(('', 8000), Verifier) as server:
750-
try:
751-
server.serve_forever()
752-
except KeyboardInterrupt:
753-
pass
754-
```
755-
665+
### Regex Detector Example
666+
[Here](/pkg/custom_detectors/CUSTOM_DETECTORS.md) is how to setup a custom regex detector with verification server.
756667

757668

758669
## :mag: Analyze

go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ module github.com/trufflesecurity/trufflehog/v3
22

33
go 1.23.1
44

5-
toolchain go1.23.6
5+
toolchain go1.24.0
66

77
replace github.com/jpillora/overseer => github.com/trufflesecurity/overseer v1.2.8
88

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# TruffleHog Custom Detector Setup Guide
2+
3+
This guide will walk you through setting up a custom detector in TruffleHog to identify specific patterns unique to your project.
4+
5+
## Steps to Set Up a Custom Detector
6+
7+
1. **Create a Configuration File**:
8+
- TruffleHog uses a configuration file, typically named `config.yaml`, to manage custom detector configuration.
9+
- If this file doesn't exist, create it in your system.
10+
11+
2. **Define the Custom Detector**:
12+
- Open `config.yaml` with a text editor.
13+
- Add a new detector under the `detectors` section.
14+
15+
Here's a template for a custom detector:
16+
17+
```yaml
18+
# config.yaml
19+
detectors:
20+
- name: HogTokenDetector
21+
keywords:
22+
- hog
23+
regex:
24+
token: '[^A-Za-z0-9+\/]{0,1}([A-Za-z0-9+\/]{40})[^A-Za-z0-9+\/]{0,1}'
25+
verify:
26+
- endpoint: http://localhost:8000/
27+
# 'unsafe' must be set to true if the endpoint uses HTTP
28+
unsafe: true
29+
headers:
30+
- "Authorization: super secret authorization header"
31+
```
32+
33+
**Explanation**:
34+
- **`name`**: A unique identifier for your custom detector.
35+
- **`keywords`**: An array of strings that, when found, trigger the regex search. If multiple keywords are specified, the presence of any one of them will initiate the regex search.
36+
- **`regex`**: Defines the patterns to identify potential secrets. You can specify one or more named regular expressions. For a detection to be successful, each named regex must find a match. Capture groups `()` within these regular expressions are used to extract specific portions of the matched text, enabling the detector to process and report on particular segments of the identified patterns.
37+
38+
- **`verify`**: An optional section to validate detected secrets. If you want to verify or unverify detected secrets, this section needs to be configured. If not configured, all detected secrets will be marked as unverified. Read [verification server examples](#verification-server-examples)
39+
40+
**Other allowed parameters:**
41+
- **`exclude_regexes_capture`**: This parameter allows you to define regex patterns to exclude specific parts of a detected secret. If a match is found within the detected secret, the portion matching this regex is excluded from the result.
42+
- **`exclude_regexes_match`**: This parameter enables you to define regex patterns to exclude entire matches from being reported as secrets.
43+
- **`entropy`**: This parameter is used to assess the randomness of detected strings. High entropy often indicates that a string is a potential secret, such as an API key or password, due to its complexity and unpredictability. It helps in filtering false-positives. While an entropy threshold of `3` can be a starting point, it's essential to adjust this value based on your project's specific requirements and the nature of the data you have.
44+
- **`exclude_words`**: This parameter allows you to specify a list of words that, if present in a detected string, will cause TruffleHog to ignore that string.
45+
46+
[Here](/examples/generic_with_filters.yml) is an example of a custom detector using these parameters.
47+
48+
3. **Run TruffleHog with the Custom Detector**:
49+
- Execute TruffleHog, specifying your configuration file:
50+
51+
```bash
52+
trufflehog filesystem <path_to_folder_or_file> --config=<path_to_file>/config.yaml
53+
```
54+
55+
- Replace `<path_to_folder_or_file>` with the path to the directory or file you want to scan, and `<path_to_file>` with the path to your `config.yaml`.
56+
- TruffleHog will scan the specified file or folder using the custom detector you've defined.
57+
58+
4. **Example**:
59+
60+
Let's use the template config provided above to search a file.
61+
62+
Assume you have a file `/tmp/data.txt` with the following content:
63+
64+
```text
65+
// this is a custom example
66+
this file has some random text and maybe a secret
67+
hog token: pOIAj9x47WT5qElx5JrI3e7O714HgaAIz2ck9sVn
68+
// end of file
69+
```
70+
71+
In this file, the keyword `hog` exists, which will trigger the regex search. The string `pOIAj9x47WT5qElx5JrI3e7O714HgaAIz2ck9sVn` matches the regex pattern, so it should be detected.
72+
73+
Run the following command:
74+
75+
```bash
76+
trufflehog filesystem /tmp --config=config.yaml
77+
```
78+
79+
The output should be similar to:
80+
81+
```
82+
🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷
83+
84+
Found verified result 🐷🔑
85+
Detector Type: CustomRegex
86+
Decoder Type: PLAIN
87+
Raw result: pOIAj9x47WT5qElx5JrI3e7O714HgaAIz2ck9sVn
88+
File: /tmp/data.txt
89+
Line: 3
90+
```
91+
92+
The `Raw result` contains the matched string. `File` is the file name where secret was detected and `Line` is the exact line in the file where that was found.
93+
94+
95+
## Verification Server Examples
96+
Unless you run a verification server, secrets found by the custom regex detector will be unverified. Here is an example Python and Go implementation of a verification server for the above config.yaml file.
97+
98+
### Python:
99+
100+
```python
101+
import json
102+
from http.server import BaseHTTPRequestHandler, HTTPServer
103+
104+
AUTH_HEADER = 'super secret authorization header'
105+
106+
class Verifier(BaseHTTPRequestHandler):
107+
def do_GET(self):
108+
self.send_response(405)
109+
self.end_headers()
110+
111+
def do_POST(self):
112+
try:
113+
if self.headers['Authorization'] != AUTH_HEADER:
114+
self.send_response(401)
115+
self.end_headers()
116+
return
117+
118+
length = int(self.headers['Content-Length'])
119+
request = json.loads(self.rfile.read(length))
120+
self.log_message("%s", request)
121+
122+
if not validateTokens(request['HogTokenDetector']['token']):
123+
self.send_response(200)
124+
self.end_headers()
125+
else:
126+
self.send_response(403)
127+
self.end_headers()
128+
except Exception:
129+
self.send_response(400)
130+
self.end_headers()
131+
132+
def validateTokens(token):
133+
return False # Implement actual validation logic
134+
135+
with HTTPServer(('', 8000), Verifier) as server:
136+
try:
137+
server.serve_forever()
138+
except KeyboardInterrupt:
139+
pass
140+
```
141+
142+
### Go
143+
```go
144+
package main
145+
146+
import (
147+
"encoding/json"
148+
"fmt"
149+
"io"
150+
"log"
151+
"net/http"
152+
)
153+
154+
const authHeader = "super secret authorization header"
155+
156+
type HogTokenDetector struct {
157+
Token string `json:"token"`
158+
}
159+
160+
type RequestBody struct {
161+
HogTokenDetector HogTokenDetector `json:"HogTokenDetector"`
162+
}
163+
164+
func validateTokens(token string) bool {
165+
return false // Implement actual validation logic
166+
}
167+
168+
func verifierHandler(w http.ResponseWriter, r *http.Request) {
169+
if r.Method != http.MethodPost {
170+
http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
171+
return
172+
}
173+
174+
if r.Header.Get("Authorization") != authHeader {
175+
http.Error(w, "Unauthorized", http.StatusUnauthorized)
176+
return
177+
}
178+
179+
body, err := io.ReadAll(r.Body)
180+
if err != nil {
181+
http.Error(w, "Bad Request", http.StatusBadRequest)
182+
return
183+
}
184+
defer r.Body.Close()
185+
186+
var requestBody RequestBody
187+
if err := json.Unmarshal(body, &requestBody); err != nil {
188+
http.Error(w, "Bad Request", http.StatusBadRequest)
189+
return
190+
}
191+
192+
log.Printf("Received Request: %+v", requestBody)
193+
194+
if validateTokens(requestBody.HogTokenDetector.Token) {
195+
http.Error(w, "Forbidden", http.StatusForbidden)
196+
} else {
197+
w.WriteHeader(http.StatusOK)
198+
}
199+
}
200+
201+
func main() {
202+
http.HandleFunc("/", verifierHandler)
203+
serverAddr := ":8000"
204+
fmt.Printf("Starting server on %s...\n", serverAddr)
205+
if err := http.ListenAndServe(serverAddr, nil); err != nil {
206+
log.Fatalf("Server failed: %s", err)
207+
}
208+
}
209+
```

0 commit comments

Comments
 (0)