Trufflehog - not able to use regexp to through custom detector

[simonaferrara]

Hi,

I'm trying to configure trufflehog to use custom regexp to detect simple secrets basing on my needs, e.g. to search for passwords or access tokens.
Unfortunately I've not found a way to do it.

I've found also another post about a similar issue ( #1317 ) and I've tried to apply the suggestion included within this post's discussion to search for an api key through a custom detector.

I've created the testAPIkeyConfig.yaml file with the following content:

detectors:
- name: custom api detector
  keywords:
  - api
  regex:
    adjective: "[a-zA-Z0-9]{32}"

and the testAPIkey.json with the following content:
"API key": "([a-zA-Z0-9]{32})"

and I've executed the following command:
trufflehog filesystem testAPIkey.json --config testAPIkeyConfig.yaml

The output by trufflehog is not the expected one. The API key is not highlighted by the tool:

What I'm doing wrong?
Your help will be useful for me to configure also other regexp that I would like to use.

Thanks!

[kashifkhan0771]

I tested using the YAML configuration you provided, and it worked as expected. The issue you're encountering might be due to the absence of the specified keyword in the data you're using. For the detection logic to trigger, the keyword defined in your YAML must be present in the input data. If the keyword is missing, the detector won't execute its logic.

So in your case the file testAPIkey.json should have a api keyword present somewhere.
Let me know if this helps or if there’s anything else I can clarify!

[RumitAP]

Hey @kashifkhan0771 , could you please clarify. I'm running into the same issue as the author here. I have a config.yaml that just has the "hog" detector defined from your README example. I created a .py file with the raw result from the example you guys provided and a file system scan did not detect the secret? It seems like scans are just not taking the config.yaml into account? I even tried adding the keyword "hog" that is defined in your config.yaml example into the file that I'm scanning and it still did not return that a secret was found.

Also what do you mean that for the detection logic to trigger, the keyword defined in the yaml must be present in the input data? Like in OP's example here, his keyword is "api" and the file he's scanning has the word "API" in the key.

[kashifkhan0771]

Hi @RumitAP,

You created a config file based on the Regex Detector Example and used the RAW result from the example output (HOGAAIUNNWHAHJJWUQYR) in a file that you want to scan using the filesystem scan.

The example config consists of two regex patterns: hogID and hogToken. For the custom detector to successfully identify a match, both regex patterns must be found in the file.

Currently, the RAW result (HOGAAIUNNWHAHJJWUQYR) corresponds to the hogID defined in the config regex. However, you also need to include a HogSecret (hogToken) in the file for the custom detector to detect it properly.

For testing, you can generate an example regex-matching string using this tool.
For instance, let’s use the following as hogToken:
5Sr1JXeclU7zeq5fBgrO1tzGzkk8uXSlBJ2NFTJl!

Now, add both values to a file (e.g., custom.txt):

<some random text>
HOGAAIUNNWHAHJJWUQYR
5Sr1JXeclU7zeq5fBgrO1tzGzkk8uXSlBJ2NFTJl!

Verification Server (Optional)

If you also want a verification server running to check whether the token is live, follow these steps to set up a Python server.

If you skip this step, the tokens will still be detected but marked as unverified since they won’t have a verification endpoint.

Running the Filesystem Scan

The command from the example is:

trufflehog filesystem <file_to_scan> --config <path_to_file>/config.yaml --results=verified,unknown

Since this command filters results to show only verified or unknown matches, you must have a verification server running for it to work properly.

If your verification server is not running, omit the --results flag when running the command:

trufflehog filesystem <file_to_scan> --config <path_to_file>/config.yaml

This will show unverified matches, with output similar to:

Found unverified result 🐷🔑❓
Detector Type: CustomRegex
Decoder Type: PLAIN
Raw result: 5Sr1JXeclU7zeq5fBgrO1tzGzkk8uXSlBJ2NFTJlHOGAAIUNNWHAHJJWUQYR
Name: HogTokenDetector
File: <path_to_file>/custom.txt
Line: 1

I hope this clarifies the usage of custom detectors. I will also talk internally to the team to update the documentation of custom detector with proper examples.

[kashifkhan0771]

This PR will improve the custom regex detector documentation.

[kashifkhan0771]

The PR is merged! Documentation is updated with more details. If you still face any issue, feel free to open a new issue. Thank you!