filesystem support for exclude and include filters

.github/workflows/release.yml

@@ -44,7 +44,7 @@ jobs:
         with:
           go-version: '1.18'
       - name: Run GoReleaser
-        uses: goreleaser/goreleaser-action@v3
+        uses: goreleaser/goreleaser-action@v4
         with:
           distribution: goreleaser
           version: latest

Dockerfile

@@ -9,7 +9,7 @@ RUN  --mount=type=cache,target=/go/pkg/mod \
      GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -o trufflehog .
 
 FROM alpine:3.15
-RUN apk add --no-cache git ca-certificates \
+RUN apk add --no-cache bash git ca-certificates \
     && rm -rf /var/cache/apk/* && \
     update-ca-certificates
 COPY --from=builder /build/trufflehog /usr/bin/trufflehog

Dockerfile.goreleaser

@@ -1,6 +1,6 @@
 FROM alpine:3.15
 
-RUN apk add --no-cache git ca-certificates \
+RUN apk add --no-cache bash git ca-certificates \
     && rm -rf /var/cache/apk/* && \
     update-ca-certificates
 WORKDIR /usr/bin/

README.md

@@ -86,7 +86,7 @@ trufflehog s3 --bucket=<bucket name> --only-verified
 + All I see is `🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷` and the program exits, what gives?
   + That means no secrets were detected
 + Why is the scan is taking a long time when I scan a GitHub org
-  + Unathenticated GitHub scans have rate limits. To improve your rate limits, include the `--token` flag with a personal access token
+  + Unauthenticated GitHub scans have rate limits. To improve your rate limits, include the `--token` flag with a personal access token
 + It says a private key was verified, what does that mean?
   + Check out our Driftwood blog post to learn how to do this, in short we've confirmed the key can be used live for SSH or SSL [Blog post](https://trufflesecurity.com/blog/driftwood-know-if-private-keys-are-sensitive/)  
 
@@ -96,7 +96,7 @@ trufflehog s3 --bucket=<bucket name> --only-verified
 TruffleHog v3 is a complete rewrite in Go with many new powerful features.
 
 - We've **added over 700 credential detectors that support active verification against their respective APIs**.
-- We've also added native **support for scanning GitHub, GitLab, filesystems, and S3**.
+- We've also added native **support for scanning GitHub, GitLab, filesystems, S3, and Circle CI**.
 - **Instantly verify private keys** against millions of github users and **billions** of TLS certificates using our [Driftwood](https://trufflesecurity.com/blog/driftwood) technology.
 
 
@@ -130,7 +130,7 @@ docker run -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --repo ht
 #### **Apple M1 users**
 
 The `linux/arm64` image is better to run on the M1 than the amd64 image.
-Even better is running the native darwin binary avilable, but there is not container image for that.
+Even better is running the native darwin binary available, but there is no container image for that.
 
 ```bash
 docker run --platform linux/arm64 -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --repo https://github.com/trufflesecurity/test_keys 
@@ -160,6 +160,7 @@ TruffleHog has a sub-command for each source of data that you may want to scan:
 - S3
 - filesystem
 - syslog
+- circleci
 - file and stdin (coming soon)
 
 Each subcommand can have options that you can see with the `--help` flag provided to the sub command:
@@ -172,27 +173,19 @@ Find credentials in git repositories.
 
 Flags:
       --help                     Show context-sensitive help (also try --help-long and --help-man).
-      --debug                    Run in debug mode
-      --version                  Prints trufflehog version.
+      --debug                    Run in debug mode.
+      --trace                    Run in trace mode.
   -j, --json                     Output in JSON format.
       --json-legacy              Use the pre-v3.0 JSON format. Only works with git, gitlab, and github sources.
-      --concurrency=1            Number of concurrent workers.
+      --concurrency=10           Number of concurrent workers.
       --no-verification          Don't verify the results.
       --only-verified            Only output verified results.
       --filter-unverified        Only output first unverified result per chunk per detector if there are more than one results.
+      --config=CONFIG            Path to configuration file.
       --print-avg-detector-time  Print the average time spent on each detector.
       --no-update                Don't check for updates.
-  -i, --include-paths=INCLUDE-PATHS
-                                 Path to file with newline separated regexes for files to include in scan.
-  -x, --exclude-paths=EXCLUDE-PATHS
-                                 Path to file with newline separated regexes for files to exclude in scan.
-      --since-commit=SINCE-COMMIT
-                                 Commit to start scan from.
-      --branch=BRANCH            Branch to scan.
-      --max-depth=MAX-DEPTH      Maximum depth of commits to scan.
-      --allow                    No-op flag for backwards compat.
-      --entropy                  No-op flag for backwards compat.
-      --regex                    No-op flag for backwards compat.
+      --fail                     Exit with code 183 if results are found.
+      --version                  Show application version.
 
 Args:
   <uri>  Git repository URL. https://, file://, or ssh:// schema expected.
@@ -269,13 +262,111 @@ repos:
     - id: trufflehog
       name: TruffleHog
       description: Detect secrets in your data.
-      entry: bash -c 'trufflehog git file://. --only-verified --fail'
+      entry: bash -c 'trufflehog git file://. --since-commit HEAD --only-verified --fail'
       # For running trufflehog in docker, use the following entry instead:
-      # entry: bash -c 'docker run -v "$(pwd):/workdir" -i --rm trufflesecurity/trufflehog:latest git file:///workdir --only-verified --fail'
+      # entry: bash -c 'docker run -v "$(pwd):/workdir" -i --rm trufflesecurity/trufflehog:latest git file:///workdir --since-commit HEAD --only-verified --fail'
       language: system
       stages: ["commit", "push"]
 ```
 
+## Regex Detector (alpha)
+
+Trufflehog supports detection and verification of custom regular expressions.
+For detection, at least one **regular expression** and **keyword** is required.
+A **keyword** is a fixed literal string identifier that appears in or around
+the regex to be detected. To allow maximum flexibility for verification, a
+webhook is used containing the regular expression matches.
+
+Trufflehog will send a JSON POST request containing the regex matches to a
+configured webhook endpoint. If the endpoint responds with a `200 OK` response
+status code, the secret is considered verified.
+
+**NB:** This feature is alpha and subject to change.
+
+### Regex Detector Example
+
+```yaml
+# config.yaml
+detectors:
+- name: hog detector
+  keywords:
+  - hog
+  regex:
+    adjective: hogs are (\S+)
+  verify:
+  - endpoint: http://localhost:8000/
+    # unsafe must be set if the endpoint is HTTP
+    unsafe: true
+    headers:
+    - 'Authorization: super secret authorization header'
+```
+
+```
+» trufflehog filesystem --directory /tmp --config config.yaml --only-verified
+🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷
+
+Found verified result 🐷🔑
+Detector Type: CustomRegex
+Decoder Type: PLAIN
+Raw result: hogs are cool
+File: /tmp/hog-facts.txt
+```
+
+#### Verification Server Example (Python)
+
+Unless you run a verification server, secrets found by the custom regex
+detector will be unverified. Here is an example Python implementation of a
+verification server for the above `config.yaml` file.
+
+```python
+import json
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+AUTH_HEADER = 'super secret authorization header'
+
+
+class Verifier(BaseHTTPRequestHandler):
+    def do_GET(self):
+        self.send_response(405)
+        self.end_headers()
+
+    def do_POST(self):
+        try:
+            if self.headers['Authorization'] != AUTH_HEADER:
+                self.send_response(401)
+                self.end_headers()
+                return
+
+            # read the body
+            length = int(self.headers['Content-Length'])
+            request = json.loads(self.rfile.read(length))
+            self.log_message("%s", request)
+
+            # check the match
+            if request['hog detector']['adjective'][-1] == 'cool':
+                self.send_response(200)
+                self.end_headers()
+            else:
+                # any other response besides 200
+                self.send_response(406)
+                self.end_headers()
+        except Exception:
+            self.send_response(400)
+            self.end_headers()
+
+
+with HTTPServer(('', 8000), Verifier) as server:
+    try:
+        server.serve_forever()
+    except KeyboardInterrupt:
+        pass
+```
+
+## Use as a library
+
+Currently, trufflehog is in heavy development and no guarantees can be made on
+the stability of the public APIs at this time.
+
 ## Contributors
 
 This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
@@ -299,3 +390,10 @@ We have published some [documentation and tooling to get started on adding new s
 ## License Change
 
 Since v3.0, TruffleHog is released under a AGPL 3 license, included in [`LICENSE`](LICENSE). TruffleHog v3.0 uses none of the previous codebase, but care was taken to preserve backwards compatibility on the command line interface. The work previous to this release is still available licensed under GPL 2.0 in the history of this repository and the previous package releases and tags. A completed CLA is required for us to accept contributions going forward.
+
+
+## Enterprise product
+
+Are you interested in continously monitoring your Git, Jira, Slack, Confluence, etc.. for credentials? We have an enterprise product that can help. Reach out here to learn more https://trufflesecurity.com/contact/
+
+We take the revenue from the enterprise product to fund more awesome open source projects that the whole community can benefit from.

action.yml

@@ -8,7 +8,8 @@ inputs:
     required: true
   base:
     description: Start scanning from here (usually main branch).
-    required: true
+    required: false
+    default: ''
   head:
     description: Scan commits until here (usually dev branch).
     required: false

entrypoint.sh

@@ -1,5 +1,6 @@
-#!/usr/bin/env ash
+#!/usr/bin/env bash
 
-# `$*` expands the `args` supplied in an `array` individually
-# or splits `args` in a string separated by whitespace.
-/usr/bin/trufflehog $*
+# Parse the last argument into an array of extra_args.
+mapfile -t extra_args < <(bash -c "for arg in ${*: -1}; do echo \$arg; done")
+
+/usr/bin/trufflehog "${@: 1: $#-1}" "${extra_args[@]}"