Skip to content

perf(trufflehog): single process for PR changed-file scan#151

Merged
isaiah-grafana merged 4 commits intomainfrom
isaiah/trufflehog-single-invocation
Apr 15, 2026
Merged

perf(trufflehog): single process for PR changed-file scan#151
isaiah-grafana merged 4 commits intomainfrom
isaiah/trufflehog-single-invocation

Conversation

@isaiah-grafana
Copy link
Copy Markdown
Contributor

@isaiah-grafana isaiah-grafana commented Apr 14, 2026

Run one trufflehog filesystem over all filtered changed paths for pull_request / merge_group instead of one process per file, avoiding repeated startup cost on large diffs.

Comment thread .github/workflows/reusable-trufflehog.yml Outdated
Copy link
Copy Markdown
Contributor

@ezebunandu ezebunandu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Feed filtered paths through GNU xargs -0 so a typical PR runs one
TruffleHog process (avoids per-file startup) while argv stays under OS
limits on very large diffs. Paths are still filtered for excludes and
missing files (e.g. deletions in the diff).
@isaiah-grafana isaiah-grafana force-pushed the isaiah/trufflehog-single-invocation branch from acb2704 to f2f56c5 Compare April 15, 2026 03:13
Copy link
Copy Markdown
Contributor

@kelnage kelnage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments to consider

if ((${#paths[@]} > 0)); then
echo "TruffleHog: ${#paths[@]} path(s), batched by xargs as needed"
: > results.ndjson
printf '%s\0' "${paths[@]}" | xargs -0 -r trufflehog filesystem \
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't --include-paths=<FILE> achieve this without needing to worry about the size of the path file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I just modified it to use --include-paths with a temp file (anchored regex per changed path after excludes) and a single trufflehog filesystem . run, so we’re not putting paths on the argv.

Comment thread .github/workflows/reusable-trufflehog.yml Outdated
Use one trufflehog filesystem invocation over . with --include-paths
(anchored re.escape per path) to avoid argv limits; addresses review.

Revert get-vault-secrets action pin to match main (f1614b2).

Made-with: Cursor
@isaiah-grafana isaiah-grafana requested a review from kelnage April 15, 2026 14:00
Copy link
Copy Markdown
Contributor

@kelnage kelnage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@isaiah-grafana isaiah-grafana merged commit edaaa2b into main Apr 15, 2026
13 checks passed
@isaiah-grafana isaiah-grafana deleted the isaiah/trufflehog-single-invocation branch April 15, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants