filesystem support for exclude and include filters #881

mac2000 · 2022-10-30T07:38:37Z

Closes: #536 and technically closes: #865

What has been done:

I have moved the git filter creation a little bit lower into its switch case
And created similar filter creation in the filesystem switch case
Because of that go asked me to put err := below

For filesystem engine, itself did not want to touch it so just added one more "method" WithFilter() so it can be used where needed

In the same way, as in the git source, we are checking if the filter is not nil and the current file passes it

Offtopic: while trying to figure out how to do it found one more crappy thing in the filter itself - it does not remove empty lines from the exclusion file, and it may lead to scenarios when by mistake we may exclude all files with space in the name

Tests: did not find any tests related to the filter in the git source so have nothing to copy from, and because I'm golang noob all that channel stuff just blows my brain so can not handle that one 🤷‍♂️

ahrav · 2022-11-15T20:53:36Z

main.go

-	if err != nil {
-		logrus.WithError(err).Fatal("could not create filter")
-	}
-
 	var repoPath string
 	var remote bool


nit: If we declare var err error here, within the switch we can use regular assignment = vs :=

ahrav

Thanks a lot @mac2000 this looks like a great addition. Really appreciate it 😄

Looks like there is a conflict in main, otherwise LGTM!

CLAassistant · 2022-11-16T17:45:05Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
15 out of 16 committers have signed the CLA.

✅ mcastorina
✅ ahrav
✅ dustin-decker
✅ thiagola92
✅ jesslam948
✅ dxa4481
✅ pulkitanz
✅ 0xflotus
✅ bill-rich
✅ clonsdale-canva
✅ yilmi
✅ goncalossilva
✅ clarkedb
✅ mac2000
✅ 0x1
❌ dependabot[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

* Enable skipping of particular key IDs * update test

* add rambbitmq detector * use fixed length redaction Co-authored-by: Dustin Decker <dustin@trufflesec.com>

* Update slack webhook detector check to text. * remove redunant slashes.

* Add custom_detectors proto * Generate proto code * Create custom_detectors package Also create protoyaml package to test YAML unmarshalling the configuration. * Simplify custom_detectors proto by removing connection * Generate proto code * Update custom_detectors parsing tests

* Add logger to context * Fatal on no org

…flesecurity#946)

* Add validation skeleton * Add custom detector validation with tests * Validate and test regex vars * Implement RegexVarString * Use RegexVarString for validating regex variables * Add numerics to the regex variable matching Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Adjust repo count for new app * Fix chunk test count

* Change chunker test source * Emit chunk if the size isn't 0

…ecurity#1006) Bumps [cloud.google.com/go/secretmanager](https://github.com/googleapis/google-cloud-go) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](googleapis/google-cloud-go@dlp/v1.9.0...asset/v1.10.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/secretmanager dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#1007) Bumps [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) from 5.5.1 to 5.5.2. - [Release notes](https://github.com/go-git/go-git/releases) - [Commits](go-git/go-git@v5.5.1...v5.5.2) --- updated-dependencies: - dependency-name: github.com/go-git/go-git/v5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…lesecurity#1008) Bumps [github.com/hashicorp/go-retryablehttp](https://github.com/hashicorp/go-retryablehttp) from 0.7.1 to 0.7.2. - [Release notes](https://github.com/hashicorp/go-retryablehttp/releases) - [Commits](hashicorp/go-retryablehttp@v0.7.1...v0.7.2) --- updated-dependencies: - dependency-name: github.com/hashicorp/go-retryablehttp dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.4.0 to 0.5.0. - [Release notes](https://github.com/golang/crypto/releases) - [Commits](golang/crypto@v0.4.0...v0.5.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Handle invalid regex for custom detector. * Add comment highlighting invalid regex.

…lesecurity#1010) * Validate custom regular expressions on detector initialization * Add regex name to error message

…rufflesecurity#1004) This reverts commit ee6817a.

* Copy metadata for line number aware sources * Improve style

* Update entrypoint. * update comment. * Call each extra arg on its own. * Update loop. * Update extra args. * Use miccah's magic script. * Fix path to bash. * update entrypoint. * Add bash to Dockerfile. * update goreleaser dockerfile.

…rufflesecurity#1020)

* Small cleanup of CircleCi source. * address comments. * Add context to methods as first param.

* Small cleanup of CircleCi source. * Add concurrency to circleci. * merge w/ cleanup branch. * Rdefine loop var. * Delete github.go * reverge file delete. * Add debug log for scan errors. * make collecting scanned errors thread safe. * pre-allocate errors slice.

dickc-sg · 2023-01-17T20:53:33Z

@mac2000 what happened here? It looks like the PR was closed rather than merged and I do not see a reason why. Would love to see this feature added to filesystem scans.

…urity#1022) Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.16.0 to 0.17.0. - [Release notes](https://github.com/getsentry/sentry-go/releases) - [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md) - [Commits](getsentry/sentry-go@v0.16.0...v0.17.0) --- updated-dependencies: - dependency-name: github.com/getsentry/sentry-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…y#1024) Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.77.0 to 0.78.0. - [Release notes](https://github.com/xanzy/go-gitlab/releases) - [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go) - [Commits](xanzy/go-gitlab@v0.77.0...v0.78.0) --- updated-dependencies: - dependency-name: github.com/xanzy/go-gitlab dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/sergi/go-diff](https://github.com/sergi/go-diff) from 1.2.0 to 1.3.1. - [Release notes](https://github.com/sergi/go-diff/releases) - [Commits](sergi/go-diff@v1.2.0...v1.3.1) --- updated-dependencies: - dependency-name: github.com/sergi/go-diff dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…output (trufflesecurity#1032)

mac2000 · 2023-01-22T09:01:26Z

Hm, sorry, it was my fault, after messing with the GitHub sync of branches, I broke everything completely 🤷‍♂️

As a result figured out how to reset everything to upstream and cherry pick changes, otherwise my pull request was a mess of all changes from a PR creation

@ahrav if there is a chance to not wait for next conflict will be nice to merge it, seems like other similar changes were merged already (e.g. circleci flag for path exclusion)

PS: I hope that GitHub will figure out what I'm trying to do, so no need to create another pull request, if I did everything correct, after reopening it we should see desired changes only

mac2000 · 2023-01-22T09:08:27Z

@dickc-sg I broke it completely 🤷‍♂️

Looking forward to second attempt #1033

@ahrav if you still here will be nice to have you approve there as well so it might be merged

mac2000 · 2023-01-27T12:56:22Z

@dickc-sg fyi seems like we have made it 💪

thanks to @dustin-decker 👍 #1033 pull request was approved and merged

dickc-sg · 2023-02-07T16:09:19Z

Thank you!

mac2000 requested review from a team as code owners October 30, 2022 07:38

jesslam948 mentioned this pull request Nov 14, 2022

Add --exclude-paths support for gitlab scanning module #919

Merged

ahrav reviewed Nov 15, 2022

View reviewed changes

ahrav approved these changes Nov 15, 2022

View reviewed changes

mac2000 closed this Nov 16, 2022

mac2000 force-pushed the fs-exclude branch from d6ac30d to d3b550a Compare November 16, 2022 17:56

ahrav and others added 22 commits November 16, 2022 10:35

Use pointer to type. (trufflesecurity#926)

b8be0a6

Use pointer to type. (trufflesecurity#926)

eb4ff43

Return an error from ReadToMax when it panics (trufflesecurity#925)

b3d3f53

Add warning about using trufflehog as a library (trufflesecurity#928)

87e7aa3

Enable skipping of particular key IDs (trufflesecurity#930)

b18edef

* Enable skipping of particular key IDs * update test

add LDAP detector (trufflesecurity#896)

ae4b387

Fix error message (trufflesecurity#933)

9757c33

Skip some FTP FPs (trufflesecurity#929)

b45369c

add rambbitmq detector (trufflesecurity#936)

6e25664

* add rambbitmq detector * use fixed length redaction Co-authored-by: Dustin Decker <dustin@trufflesec.com>

Update slack webhook detector string check (trufflesecurity#932)

054e98d

* Update slack webhook detector check to text. * remove redunant slashes.

S3 scanner improvements (trufflesecurity#938)

28dd25b

Add newlines when file is split (trufflesecurity#937)

79cae3b

fix: small errors (trufflesecurity#917)

45ebaff

Close files to clean up tmp files (trufflesecurity#940)

f1ec9e7

Use new diskbufferreader version (trufflesecurity#941)

c26142d

Add logger to context (trufflesecurity#947)

3b055ce

* Add logger to context * Fatal on no org

Add test for configuring custom regex with webhook verification (truf…

5a339b0

…flesecurity#946)

Update README.md (trufflesecurity#952)

c3e596e

Integration test fixes (trufflesecurity#956)

9f99ee4

* Adjust repo count for new app * Fix chunk test count

Change chunker test source (trufflesecurity#959)

1a1c2e2

* Change chunker test source * Emit chunk if the size isn't 0

dependabot bot and others added 16 commits January 9, 2023 09:29

Handle invalid regex for custom detector. (trufflesecurity#1005)

09d4422

* Handle invalid regex for custom detector. * Add comment highlighting invalid regex.

Capture callstack of canceled contexts (trufflesecurity#979)

74831f6

Validate custom regular expressions on detector initialization (truff…

e5ede17

…lesecurity#1010) * Validate custom regular expressions on detector initialization * Add regex name to error message

Revert "Allow for default value to be used. (trufflesecurity#999)" (t…

864cf00

…rufflesecurity#1004) This reverts commit ee6817a.

Copy metadata for line number aware sources (trufflesecurity#1011)

8b2e1d3

* Copy metadata for line number aware sources * Improve style

Rename and export isGitSource (trufflesecurity#1016)

430d5c7

Update entrypoint (trufflesecurity#1013)

477e2a1

* Update entrypoint. * update comment. * Call each extra arg on its own. * Update loop. * Update extra args. * Use miccah's magic script. * Fix path to bash. * update entrypoint. * Add bash to Dockerfile. * update goreleaser dockerfile.

Use access-token endpoint for validity check (trufflesecurity#991)

0aa8e1c

Record timestamp when a context was cancelled (trufflesecurity#1018)

45b02f4

remove logger from retryable client, it is not respecting loglevels (t…

bc27fef

…rufflesecurity#1020)

[chore] - Small cleanup of CircleCi source (trufflesecurity#1028)

319ae64

* Small cleanup of CircleCi source. * address comments. * Add context to methods as first param.

dependabot bot and others added 6 commits January 17, 2023 15:20

Updated stdout to print results in alphabetical order for consistent …

2060ae1

…output (trufflesecurity#1032)

fix filter issue - empty lines should be ignored

06dc01e

filesystem support for filter exclude

bbdd44c

mac2000 reopened this Jan 22, 2023

mac2000 closed this Jan 22, 2023

mac2000 mentioned this pull request Jan 22, 2023

filesystem support for exclude and include filters (2nd attemp) #1033

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filesystem support for exclude and include filters #881

filesystem support for exclude and include filters #881

mac2000 commented Oct 30, 2022

ahrav Nov 15, 2022

ahrav left a comment •

edited

CLAassistant commented Nov 16, 2022 •

edited

dickc-sg commented Jan 17, 2023

mac2000 commented Jan 22, 2023

mac2000 commented Jan 22, 2023

mac2000 commented Jan 27, 2023

dickc-sg commented Feb 7, 2023

filesystem support for exclude and include filters #881

filesystem support for exclude and include filters #881

Conversation

mac2000 commented Oct 30, 2022

ahrav Nov 15, 2022

Choose a reason for hiding this comment

ahrav left a comment • edited

Choose a reason for hiding this comment

CLAassistant commented Nov 16, 2022 • edited

dickc-sg commented Jan 17, 2023

mac2000 commented Jan 22, 2023

mac2000 commented Jan 22, 2023

mac2000 commented Jan 27, 2023

dickc-sg commented Feb 7, 2023

ahrav left a comment •

edited

CLAassistant commented Nov 16, 2022 •

edited