Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for configuring the buffered file writer #2319

Merged
merged 70 commits into from
Jan 30, 2024
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
f2f13b2
Write large diffs to tmp files
ahrav Jan 15, 2024
7d2452a
address comments
ahrav Jan 16, 2024
0802a62
Move bufferedfilewriter to own pkg
ahrav Jan 17, 2024
0365023
update test
ahrav Jan 17, 2024
57e5d6a
swallow write err
ahrav Jan 17, 2024
dad92c5
Merge branch 'main' into feat-tmp-file-diffs
ahrav Jan 17, 2024
fd41709
use buffer pool
ahrav Jan 17, 2024
da2ff88
use size vs len
ahrav Jan 18, 2024
b3260ff
Merge branch 'main' into feat-tmp-file-diffs
ahrav Jan 19, 2024
73903e7
use interface
ahrav Jan 19, 2024
994c268
fix test
ahrav Jan 19, 2024
f8e87e9
update comments
ahrav Jan 19, 2024
bba85f4
fix test
ahrav Jan 20, 2024
d285355
Allow for configuring the buffered file writer
ahrav Jan 20, 2024
2e7479e
remove unused
ahrav Jan 20, 2024
f78e3f9
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 20, 2024
4320c9e
add missing method
ahrav Jan 20, 2024
8cc4246
remove
ahrav Jan 20, 2024
6d0a5ba
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 20, 2024
f114a4a
remove unused
ahrav Jan 20, 2024
bf25bd2
Merge branch 'feat-tmp-file-diffs' of github.com:trufflesecurity/truf…
ahrav Jan 20, 2024
b3c3ce5
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 20, 2024
390a5a5
move parser and commit struct closer to where they are used
ahrav Jan 20, 2024
19b9beb
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 20, 2024
bb82471
linter change
ahrav Jan 20, 2024
0cec9a5
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 20, 2024
889596a
fix snifftest
ahrav Jan 20, 2024
1573a3a
address comments
ahrav Jan 22, 2024
9af37d8
merge main
ahrav Jan 23, 2024
bab3f04
add more kvp pairs to error
ahrav Jan 23, 2024
f26ac45
fix test
ahrav Jan 23, 2024
cbd4ddf
update
ahrav Jan 23, 2024
6688d1c
add back missing metadata fields
ahrav Jan 23, 2024
7aef649
address comments
ahrav Jan 23, 2024
47544ad
remove bufferedfile writer
ahrav Jan 24, 2024
90062c2
add buffered file writers
ahrav Jan 24, 2024
c37614e
fix
ahrav Jan 24, 2024
3c7a1d5
Merge branch 'main' into feat-tmp-file-diffs
ahrav Jan 24, 2024
6b8c52a
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 24, 2024
a9ee71a
Merge branch 'main' into feat-tmp-file-diffs
ahrav Jan 26, 2024
91b2fa7
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 26, 2024
419107e
address comments
ahrav Jan 29, 2024
973d6c7
use unint8
ahrav Jan 29, 2024
febfa26
update interface
ahrav Jan 29, 2024
1bd8b27
adjust interface
ahrav Jan 29, 2024
4d32d82
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 29, 2024
0effd07
fix tests
ahrav Jan 29, 2024
c259b84
Merge branch 'main' into feat-tmp-file-diffs
ahrav Jan 29, 2024
9f627e8
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 29, 2024
4ff883d
make linter happy
ahrav Jan 29, 2024
1485c9a
fix finalize
ahrav Jan 29, 2024
a2529f6
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 29, 2024
73a766f
address comments
ahrav Jan 30, 2024
93ce7ca
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 30, 2024
d503a4e
update test
ahrav Jan 30, 2024
da07843
address comments
ahrav Jan 30, 2024
1699097
lint
ahrav Jan 30, 2024
592d4e8
lint
ahrav Jan 30, 2024
3ac95f5
remove guard
ahrav Jan 30, 2024
f436939
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 30, 2024
5bdbea4
fix test
ahrav Jan 30, 2024
e442142
Merge branch 'main' into feat-tmp-file-diffs
ahrav Jan 30, 2024
9c07992
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 30, 2024
bff34a4
fix
ahrav Jan 30, 2024
fd5181e
Merge branch 'feat-tmp-file-diffs' of github.com:trufflesecurity/truf…
ahrav Jan 30, 2024
aa9daa2
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 30, 2024
1c577db
add TODO
ahrav Jan 30, 2024
2f27d64
Merge branch 'feat-tmp-file-diffs' into use-buffered-file-writer
ahrav Jan 30, 2024
8c02d0f
merge main
ahrav Jan 30, 2024
580e314
fix tests
ahrav Jan 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions hack/snifftest/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/source_metadatapb"
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/sourcespb"
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
"github.com/trufflesecurity/trufflehog/v3/pkg/sources/git"

Check failure on line 25 in hack/snifftest/main.go

View workflow job for this annotation

GitHub Actions / golangci-lint

could not import github.com/trufflesecurity/trufflehog/v3/pkg/sources/git (-: # github.com/trufflesecurity/trufflehog/v3/pkg/sources/git
)

var (
Expand Down Expand Up @@ -188,8 +188,16 @@

logger.Info("cloned repo", "repo", r)

s := git.NewGit(sourcespb.SourceType_SOURCE_TYPE_GIT, 0, 0, "snifftest", false, runtime.NumCPU(),
func(file, email, commit, timestamp, repository string, line int64) *source_metadatapb.MetaData {
cfg := &git.Config{
SourceName: "snifftest",
JobID: 0,
SourceID: 0,
SourceType: sourcespb.SourceType_SOURCE_TYPE_GIT,
Verify: false,
SkipBinaries: true,
SkipArchives: false,
Concurrency: runtime.NumCPU(),
SourceMetadataFunc: func(file, email, commit, timestamp, repository string, line int64) *source_metadatapb.MetaData {
return &source_metadatapb.MetaData{
Data: &source_metadatapb.MetaData_Git{
Git: &source_metadatapb.Git{
Expand All @@ -202,9 +210,8 @@
},
}
},
true,
false,
)
}
s := git.NewGit(cfg)

logger.Info("scanning repo", "repo", r)
err = s.ScanRepo(ctx, repo, path, git.NewScanOptions(), sources.ChanReporter{Ch: chunksChan})
Expand Down
Loading
Loading