Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optmiziation: Concurrent Patch generation #476

Closed
zricethezav opened this issue Dec 6, 2020 · 2 comments
Closed

Optmiziation: Concurrent Patch generation #476

zricethezav opened this issue Dec 6, 2020 · 2 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@zricethezav
Copy link
Collaborator

zricethezav commented Dec 6, 2020 •
edited

Is your feature request related to a problem? Please describe.
Generating patches is the biggest bottle neck. It would be great if we could get patch generation to run without race conditions while iterating through commits. This speeds up scans by a substantial amount

Race condition:

==================
WARNING: DATA RACE
Read at 0x00c0001ac170 by goroutine 35:
  github.com/go-git/go-git/v5/plumbing.(*MemoryObject).Size()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/memory.go:35 +0x3e
  github.com/go-git/go-git/v5/plumbing/cache.(*ObjectLRU).Put()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/cache/object_lru.go:50 +0x26e
  github.com/go-git/go-git/v5/storage/filesystem.(*ObjectStorage).getFromUnpacked()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/storage/filesystem/object.go:414 +0x5ac
  github.com/go-git/go-git/v5/storage/filesystem.(*ObjectStorage).EncodedObject()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/storage/filesystem/object.go:319 +0x655
  github.com/go-git/go-git/v5/storage/filesystem.(*Storage).EncodedObject()
      <autogenerated>:1 +0x8f
  github.com/go-git/go-git/v5/plumbing/object.GetCommit()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/object/commit.go:55 +0x91
  github.com/go-git/go-git/v5/plumbing/object.(*Commit).Parent()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/object/commit.go:129 +0x207
  github.com/zricethezav/gitleaks/v7/scan.(*CommitScanner).Scan()
      /Users/zacharyrice/Go/src/github.com/zricethezav/gitleaks/scan/commit.go:54 +0x1b4e
  github.com/zricethezav/gitleaks/v7/scan.(*RepoScanner).Scan.func2()
      /Users/zacharyrice/Go/src/github.com/zricethezav/gitleaks/scan/repo.go:87 +0x1c4
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /Users/zacharyrice/Go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x94

Previous write at 0x00c0001ac170 by goroutine 36:
  github.com/go-git/go-git/v5/plumbing.(*MemoryObject).Write()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/memory.go:55 +0x124
  io.copyBuffer()
      /usr/local/Cellar/go/1.15/libexec/src/io/io.go:411 +0x28b
  io.Copy()
      /usr/local/Cellar/go/1.15/libexec/src/io/io.go:368 +0x615
  github.com/go-git/go-git/v5/storage/filesystem.(*ObjectStorage).getFromUnpacked()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/storage/filesystem/object.go:416 +0x5ad
  github.com/go-git/go-git/v5/storage/filesystem.(*ObjectStorage).EncodedObject()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/storage/filesystem/object.go:319 +0x655
  github.com/go-git/go-git/v5/storage/filesystem.(*Storage).EncodedObject()
      <autogenerated>:1 +0x8f
  github.com/go-git/go-git/v5/plumbing/object.GetCommit()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/object/commit.go:55 +0x91
  github.com/go-git/go-git/v5/plumbing/object.(*Commit).Parent()
      /Users/zacharyrice/Go/pkg/mod/github.com/go-git/go-git/v5@v5.2.0/plumbing/object/commit.go:129 +0x207
  github.com/zricethezav/gitleaks/v7/scan.(*CommitScanner).Scan()
      /Users/zacharyrice/Go/src/github.com/zricethezav/gitleaks/scan/commit.go:54 +0x1b4e
  github.com/zricethezav/gitleaks/v7/scan.(*RepoScanner).Scan.func2()
      /Users/zacharyrice/Go/src/github.com/zricethezav/gitleaks/scan/repo.go:87 +0x1c4
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /Users/zacharyrice/Go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x94

Describe the solution you'd like
A clear and concise description of what you want to happen.
^ see above

Describe alternatives you've considered
Forking go-git and attempting to create read-only memory-objects? Perhaps using https://github.com/libgit2/git2go might be faster?

Additional context
Add any other context or screenshots about the feature request here.
Working branch: https://github.com/zricethezav/gitleaks/tree/concurrent-patch
Chunk I'm working on: https://github.com/zricethezav/gitleaks/blob/concurrent-patch/scan/repo.go#L50-L96

cc @zricethezav
@zricethezav zricethezav added enhancement New feature or request help wanted Extra attention is needed labels Dec 6, 2020
@zricethezav
Copy link
Collaborator Author

Related issue: go-git/go-git#48

@zricethezav
Copy link
Collaborator Author

https://github.com/zricethezav/gitleaks/releases/tag/v7.1.0 😬

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@zricethezav