Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of content collision detection #387

Closed
erikgeiser opened this issue Oct 20, 2021 · 1 comment · Fixed by #388
Closed

Improve performance of content collision detection #387

erikgeiser opened this issue Oct 20, 2021 · 1 comment · Fixed by #388

Comments

@erikgeiser
Copy link
Member

erikgeiser commented Oct 20, 2021

This issue was raised by @mjmjelde in #385.

In ExpandContentGlobs we try building the content list by appending only elements with unique destinations. This increases the complexity in way that it does not scale for large amount of files. I hacked together a POC solution where the collision detection is performed once near the end before sorting. The benchmarks are promising:

========= 10000 Content Entries =========
goos: darwin
goarch: amd64
pkg: github.com/goreleaser/nfpm/v2/files
cpu: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
BenchmarkExpandContentGlobs-4       	       1	8943350742 ns/op	5948665080 B/op	 2963056 allocs/op
BenchmarkExpandContentGlobsFast-4   	       2	 690795220 ns/op	503257972 B/op	 1260368 allocs/op
PASS
ok  	github.com/goreleaser/nfpm/v2/files	11.305s

========= 100000 Content Entries =========
goos: darwin
goarch: amd64
pkg: github.com/goreleaser/nfpm/v2/files
cpu: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
BenchmarkExpandContentGlobs-4       	       1	1116330817406 ns/op	545598423120 B/op	185560019 allocs/op
BenchmarkExpandContentGlobsFast-4   	       1	25918381508 ns/op	41183230936 B/op	12604194 allocs/op
PASS
ok  	github.com/goreleaser/nfpm/v2/files	1142.497s

For large packages, this can be more than a 40x speedup.

I'll submit a PR if I find the time to clean up the very hacky POC.

@erikgeiser erikgeiser changed the title Improve performance of content de-duping Improve performance of content collision detection Oct 23, 2021
@erikgeiser
Copy link
Member Author

See also #393

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant