Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing #184 out of memory #185

Merged
merged 10 commits into from
Feb 2, 2023
Merged

Fixing #184 out of memory #185

merged 10 commits into from
Feb 2, 2023

Conversation

rusq
Copy link
Owner

@rusq rusq commented Jan 30, 2023

Fixes #184

  • benchmark and fixture generator
  • memoise message time
  • isolate init from the benchmark
  • optimised slicing
  • use slice of pointers, instead of slice of exportMessages
  • address by index, save by pointer

Running a benchmark for comparison:

go test -benchmem -run=^$ -bench ^BenchmarkByDate$ github.com/rusq/slackdump/v2/export

Test rig:

goos: darwin
goarch: amd64
pkg: github.com/rusq/slackdump/v2/export
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

Benchmark stats for 10,000 generated messages:

* Initial *
BenchmarkByDate-16    	      16	  67085111 ns/op	52772423 B/op	  368681 allocs/op

* Intermediate *
BenchmarkByDate-16            54          22102931 ns/op        43947099 B/op      30025 allocs/op

* Final *
BenchmarkByDate-16           258           4612715 ns/op         1638080 B/op      20023 allocs/op

For million messages:

* Intermediate *
BenchmarkByDate-16             1        4210968611 ns/op        4777708088 B/op  3000049 allocs/op

* Final *
BenchmarkByDate-16             2         813553898 ns/op        169678572 B/op   2000043 allocs/op

Summary:

  • 14.54 times faster
  • 19.7 times less Bytes allocated per operation
  • 17.7 times less allocations per operation

@rusq rusq merged commit d4b7c94 into master Feb 2, 2023
@rusq rusq deleted the i184-oom branch February 2, 2023 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OOM while exporting a large conversation
1 participant