Golang concurrency practice
I took the example problem from Sameer Ajmani's blog Go concurrency pipeline and cencellation, wrote a fan-in-out pipeline,benchmark tests and improved the scalability to calculate the checksum of files concurrently. Some details are described below.
go to folder- go_pipeline/file_pipeline
bounded_file_checksum.go - code file
file_pipeline_test.go - Benchmark test file
change the path variable to a local folder(having some mb of data) of your machine.
run - go build bounded_file_checksum.go ./bounded_file_checksum.exe
flags option -
-gr - pass the number of goroutine you want to launch for hash calculation. -printresult - print the result of each file otherwise retruns a map.
for more get help - go run .\bounded_file_checksum.go -h
- Minimum sync block of goroutines due to channel syncronisation.
- Graceful error handling and cencelling handling (SIGTERM) to avoid any memory memory leak.
- Written beanchmark tests.
- Collect profiling data ( cpuprofile,blocking profile, goroutine trace etc attached in go_pipeline/file_pipeline folder ) and understand every espect of excution.
- Avoided reading full file in memory and doing operation at once which was causing too frequent GC, and used the io.Writer interface of MD5 and process chunk by chunk.
go test -bench=BenchmarkCalculate -benchtime=30s
goos: windows goarch: amd64
| BenchmarkFunction | total calls | time taken per call |
|---|---|---|
| BenchmarkCalculate1-16 | 5 | 7059610060 ns/op |
| BenchmarkCalculate4-16 | 18 | 1840480306 ns/op |
| BenchmarkCalculate8-16 | 37 | 952633289 ns/op |
| BenchmarkCalculate16-16 | 67 | 536108457 ns/op |
| BenchmarkCalculate32-16 | 64 | 553310138 ns/op |
| BenchmarkCalculate64-16 | 63 | 568016006 ns/op |
PASS
where suffix 1-16 means that one goroutine was launched on 16 represents the numberr of cores on my machine.
The second column means how many time program was executed in 30 seconds and last columns says time per execution.
Note that using one goroutine we are doing the task in around 7 seconds while with 16 goroutines we are close to one 500 millisecond.
Make this frameworrk more generic.