Expected speed boost from using goroutines?

[leehambley]

I've been experimenting with making a program run concurrently using goroutines, I've implemented my program as mentioned here https://gist.github.com/27e84d3dde014f4d63fe (with and without GOMAXPROCS)

I expected that the execution speed would improve (each modifier function makes a query to the database, and iterates over the rows in the result set, appending them to slices on the struct.) that seemed to be classical IO bound, and prime for concurrent execution.

When researching how best to parallelise my script, I started using goroutines as above (as against a weird channel based solution which made the code quite tough on the eyes) I stumbled upon the following:

Goroutines will yield on any kind of IO,syscalls,channel operations or
a call to runtime.Gosched()

This is exactly how so-called green threads work in Ruby, so I'm quite familiar with the concept...

I wanted to ask if this was something you needed to implement in mymysql, or whether you already had, or whether it's something you get for free by using the socket library.

Finally would be the query about whether you would expect to see an improvement from using goroutines in that kind of situation. Might it also be that I am reusing the same db object throughout all goroutines?

Thanks, not really sure if this is an issue, or a feature request or a cry for help. Feel free to close if it's way out of bounds, thanks.

[ziutek]

You probably use only one connection shared between multiple gorutines, so there will be no spectacular difference between using 2 or 20 gorutines. Use database/sql and mymysql/godrv to dynamically create pool of connections for more gorutines (or create such poll yourself). But even with database/sql and big connection pool, if you connect to only one MySQL server there will be probably no big difference between 20 and 200 gorutines.

Read documentation (mymysql, database/sql, MySQL) and see where are the limits that you can't exceed.

[leehambley]

@ziutek thanks for the detailed response, I had suspected in any case that the bottleneck was the local, unoptimised MySQL server, I will experiment with the database/sql and mysql/godrv packages and see if I see any improvement.