Parallel queries

[GTANAdam]

Hello,
Has there been any implementations leveraging Go's concurrency capabilities?
I'm quite sure it wouldn't be hard to implement read-only parallel queries with sync.mutexes if it hasn't been done yet

[timsehn]

Agreed. This is high on our list. Since you asked we'll spec it out this week and give you a date.

…

On Sat, May 23, 2020 at 8:34 PM Adam ***@***.***> wrote: Hello, Has there been any implementations leveraging Go's concurrency capabilities? I'm quite sure it wouldn't be hard to implement read-only parallel queries with sync.mutexes if it hasn't been done yet — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#111>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJAR3E5PMBRQMGNDWOHR2LRTCIVJANCNFSM4NIW2KBA> .

[zachmu]

This is already possible, but hasn't been tested to our satisfaction yet. We could help in this effort if you have a use case that requires parallel execution.

When constructing an engine, use the WithParallelism method on the AnalyzerBuilder:

https://github.com/liquidata-inc/go-mysql-server/blob/master/sql/analyzer/analyzer.go#L51

It parallelizes on partitions, so if your data source can be partitioned it should be possible to run each query in parallel on each partition (up to some max) simultaneously. But like I said: we don't have full faith in the implementation yet, and could use help proving it works.

[zachmu]

We now have a convincing proof of concept that parallel execution works in dolt. Let us know if you try it out and find any issues.