ARROW-17455: [Go] Function and Kernel execution architecture #13964
Conversation
This implements the interface for Kernels along with a `KernelSignature` struct and the type matching for `InputType` and `OutputType` via a `TypeMatcher` interface. Along with tests for all of the above. Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
go/arrow/compute/exec.go
Outdated
| ctx, cancel := context.WithCancel(context.Background()) | ||
| defer cancel() | ||
|
|
||
| ch := make(chan Datum, 10) |
There was a problem hiding this comment.
10 looks without strong support, but I don't have better suggestion.
Is channel strictly necessary here? I think channel make the code easier to read than C++ which collects all results with a datum accumulator.
There was a problem hiding this comment.
You're correct that a channel is not strictly necessary here, I did it for the reason you suggested: it makes it more readable than the C++ using a datum accumulator.
I'll make the 10 a configurable setting via the ExecCtx and just leave 10 as the default for now. Ultimately i just wanted to make sure that neither the Exec nor the WrapResults got blocked in most cases on how quickly the execution works.
|
@cyb70289 you can see some simple kernels in the tests which are used to test the infrastructure 😄 |
|
@cyb70289 also following this I'm already starting to implement kernels for various functions, so aside from the tests there'll also be lots of implemented kernels to use for examples. |
|
Benchmark runs are scheduled for baseline = 48e5e16 and contender = 7e7b8e1. 7e7b8e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
…13964) In addition to implementing the function execution architecture, this also shifts some files around for better package naming and scoping. A new package `exec` is created inside of `compute/internal/` to make it easier to scope other internal-only methods / functionality such as the kernel implementations themselves. This is a fairly large change because so much is interconnected, but it also includes extensive tests covering ~85% of the `internal/exec` package and nearly 80% of the entire `compute` package. More tests will be added as functionality is added. Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…13964) In addition to implementing the function execution architecture, this also shifts some files around for better package naming and scoping. A new package `exec` is created inside of `compute/internal/` to make it easier to scope other internal-only methods / functionality such as the kernel implementations themselves. This is a fairly large change because so much is interconnected, but it also includes extensive tests covering ~85% of the `internal/exec` package and nearly 80% of the entire `compute` package. More tests will be added as functionality is added. Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…13964) In addition to implementing the function execution architecture, this also shifts some files around for better package naming and scoping. A new package `exec` is created inside of `compute/internal/` to make it easier to scope other internal-only methods / functionality such as the kernel implementations themselves. This is a fairly large change because so much is interconnected, but it also includes extensive tests covering ~85% of the `internal/exec` package and nearly 80% of the entire `compute` package. More tests will be added as functionality is added. Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
In addition to implementing the function execution architecture, this also shifts some files around for better package naming and scoping. A new package
execis created inside ofcompute/internal/to make it easier to scope other internal-only methods / functionality such as the kernel implementations themselves.This is a fairly large change because so much is interconnected, but it also includes extensive tests covering ~85% of the
internal/execpackage and nearly 80% of the entirecomputepackage. More tests will be added as functionality is added.