Full chunk-based computing in coprocessor

[TennyZhuang]

Using TiDB Chunk format in the coprocessor framework instead of VectorValue.

Detail Design (Draft): tikv/rfcs#43

Development Task

• Implement type safe chunk format.
• Modify rpn_fn macro.
• Modify all builtin functions using new signature.

Mentor

• @TennyZhuang
• @breeswish

Please contact with mentors before start working on the task.

[ranjithkumar007]

I am interested in this project.
Are there any pre-requisite tasks to be completed to participate in this community bridge project?

[TennyZhuang]

Sorry for late reply. the issue is currently a draft without detailed design, but feel free to apply for the project.

Currently there are no pre-requisite tasks because we are still working on the RFC (link can be found in the issue description).

[championshuttler]

Hey @TennyZhuang ! I am willing to get mentored for this issue for the Community Bridge program. Can you please help me in getting started with it?

Thanks

[breezewish]

Still have some remaining tasks, re-open.

[choleraehyq]

Do we have any benchmark result about this?

[skyzh]

@choleraehyq Yep but I forgot where I put it. We have micro-benchmarks and we saw 10x performance improvements on varcher processing. However, there are not that much improvements in fixed-size types probably due to the compiler is not using vectorization instruction sets for these operations, also branching for Option.

[choleraehyq]

@choleraehyq Yep but I forgot where I put it. We have micro-benchmarks and we saw 10x performance improvements on varcher processing. However, there are not that much improvements in fixed-size types probably due to the compiler is not using vectorization instruction sets for these operations, also branching for Option.

But TiKV store values in row format, and converting from row format to chunk structure has cost. I'm curious that if this optimization does have positive impact for all workloads?

[skyzh]

TiKV / TiDB are already using column-based memory format before this work starts.