Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Window function pipelining tasks tracker #23807

Closed
7 tasks done
ichn-hu opened this issue Apr 1, 2021 · 2 comments
Closed
7 tasks done

Window function pipelining tasks tracker #23807

ichn-hu opened this issue Apr 1, 2021 · 2 comments
Labels
type/enhancement

Comments

@ichn-hu
Copy link
Contributor

ichn-hu commented Apr 1, 2021 •
edited

Development Task

In #18444, we identified an OOM case caused by window function (namely row_number), and we proposed to refactor the window function implementation in a pipeplined manner in feature request #21636, and we later wrote a proposal on it #23028, and this issue is meant for tracking the development process.

Development process

  • Create PipelinedWindowExec based on current implementation and modify the windowProcessor interface (see PR23022)
  • Change data flow, make Next() pulling data from windowProcessor, and windowProcessor calls fetchChild and process data at maximum effort
  • Modify RN to be SlidingWindowAggFunc, and add planner support (expect 04-19 ~ 04-21, 3 days)
    • update (04-22) RN now worked.
  • Modify Slide semantic and add FinishUpdate function on SlidingWindowAggFunc interface, and modify correspondingly on each window function (expect 04-22 ~ 04-26, 4 days)
  • Done pipelining for SlidingWindowAggFunc, add test to make sure it is correct (expect 04-27~29, 3 days)
  • Add test for RN (expect 04-30, 1 day)
  • Benchmark, make sure it has constant memory consumption and no execution time regression (expect 05-06 ~ 05-11, 5 days)
@ichn-hu
Copy link
Contributor Author

ichn-hu commented Apr 30, 2021

got some preliminary benchmark result

  • memory usage looks good when number of rows is small, but not expected when large
  • time consumption is higher

I'll continue invesitigate the problems.

/tmp/___BenchmarkWindowRows_in_github_com_pingcap_tidb_executor.test -test.v -test.paniconexit0 -test.bench ^\QBenchmarkWindowRows\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/pingcap/tidb/executor
cpu: AMD Ryzen 9 3900X 12-Core Processor            
BenchmarkWindowRows
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:1._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:1._pipelined:false)-24         	    4444	    442796 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:1._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:1._pipelined:true)-24          	    2247	    556846 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:2._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:2._pipelined:false)-24         	    1764	   1418121 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:2._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:2._pipelined:true)-24          	     428	   2846262 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:4._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:4._pipelined:false)-24         	    1719	   3684257 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:4._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:1000,_sorted:false,_concurrency:4._pipelined:true)-24          	     144	   9005981 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:1._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:1._pipelined:false)-24        	    2064	    614268 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:1._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:1._pipelined:true)-24         	    2004	    679962 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:2._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:2._pipelined:false)-24        	    1725	   1549281 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:2._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:2._pipelined:true)-24         	     385	   3133392 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:4._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:4._pipelined:false)-24        	    1675	   4243138 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:4._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:1000,_sorted:false,_concurrency:4._pipelined:true)-24         	     138	  10466583 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:1._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:1._pipelined:false)-24      	    1280	    896855 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:1._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:1._pipelined:true)-24       	    1164	    994761 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:2._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:2._pipelined:false)-24      	    1324	   1538329 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:2._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:2._pipelined:true)-24       	     546	   2787313 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:4._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:4._pipelined:false)-24      	    1506	   4401381 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:4._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:1000,_sorted:false,_concurrency:4._pipelined:true)-24       	     171	   8665594 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:1._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:1._pipelined:false)-24       	      39	  33590118 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:1._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:1._pipelined:true)-24        	      58	  34155193 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:2._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:2._pipelined:false)-24       	      26	  53165930 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:2._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:2._pipelined:true)-24        	      20	  51364356 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:4._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:4._pipelined:false)-24       	      25	  50302004 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:4._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1,_rows:100000,_sorted:false,_concurrency:4._pipelined:true)-24        	      27	  50142253 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:1._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:1._pipelined:false)-24      	      20	  56432256 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:1._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:1._pipelined:true)-24       	      21	  55623019 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:2._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:2._pipelined:false)-24      	      20	  58194219 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:2._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:2._pipelined:true)-24       	      21	  51803826 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:4._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:4._pipelined:false)-24      	      39	  38653039 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:4._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:10,_rows:100000,_sorted:false,_concurrency:4._pipelined:true)-24       	      31	  35460337 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:1._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:1._pipelined:false)-24    	      19	  87690624 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:1._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:1._pipelined:true)-24     	      19	  86903690 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:2._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:2._pipelined:false)-24    	      16	  67230612 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:2._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:2._pipelined:true)-24     	      18	  62301687 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:4._pipelined:false)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:4._pipelined:false)-24    	      24	  46989541 ns/op
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:4._pipelined:true)
BenchmarkWindowRows/(func:row_number,_aggColType:double,_numFunc:1,_ndv:1000,_rows:100000,_sorted:false,_concurrency:4._pipelined:true)-24     	      24	  45058475 ns/op
PASS

Process finished with the exit code 0

@ichn-hu
Copy link
Contributor Author

ichn-hu commented Jun 11, 2021

development done in #23022

@ichn-hu ichn-hu closed this as completed Jun 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ichn-hu