Compilation OU and Metrics #1634

17zhangw · 2021-07-28T01:37:34Z

Description

PR augments the system with ability to model compilation. This PR introduces the following set of changes to the system:

System-wide enable_compilation_cache which determines whether or not we recompile the module upon subsequent query invocations.
Introduces a CompilationOperatingUnit that describes the instruction size, data size, number of functions, and number of static locals for each compilation module
compilation_runner that compiles valid TPL files based on sample_tpl/tpl_tests.txt to generate training data for the compilation OU.
Added a new COMPILATION metric which handles recording compilation OUs
Modifications to ou_model_trainer to handle compilation.csv

Remaining Tasks

Modify test to check compilation model train/infer

noisepage-checks · 2021-07-28T21:39:06Z

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

codecov · 2021-07-28T21:39:06Z

Codecov Report

Merging #1634 (574b924) into master (c7e6949) will decrease coverage by 0.03%.
The diff coverage is 58.86%.

@@            Coverage Diff             @@
##           master    #1634      +/-   ##
==========================================
- Coverage   81.09%   81.06%   -0.04%     
==========================================
  Files         761      763       +2     
  Lines       55169    55289     +120     
==========================================
+ Hits        44741    44819      +78     
- Misses      10428    10470      +42

Impacted Files	Coverage Δ
...g/modeling/training_util/data_transforming_util.py	`93.84% <ø> (ø)`
src/execution/ast/context.cpp	`90.57% <ø> (ø)`
src/execution/ast/type.cpp	`87.50% <ø> (ø)`
src/execution/exec/execution_context.cpp	`76.00% <ø> (ø)`
src/execution/sql/sorter.cpp	`97.28% <ø> (ø)`
src/include/execution/compiler/codegen.h	`100.00% <ø> (ø)`
...c/include/execution/compiler/compilation_context.h	`100.00% <ø> (ø)`
src/include/execution/exec/execution_context.h	`96.42% <ø> (ø)`
src/include/metrics/bind_command_metric.h	`0.00% <ø> (ø)`
src/include/metrics/compilation_metric.h	`0.00% <0.00%> (ø)`
... and 40 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7e6949...574b924. Read the comment docs.

noisepage-checks · 2021-07-29T03:30:04Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-3.86%	tpcc	RAM disk	Details master tps=22187.14, commit tps=21330.89, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-2.04%	tpcc	None	Details master tps=29900.57, commit tps=29291.0, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-2.29%	tpcc	HDD	Details master tps=22087.94, commit tps=21581.15, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-0.86%	tatp	RAM disk	Details master tps=6678.51, commit tps=6621.0, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-5.64%	tatp	None	Details master tps=7523.04, commit tps=7098.86, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.81%	tatp	HDD	Details master tps=6683.32, commit tps=6495.64, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-07-29T18:59:04Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
-4.69%	tpcc	RAM disk	Details master tps=22379.94, commit tps=21330.89, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-0.02%	tpcc	None	Details master tps=29295.41, commit tps=29291.0, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-3.34%	tpcc	HDD	Details master tps=22327.24, commit tps=21581.15, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
0.19%	tatp	RAM disk	Details master tps=6608.54, commit tps=6621.0, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-4.66%	tatp	None	Details master tps=7445.61, commit tps=7098.86, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.51%	tatp	HDD	Details master tps=6663.21, commit tps=6495.64, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-07-29T21:11:57Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-3.84%	tpcc	RAM disk	Details master tps=22379.94, commit tps=21521.14, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-1.78%	tpcc	None	Details master tps=29295.41, commit tps=28775.38, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-3.67%	tpcc	HDD	Details master tps=22327.24, commit tps=21507.52, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-1.79%	tatp	RAM disk	Details master tps=6608.54, commit tps=6490.3, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-7.49%	tatp	None	Details master tps=7445.61, commit tps=6887.76, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-1.72%	tatp	HDD	Details master tps=6663.21, commit tps=6548.86, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-07-30T00:02:34Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-3.84%	tpcc	RAM disk	Details master tps=22379.94, commit tps=21521.14, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-1.78%	tpcc	None	Details master tps=29295.41, commit tps=28775.38, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-3.67%	tpcc	HDD	Details master tps=22327.24, commit tps=21507.52, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-1.79%	tatp	RAM disk	Details master tps=6608.54, commit tps=6490.3, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-7.49%	tatp	None	Details master tps=7445.61, commit tps=6887.76, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-1.72%	tatp	HDD	Details master tps=6663.21, commit tps=6548.86, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-07-30T00:37:38Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
-6.2%	tpcc	RAM disk	Details master tps=22379.94, commit tps=20992.26, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-2.26%	tpcc	None	Details master tps=29295.41, commit tps=28634.76, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-5.32%	tpcc	HDD	Details master tps=22327.24, commit tps=21139.2, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-1.8%	tatp	RAM disk	Details master tps=6608.54, commit tps=6489.39, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-5.01%	tatp	None	Details master tps=7445.61, commit tps=7072.3, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.19%	tatp	HDD	Details master tps=6663.21, commit tps=6517.04, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-11-05T17:15:38Z

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

17zhangw added 6 commits July 26, 2021 16:11

Bypass compilation cache

04dee05

Huge diff that includes feature changes

0957df9

revert TPL

af153bb

Script/model training

ee557b8

check clang tidy

044ab99

Merge remote-tracking branch 'cmudb/master' into compilation_metrics

b3ca1b0

17zhangw self-assigned this Jul 28, 2021

17zhangw added in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. ready-for-ci Indicate that this build should be run through CI. labels Jul 28, 2021

17zhangw added 4 commits July 28, 2021 14:48

Merge remote-tracking branch 'cmudb/master' into compilation_metrics

c1b2fd0

Add testing infra

2c67188

Purge Jenkinsfile for testing

eb82708

Another fix

9f57cbb

17zhangw added 2 commits July 28, 2021 18:42

Revert Jenkinsfile

114be9b

Revert metrics_test

574b924

17zhangw requested a review from linmagit July 29, 2021 07:21

17zhangw added the ready-for-review This PR passes all checks and is ready to be reviewed. Mark PRs with this. label Jul 29, 2021

Merge branch 'master' into compilation_metrics

3df85c0

Merge branch 'master' into compilation_metrics

a53a989

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compilation OU and Metrics #1634

Compilation OU and Metrics #1634

17zhangw commented Jul 28, 2021 •

edited

noisepage-checks bot commented Jul 28, 2021

codecov bot commented Jul 28, 2021 •

edited

noisepage-checks bot commented Jul 29, 2021

noisepage-checks bot commented Jul 29, 2021

noisepage-checks bot commented Jul 29, 2021

noisepage-checks bot commented Jul 30, 2021

noisepage-checks bot commented Jul 30, 2021

noisepage-checks bot commented Nov 5, 2021

Compilation OU and Metrics #1634

Are you sure you want to change the base?

Compilation OU and Metrics #1634

Conversation

17zhangw commented Jul 28, 2021 • edited

Description

Remaining Tasks

noisepage-checks bot commented Jul 28, 2021

Performance Boost!

codecov bot commented Jul 28, 2021 • edited

Codecov Report

noisepage-checks bot commented Jul 29, 2021

Major Decrease in Performance

noisepage-checks bot commented Jul 29, 2021

Minor Decrease in Performance

noisepage-checks bot commented Jul 29, 2021

Major Decrease in Performance

noisepage-checks bot commented Jul 30, 2021

Major Decrease in Performance

noisepage-checks bot commented Jul 30, 2021

Major Decrease in Performance

noisepage-checks bot commented Nov 5, 2021

Performance Boost!

17zhangw commented Jul 28, 2021 •

edited

codecov bot commented Jul 28, 2021 •

edited