New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[15721]Index Tuning with RL #1338

Open
wants to merge 393 commits into
base: master
from

Conversation

Projects
None yet
8 participants
@saatviks
Contributor

saatviks commented May 4, 2018

Least-Squares Policy Iteration based Index tuning - Inspired from Cost-Model Oblivious DB Tuning by Basu et. al..
All the main source files are under src/brain/indextune:

  1. Compressed IndexConfig Container: A container that holds a compressed representation of possible index configurations and corresponding relevant functionality(eg. prefix closures).
  2. Compressed IndexConfig Util: A Util class for relevant functionality such as generating Add/Drop candidates, featurization, etc.
  3. LSPI Tuner(and related models): The actual tuning classes, and models for least squares value function and cost estimation.

We currently only have a few sanity tests for the end to end setup, but have tested the CompressedIndexConfig* classes and RLSEModel sufficiently.

Blade-Lee and others added some commits Apr 28, 2018

saatviks and others added some commits May 13, 2018

added the files for cost evaluation
llvm for mac

Basic classes

added the configuration enumeration files

Add Whatif API

Add optimizer cost query func skeleton

Complete what if API implementation. Testing pending.

1. Add test file in brain for what-if API.
2. Implement a basic test to insert some tuples and hypothetical
indexes and get the cost. (Not working)

Ignore query planning

Analyze tables was missing. Fixed it

fix the query

add comments, fix some code style

Fix whatif API test

run formatter

Add index selection module skeleton

skeleton for admissible column parsing

adding cost model classes

cleanup and reorganize the code

Intermediate changes. Query parser not complete.

Intermediate changes. Query parser not complete.

removed cost model class

Add IndexObject Pool

Memoization support completed

Complete query parser

Complete query parser

multi column index, wip

Add tests for admissible indexes

Fix what if index and admissive indexes test

added outline for naive enumeration method

Fix get admissible indexes test

Fix get admissible indexes test

Added the IndexConfiguration set difference

Minor BUg Fix

Split computing and getting const

Fix compilation error and typos

Finish Configuration Enumeration module

Fix the main index selection algorithm

Finish Merging

Merge

cleanup

Restructure code

More refactoring

added comments to index selection context

Added the comparator for the candidate index enumeration

Adding comments

Restructure generate candidate indexes

Fix merge

partial test for multi columnindex generation

Add candidate index gen test

Minor change to ComputeCost. Formatting and comments.

Add comments

comments

More formatting and comments.

more comments

brief comments.

rename pl_assert to peloton_assert

Remove GetCost and rename ComputeCost to GetCost

fix multicolumnindex generation

minor fixes

Fix admissible index and candidate pruning tests

Fix unused variables

Add more tests to WhatIfAPI and IndexSelection

Implement the suggestions mentioned in the code review

Uncomment the choose best plan call

Fix tests

Add support for multi-column index

Conflicts:
	src/optimizer/rule_impls.cpp
	src/optimizer/stats_calculator.cpp

Fix conflicts after merge

nit fixes

Fix what-if index tests

Add more multi-column index sets in the test cases.

Add testing utility class for index suggestion tests

Add to cmake for the files in the previous commit

Modify what-if tests to use the utility class

Fix formatting

Code review fix

fix tests

nit

Fix memory leaks and misc nit fixes

fixed the test temportarily for the index bug

Rename IndexObject to HypotheticalIndexObject

debugging the shared pointer issue

Fix segfault. Some more Renames

check the exact indexes

Fix the tests to use the util

fixing the index selection

Fix formatting

Rebase and fix conflicts while rebasing

latest tests

Better tests

Add get workload support to the testing utility class.

Fix stray

Comment out the debug code in optimizer

Add index suggestion task skeleton

Add query history catalog GET methods.

Fix formatting

Update index suggestion task

Add new workload

Add new test - incomplete

Add more than 3 columns cost model test

Fix join query parsing for table name extraction

Add more queries to workload D

DEBUG -> TRACE

Fix compilation error

Complete the index suggestion task - RPC is pending.

Changed the columns from a set to vector

Get args at RPC handler

Fix compilation issue and list serialization

Refactored the tests

Complete RPC handler

Fix compilation error in peloton-bin

fix logs

run brain and server together in one process for testing

Add dropIndex RPC

MOved tunable knobs into a separate structure

Fix index selection job -- rename some stuff

changed the arguments of the constructor

completed the refactor

Rename more stuff

 minor style changes

More renames

Fix DML statement handling in workload

Fix cost model bug for more than 2 column indexes

Add an extensive test on multi-column optimizer cost model test

Add drop indexes RPC

Run formatter

concrete test case to show the issues with non-deterministic set of indexes

Fix drop indexes

Fix a bug in config enumeration for case where no index is better

Fix the test that is failing non-deteministically due to the optimizer cost evaluation module

Fix formatter issue

Fix travis error

Use only one transaction for the entire run of the job. Also, generate stats for every run of the job

hopefully, final version of the algorithm

added multiple choices for the output

more index selection tests

Add missing populate index

Consider non-equality predicates for index scan in the cost model

fixed precision issues

Drop the indexes only if it is not suggested this time

minor fixes

Fix the AnalyzeStats crash

Fix: Index Selection returns empty set because the
catalog cache eviction is not done properly.

Fix a bug during where clause parsing to make it work with TPCC

Fix the compilation error

Address some of the code review comments

Fix create/drop index -- running TPCC

Fix analyze stats crash. Fix query history logging for PREPARED statements

Change knobs

More misc

addressing commits

Restructure code

Reformat code

small correction to make it compile in debug mode

remove the unnecessary commented parts of test and code

remove #define

Restructure code, fix nits

fix errors for compilation in debug mode

Restructure code

Run formatter

fix query logger test

trying to pass the compilation on travis

change debug logging to trace level logging

Fix warning in IndexConfigComparator

warning: the specified comparator type does not provide a const call
operator [-Wuser-defined-warnings]

trace-->debug

Hack to make travis pass the build.

DEFUALT_SCHEMA_NAME can't be found error. Fix this when merging with
master.

Hack to make travis pass the build.

DEFUALT_SCHEMA_NAME can't be found error. Fix this when merging with
master.

remove multiple of unnecessary debug statements
@coveralls

This comment has been minimized.

coveralls commented Jun 13, 2018

Coverage Status

Coverage decreased (-76.6%) to 0.0% when pulling 9ba691e on Blade-Lee:brain_rl_testing_framework into 196f663 on cmu-db:master.

@saatviks saatviks force-pushed the Blade-Lee:brain_rl_testing_framework branch from 9ba691e to 58a4ab5 Jul 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment