Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBMain and Settings Manager #336

Merged
merged 666 commits into from May 27, 2019

Conversation

@DarkForte
Copy link
Contributor

DarkForte commented Apr 6, 2019

Added the Settings Manager.

Finished:

  • Peloton code migration
  • Callback framework
  • gflags
  • migrate to TransientValue
  • An example of mutable parameter (object pool size)
yeshengm and others added 8 commits Apr 4, 2019
…arator and peek function; TODO: migrate to TransientValue
@DarkForte DarkForte self-assigned this Apr 6, 2019
YuzeLiao and others added 19 commits Apr 6, 2019
… into settings_manager
… into 15-721-project2-stats
wenxuanq
… into 15-721-project2-stats
Copy link
Collaborator

mbutrovich left a comment

We'll keep refining the SettingsManager as we start using it more.

@mbutrovich mbutrovich requested review from yeshengm and removed request for yeshengm May 27, 2019
@mbutrovich mbutrovich dismissed stale reviews from lagoon0o0, yeshengm, and Calculuser May 27, 2019

Outdated.

@mbutrovich mbutrovich merged commit e7c4d2b into cmu-db:master May 27, 2019
2 of 4 checks passed
2 of 4 checks passed
codecov/patch 57.83% of diff hit (target 87.6%)
Details
codecov/project 87.48% (-0.13%) compared to 3d88f8c
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
17zhangw added a commit to 17zhangw/terrier that referenced this pull request May 30, 2019
commit 3633d67
Author: Tianyu Li <tli2@andrew.cmu.edu>
Date:   Wed May 29 22:45:33 2019 -0400

    Add ManagedPointer (cmu-db#394)

    * Add ManagedPointer

    * remove redundant includes.

    * rename to managed_ptr.h

commit c3d7675
Author: tanujnay112 <tanujnay112@live.com>
Date:   Wed May 29 12:22:33 2019 -0700

    Fix typo in network README (cmu-db#354)

commit e7c4d2b
Author: DarkZero <weichenk@andrew.cmu.edu>
Date:   Mon May 27 09:15:14 2019 -0700

    DBMain and Settings Manager (cmu-db#336)

commit 3d88f8c
Author: Gustavo Angulo <GustavoAngulo@users.noreply.github.com>
Date:   Thu May 23 23:12:43 2019 -0400

    Fix bulk insert for InsertPlanNode (cmu-db#389)

    * Change bulk insert for InsertPlanNode

    * Fix return value error

    * Fix comment

    * Add TODO to flatten vector in the future

    * I hate clang-tidy

commit 0586bbf
Author: Matt Butrovich <mbutrovich@users.noreply.github.com>
Date:   Tue May 21 16:14:37 2019 -0400

    Add dedicated GC thread class for tests and main to use. (cmu-db#385)

commit be93dca
Author: Wan Shen Lim <wanshenl@andrew.cmu.edu>
Date:   Fri May 17 07:49:10 2019 -0400

    Extend parser CREATE INDEX (cmu-db#376)

    * Parse the expression in CREATE INDEX, with tests. Note that the Postgres parser's node either comes with a name or an expression, not both.

    * Doxygen.

commit c1c3979
Author: Matt Butrovich <mbutrovich@users.noreply.github.com>
Date:   Thu May 16 16:46:27 2019 -0400

    BwTreeIndex: Add visibility logic, abort and deferred delete logic, descending and limit scans (cmu-db#356)

commit e897de8
Author: Matt Butrovich <mbutrovich@users.noreply.github.com>
Date:   Thu May 16 15:13:13 2019 -0400

    Move clang-tidy to Jenkins. (cmu-db#382)

commit c9eb369
Author: Zhang Yuning <codeworm96@outlook.com>
Date:   Thu May 16 10:28:03 2019 -0400

    Prevent Travis from killing clang-tidy (cmu-db#378)

    * Fix CI: update progress for each file to keep CI alive

    * Make check-clang-tidy do per file reporting

commit 72eeb65
Author: Yesheng Ma <kimi.ysma@gmail.com>
Date:   Wed May 15 12:59:13 2019 -0400

    Fix wrong cmake "if" usage in FindPOSTGRES (cmu-db#368)
apavlo added a commit that referenced this pull request Jun 5, 2019
* Ported LogicalFilter and LogicalProjection

* Completed LogicalFilter + LogicalProjection

* logical dependent join; logical mark join

; sync operator== functions' param names

* Completed LogicalInsert + LogicalInsertSelect

* logical inner/single join

* logical left/right join

* Completed LogicalDistinct + LogicalLimit

* logical outer/semi join

* Completed LogicalUpdate + LogicalDelete

* logical groupby

* Completed LogicalExportExternalFile

*  'node' or 'r', that is a question

* Directly compare 2 vectors with ==

Majority of the vector comparison have not been changed to this form (originally the code traverse the 2 vectors and compare each element pair

* add hash and == for logical outer/semi/left/right join

(physical) outer/semi/right/left xxxx join might still not have hash or == function

* linting & const removal

* Comparison of AnnotatedExpression

* Visit function for logical operators

* linting

* use auto when initializing with new to avoid duplicating the type name

* correct operator test

* format the testcases for operator test

nice & tidy now

* clang-tidy lint

* lint: check-format

* doxygen fix

* fix type

* check-format

* Fix bulk insert for InsertPlanNode (#389)

* Change bulk insert for InsertPlanNode

* Fix return value error

* Fix comment

* Add TODO to flatten vector in the future

* I hate clang-tidy

* New tests for HashUtil

* Adding accessor methods and test cases for LogicalExportExternalFile

* basic test loigical get/ExternalFileGet/QueryDerivedGet; correct typo

* Ah, linting!

* LogicalGet: getters and non-trivial testcases

* LogicalExternalFileGet: getters and non-trivial testcases

* LogicalQueryDerivedGet: getters and non-trivial testcases

* Partial fix for using HashUtil::Hash with strings. We still need to fix HashUtil::CombineHashInRange

* Added test case for HashUtil::SumHashes

* Added new LogicalExportExternalFile test

* Change hashing of vector; logicalDependentJoin test&getters;

Change in some operator hash function:
from iterate the loop and combineHash each entry
to using HashInRange function

* logicalMarkJoin: getter& test

* LogicalSingleJoin: getter & test

* Logicalinnerjoin: getter & test

* Working on expanding LogicalUpdate

* LogicalLeftJoin: getter & testcase; comment out hash tests in operators

* Dirty hack for supporting HashUtil::Hash() on string literals

* Fixed up test case for LogicalUpdate. Switch its internal vector to use raw UpdateClause pointers. This may be a bad idea...

* DBMain and Settings Manager (#336)

* Added getters + tests for LogicalDistinct and LogicalLimit

* Added getters + tests for LogicalInsert and LogicalInsertSelect

* some linting

* annotated expr, logical right/left join compare by content

TODO: annotated expr testcase?

* logical left right join test

* logical outer join: getter and test

* logical semi join: getter & test

* logical projection and filter

* linting

* Added getters + tests for LogicalDelete and LogicalUpdate

* Ran format + lint. The terrier is watching me as I type this...

* Added a bunch NOLINT hints about copies of values in tests where we do want a copy

* Only check for TERRIER_ASSERT on debug builds in LogicalInsertTest

* Add ManagedPointer

* remove redundant includes.

* better hashing vector of annotatedExpression

* Fix typo in network README (#354)

* prevent undefined behavior when hashing with null involved

mainly when annotated_expr->GetExpr()->Hash() is called.

* rename to managed_ptr.h

* Add ManagedPointer (#394)

* Add ManagedPointer

* remove redundant includes.

* rename to managed_ptr.h

* Traffic cop with sqlite (#328)

* use ManagedPointer: queryDerivedGet

* a bit of linting

* use ManagedPointer: LogicalQueryDerivedGet

* a bit of linting

* use ManagedPointer: logicalProjection

* use ManagedPointer in logical ops and annotatedExpression

also add boolean conversion in the managedPointer

* linting

* New test cases for ManagedPointer. We mostly check to make sure that it has the same semantics as std::shared_ptr. I am writing this outside of Gates while the terrier runs around looking for rabbits... (#396)

* linting

* logical limit/update w imperfect fix in updateClause

* logicalget test

* logical filter

* add const

* logical Op cleaning up

* raw to managed in logicalInsert

* lint

* Revert "logical Op cleaning up"

This reverts commit 087e3c9.
apavlo added a commit that referenced this pull request Jun 5, 2019
* Bugfix: nonexistent column access would always map to smallest column.

* Address requested changes, replace magic numbers with #define, renumber OID to always reserve 0.

* Remove duplicate definition of db_oid_t

* add base operator nodes

* initial bring over operators

* more operator changes

* porting over physical operators

* Make oid increment atomic

* remove sql expressions

* lint update

* lint update

* lint update

* lint update

* lint update

* format update

* lint update

* lint update

* lint update

* Only one counter

* Add string interface

* Format

* Revert logger level

* Add missing documentation

* Add more documentation

* Add tablespace_handle and create pg_class

* 1) add tablespace handle and tablespace handle test
2) document what intergers are used to represent strings
3) add table handle and table handle test

* Use a global counter

* Fix documentation

* Minor fixes

* Add documentation for throwing exceptions

* Change tracing message

* WIP - use Sqltable helper class for catalog operations (not complete)

* WIP

* WIP - modify tablespace to use sql table rw wrapper

* WIP - change namespace handles and entries to use sql table rw wrapper

* Template for create table access hierachy

* WIP - cleanup database handle conversion to sql table rw helper

* WIP - support for varlens in pg_database and pg_tablespace

Note: table_handle_test fails, as a consequence of above changes.

* WIP - updates to comments

* Fix merge

* Fix table handle to use SQLTableRW

* Fix sql table memory bug

* Use Varlen in catalog

* Pass clang-tidy

* Documentation

* Pass check-lint

* Add proper type checking in As()

* Remove test util in catalog sql table

* Removed unused functions in catalog sql_table

* Clean up catalog APIs

* Add create namespace test

* Add public namespace

* Add create table and. Make table handle only contain tables under its namspace

* Fix memory leaks

* doxygen

* Add a column to store sql tables

* Add operator tests

* fix clang-tidy

* format

* format

* fix lint errors

* fix lint errors

* fix an operator test

* rename optimizer_def.h to optimizer_defs.h

* rename optimizer_def.h to optimizer_defs.h

* remove optimizer_def.h

* rename test directory sql->optimizer

* rename test directory sql->optimizer

* rename test directory sql->optimizer

* rename test directory sql->optimizer

* format

* WIP - Updates to catalog sql table

- Use scan for searching sql tables
- Add search using a vector of values, returning a similar row
  (vector of values)

* fix typos

* WIP - modify Entries to be Value vectors.

- unittest, format and lint clean
- Deprecated code not removed yet.

* WIP - add support for unused catalog columns

- adds unused columns to pg_database
- remove misc. deprecated code

* add index oids to update, insert, delete

* update operator test

* Add unsed cols in other catalogs

* use transient values

* format

* Make things compile

* Documentations

* Add pg_attribute skeloton

* Add function for insert a vector of values as a row into sql table

* Add copy assignment operator for value

* Add back the value type test

* Make table handle to return Value type

* Remove deprecated FindRow functions

* Remove old handle entries

* Make return type consistant

* Remove more unused functions

* Keep SetXXXColInRow because InsertRow is noticeably slower. Add comments to address that

* Add a function for inserting a value into a row

* Fix value type bug

* Remove SetXXXColInRow functions and fix bugs

* Use SetColInRow instead of SetXXXColInRow for each type

* Remove more SetXXXColInRow function

* Bootstrap pg_attribute

* Add attribute handle and test

* Format

* placeholder for pg_type

* - catalog benchmark, performance fixes and cmake bug fix

* Formatting, lint etc.

* - switch to using InsertRow to make insertion thread safe
- add catalog_test
- fix a couple of bugs in Values, since we haven't transitioned
  to TransientValues yet.
- formatting and lint fixes

* WIP - Add support for database delete.
- passes unit test, but not clang-tidy clean
- small TODOs still outstanding for delete.

* WIP - terminate transaction in the catalog tests

* WIP - fix table access to check for nulls, fix clang-tidy

Fixes all but one clang-tidy complaint, in type_handle.h

* format & fix linter errors & pg_type WIP

* built-in types for pg_type

* add type handle impl

* WIP - cleanups. Need to be extended to remaining catalog classes

- move schema column definitions out of catalog and into each class
- add Create method to each class. Invoked during bootstrapping
- add AddEntry method to class. Ensures a consistent way of handling
  unused columns
- add ClassHandle and ClassEntry

* Comment update

* Small TODOs and cleanups

* More TODO resolution and cleanups

* - changes to DeleteEntry and some debug support

- DeleteEntry is now a Handle method, to avoid adding unnecessary state
  to the Entry classes.
- Add a Dump method for debugging (making catalog contents visible)

* Remove temp. Dump calls from catalog test

* Debug support + call initialize of pg_type

* Make pg_attribute rows link correctly to pg_type

- update pg_attribute intiialization
- change catalog initialization to allow pg_attribute to be correctly
  populated
- temporarily disable deletion of use tables on shutdown

* change SchemaCols to SchemaCol

* addressing review comments

* Format, lint and clang-tidy fixes

* Small cleanups

* savepoint

* add doc & pg_type unittests

* roll back a bit

* Add pg_settings catalog

* Settings fixes (mostly)

- fix typo in settings column numbering
- initialize with global catalogs
- add global method to get handle
- add trivial unit test
- add assertion (all catalogs) that all columns are initialized.
  Unset varchar columns in particular, may generate segfaults if
  access is attempted.

* Settings catalog test

* WIP Add comments, cleanup on tablespace handle, update to Dump

* WIP DeleteDatabase test, stubs for DeleteTable

* Correct the schema for pg_settings

* Correct pg_settings schema + update test to match schema

* Add more documentation for TableEntry

* Support for deleting catalog entries

* huge commit for refactoring

* PeekVarChar usage fixes, so we don't leak memory

* fix leaks and migrate to string_view

* Compilation fixes

* fix doc

* fix jenkins

* move catalog entries outside catalog handles, prep for later refactoring

* refactored entries

* add test commit

* - modify Attribute handle to be use a table_oid rather than keeping
  the table ptr
- Add methods to NamespaceEntry to get ctypes out by specifying column name

* Formatting

* Fix documentation

* Doc fix

* Partial review fixes

* Review fixes - remove deprecated code from AttributeHandle

* Quick fix for issue #339, segfault in catalog due to transaction
deletion after bootstrapping. This is until a more comprehensive fix
can be created.

* Format and tidy fixes

* More review fixes and doc fixes

* More review fixes

* More review fixes

* Use raw pointers for SqlTableRW instances

* Partial changes for namespace fixes

* WIP - namespace support changes

* Documentation updates

* Doc and format fixes

* More review changes

- merge used and unused columns when creating tables
- remove unused methods

* Add more (unused) columns to pg_database, for compatibility

* Cleanups

* remove catalog refs

* checkpoint

* operators cleanup #1

* operator cleanup #2

* operator cleanup #3

* operator cleanup #4

* operator cleanup #5

* Simple fixes to get operators.h to compile. This is still incomplete (#387)

* Ported LogicalFilter and LogicalProjection (#388)

* Logical operators (#391)

* Ported LogicalFilter and LogicalProjection

* Completed LogicalFilter + LogicalProjection

* logical dependent join; logical mark join

; sync operator== functions' param names

* Completed LogicalInsert + LogicalInsertSelect

* logical inner/single join

* logical left/right join

* Completed LogicalDistinct + LogicalLimit

* logical outer/semi join

* Completed LogicalUpdate + LogicalDelete

* logical groupby

* Completed LogicalExportExternalFile

*  'node' or 'r', that is a question

* Directly compare 2 vectors with ==

Majority of the vector comparison have not been changed to this form (originally the code traverse the 2 vectors and compare each element pair

* add hash and == for logical outer/semi/left/right join

(physical) outer/semi/right/left xxxx join might still not have hash or == function

* linting & const removal

* Comparison of AnnotatedExpression

* Visit function for logical operators

* linting

* use auto when initializing with new to avoid duplicating the type name

* correct operator test

* format the testcases for operator test

nice & tidy now

* clang-tidy lint

* lint: check-format

* doxygen fix

* fix type

* check-format

* Fix bulk insert for InsertPlanNode (#389)

* Change bulk insert for InsertPlanNode

* Fix return value error

* Fix comment

* Add TODO to flatten vector in the future

* I hate clang-tidy

* New tests for HashUtil

* Adding accessor methods and test cases for LogicalExportExternalFile

* basic test loigical get/ExternalFileGet/QueryDerivedGet; correct typo

* Ah, linting!

* LogicalGet: getters and non-trivial testcases

* LogicalExternalFileGet: getters and non-trivial testcases

* LogicalQueryDerivedGet: getters and non-trivial testcases

* Partial fix for using HashUtil::Hash with strings. We still need to fix HashUtil::CombineHashInRange

* Added test case for HashUtil::SumHashes

* Added new LogicalExportExternalFile test

* Change hashing of vector; logicalDependentJoin test&getters;

Change in some operator hash function:
from iterate the loop and combineHash each entry
to using HashInRange function

* logicalMarkJoin: getter& test

* LogicalSingleJoin: getter & test

* Logicalinnerjoin: getter & test

* Working on expanding LogicalUpdate

* LogicalLeftJoin: getter & testcase; comment out hash tests in operators

* Dirty hack for supporting HashUtil::Hash() on string literals

* Fixed up test case for LogicalUpdate. Switch its internal vector to use raw UpdateClause pointers. This may be a bad idea...

* DBMain and Settings Manager (#336)

* Added getters + tests for LogicalDistinct and LogicalLimit

* Added getters + tests for LogicalInsert and LogicalInsertSelect

* some linting

* annotated expr, logical right/left join compare by content

TODO: annotated expr testcase?

* logical left right join test

* logical outer join: getter and test

* logical semi join: getter & test

* logical projection and filter

* linting

* Added getters + tests for LogicalDelete and LogicalUpdate

* Ran format + lint. The terrier is watching me as I type this...

* Added a bunch NOLINT hints about copies of values in tests where we do want a copy

* Only check for TERRIER_ASSERT on debug builds in LogicalInsertTest

* Add ManagedPointer

* remove redundant includes.

* better hashing vector of annotatedExpression

* Fix typo in network README (#354)

* prevent undefined behavior when hashing with null involved

mainly when annotated_expr->GetExpr()->Hash() is called.

* rename to managed_ptr.h

* Add ManagedPointer (#394)

* Add ManagedPointer

* remove redundant includes.

* rename to managed_ptr.h

* Traffic cop with sqlite (#328)

* use ManagedPointer: queryDerivedGet

* a bit of linting

* use ManagedPointer: LogicalQueryDerivedGet

* a bit of linting

* use ManagedPointer: logicalProjection

* use ManagedPointer in logical ops and annotatedExpression

also add boolean conversion in the managedPointer

* linting

* New test cases for ManagedPointer. We mostly check to make sure that it has the same semantics as std::shared_ptr. I am writing this outside of Gates while the terrier runs around looking for rabbits... (#396)

* linting

* logical limit/update w imperfect fix in updateClause

* logicalget test

* logical filter

* add const

* logical Op cleaning up

* raw to managed in logicalInsert

* lint

* Revert "logical Op cleaning up"

This reverts commit 087e3c9.

* Blowing away things that we don't care about

* Copying in correct versions from master

* Copying in correct versions from master

* shared to managed ptr; remove get info

* Removing catalog files that should have been removed

* delete operator_expr cpp

 which only contains get_info implementation

* fix logicalinsert

* lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.