diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000000..87022cc0f0 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,111 @@ +# Non-Blocking Alter Table Support (SqlTable) + +## Overview +The overall goal of this project is to implement lazy, non-blocking schema changes. The current storage layer does not support any schema changes. The first goal of this project is to support schema change operations including add column, drop column, and default value changes. However, the overall goal would be to carry this out in a non-blocking fashion, for which we decided to go with a lazy evaluation approach. This means that any schema change would not migrate existing tuples to the new schema until information that only exists in the new schema is modified. + +## Scope +Almost all of the work will be localized to the SqlTable object as that is the access point from the execution layer to the storage layer. Also our design will not affect the underlying structures of the tuples or the DataTables and after this change they should still be able to provide their functionality without having to go through the SqlTable. Within the SqlTable we will be changing the following modules. Below we will refer to expected version as the version of the tuple the user expects to see and actual version as the version the tuple is currently in. These two versions will differ when a user has updated schema, but due to the lazy evaluation the tuple hasn’t been transformed into the latest version. + +### SqlTable::Insert +- Insert will now take in a schema version number that indicates the version of the tuple being inserted. +- In this case the actual and expected versions will be the same as an insert will always put a tuple in the version that is passed in. + +### SqlTable::Update +- Update will be required to take in a schema version number that indicates the expected schema version number of the tuple being updated +- In this case the actual version number will differ from the expected version in cases where the tuple being updated has not been shifted to the expected version. +- If the update modifies columns that are not in the actual version then the tuple will be shifted to the expected version before applying the update. + +### SqlTable::Scan +- Scan will be required to take in a schema version number that indicates the schema version number that the current transaction sees. + +### SqlTable::Delete +- Delete will be required to take in the schema version number that indicates the expected schema version number of the tuple being deleted + +### Catalog +- The catalog will keep track of the visible schema versions for each transactions based on its timestamp. This schema version is then passed on to the SqlTable layer. + +### DataTable/Projected Row +- DataTable::SelectIntoBuffer iterates across a projected_row/column’s column ids and fills in the data for each column + + - For this operation we use the column_id VERSION_POINTER_COLUMN_ID as a sentinel id to represent a column that the DataTable should skip over and not fill in. We will go into detail as to why this happens within the architectural design below. + + +## Architectural Design +The design of this project will center around the modifications to SqlTable. The design for other components in the storage layer will remain unchanged. On a schema change the SqlTable will create a new DataTable that will store all the tuples inserted from that point on into the new version. To be lazy it will not modify already existing tuples to transform them into the latest version. In order to support this lazy schema change we need two functionalities: maintaining tuples in multiple different schema versions and providing methods of transforming them into the desired version. + +### Multi-versioning +To address the multi-versioning the SqlTable will maintain a map from the schema_version_number to a DataTable. There will be one DataTable for each schema version. The functionality for accessing tuples is already present in DataTable and this way SqlTable will only need to manage the two functionalities we described above. The rest of the functionalities will be handled by the already existing DataTable implementation. Furthermore, each block will maintain metadata of which version all of the tuples within the block belong to. Since each DataTable can only be a single version, a block cannot contains tuples from multiple versions. Below is a description of the multiversion design for each of the SqlTable operations we are modifying and we will refer to expected version as the version of the tuple the user expects to see and actual version as the version the tuple is currently in. + +#### SqlTable::Insert +Insert will always insert the passed in tuple into the DataTable of the schema version number passed in + +#### SqlTable::Update +- Update has three cases + 1. The expected schema version matches the actual schema version + - The update will happen in place on the DataTable of the actual schema version + 2. The expected schema version doesn’t match the actual schema version but the update doesn’t touch any columns not in the actual schema version + - The update will happen in place on the DataTable of the actual schema version + 3. The expected schema version doesn’t match the actual schema version and the update touches not in the actual schema version. The following steps occur + - Retrieve the tuple from the actual version DataTable + - Transform the tuple to the expected version + - Delete the tuple from the actual version DataTable + - Insert the tuple into the expected version DataTable + +#### SqlTable::Scan +- SqlTable will maintain its own slot iterator which will be used to iterate across all the schema version. The iterator interface exposed to the user will not change +- The iterator for Scan must always begin on the latest version + - In cases where the user is interchanging scan call and update calls, if the scan iterator were to start on an older version then it is possible for a tuple retrieved from the scan to be updated, which could move it into the latest schema version + - This would mean that when the iterator gets to the latest schema version it will read all the tuples in the DataTable for that version and this would cause that tuple to have been read twice within the scan. + +#### SqlTable::Select +- There are two cases + 1. Expected schema version matches actual schema version + - The tuple is directly selected from the DataTable for that schema version + 2. Expected schema version differs from the actual schema version + - The tuple is selected from the DataTable for that schema version and during this process it is transformed to the expected schema version + - This transformed tuple is returned + +#### SqlTable::Delete +- Delete will directly delete the tuple from the DataTable of the actual version of that tuple + +#### SqlTable::UpdateSchema +- This function is the access point through which users can update the schema by passing in a new schema object +- The SqlTable will construct a new DataTable to maintain all of the tuples inserted for this version. +- To be lazy, none of the already existing tuples will be modified in this call. + +### Transformation +In the current interface the user can only retrieve data from a SqlTable through a ProjectedRow or a ProjectedColumn, we will refer to both as Projection in this section for simplicity. The user passes in a Projection which is filled by the storage layer. The Projection passed in by the user will be in the expected version but the actual version of the data could be different so we need to provide a way of transforming between versions. To do this we modify the header of the Projection. + +The header contains metadata regarding column_ids and column_offsets which is used by the DataTable to populate the Projection. Furthermore the column_ids can be different between schema versions. Before passing the Projection to the DataTable of the actual version we translate the column_ids that are in the expected version to column_ids of the actual version. Then for any column that is not present in the actual version we set the column_id to a sentinel value (VERSION_POINTER_COLUMN_ID, as no column in the Projection should have that id). We pass this modified Projection to the DataTable which populates it, skipping over any columns with the id set to the sentinel value. Then the we reset the Projection header to the original header and fill in any default values for column that were not present in the actual version. This way we avoid having to copy data from one version to another and it is filled in only once. + +## Design Rationale +In order to support lazy schema changes we will need to maintain tuples in multiple different schema versions and provide methods of transforming them into the desired version. + +### Backend for different schema versions +The initial decision for storing schema versions within an SQL table was whether to back it by a vector or map. While we appreciated the simplicity and probable performance benefits of a vector, we ultimately decided to go with a map because it did not force our versioning to start at 0. While this constraint does not seem significant at first, we realized that if the database is restored from a checkpoint we should restore the schema version number (since it may be exposed to the user or tracked by a hot-backup) rather than reinitialize the versions to 0. + +The second decision for our backend was whether we should protect the underlying data structure with latches or use a latch-free structure. We ultimately decided to use a latch-free data structure because we decided it would be difficult to reason about every possible point the latch would be needed (essentially any version check) without wrapping the map in another abstraction level. Additionally, we had serious concerns about introducing 4 to 10 latch operations on the path of every single SQL action in every single transaction and that would guarantee large numbers of cache invalidations. We therefore decided to go with the ConcurrentMap implementation in the database (wrapper of tbb::concurrent_hash_map) which supports the common case of concurrent insertions and lookups. However, this creates future difficulties for supporting compaction/deletion of obsolete schema versions because erasures are unsafe. Unfortunately, we are not aware of any candidate hash map implementation that supports safe insertions, deletions, and lookups without utilizing latches. + +### Transforming old versions into the expected version +We recognized two possible ways to transform the data stored under old schemas into the expected schema for a transaction: (1) attribute-wise copying of the data from an old ProjectedRow to a new one and (2) rewriting the header on the new ProjectedRow (provided by the caller) to be readable by an older data table. We initially implemented (1) because it was far simpler logic. However, when we benchmarked the implementation for cross-version selects we observed a significant performance penalty (factor of 10x). We therefore have switched to (2) and have reduced the penalty to a factor of 5x. We are still working on improving this even further. + +## Testing Plan +Our current testing plan is to implement two new test suites that test both the sequential correctness and concurrent robustness of the implementation. The sequential test suite focuses on ensuring that known edge cases are handled correctly. We focus on a sequential test for these situations because we can more tightly control the ordering of actions. We are also implementing a concurrent test suite which will ensure that performs a mini-stress test that tries to verify that rare, but possible, race conditions are likely to be detected. For this we focus on straining access to the versioning scheme by ensuring we are doing concurrent inserts and reads on the hash map. + +In addition to formal tests, we are also benchmarking our implementation as we go to ensure we measure and understand the performance impacts of our changes. Specifically, we are focusing on performance impact across a range of simulated workloads (selects, inserts, updates, and a mix) and in two general situations: a single schema version and multiple versions. The goal here is to ensure we can quantify our impact against the current single-versioned implementation as well as quantify the performance degradation for on-the-fly schema manipulation of old data that has not migrated to the new schema version. + +## Trade-offs and Potential Problems +**Trade-off:** TBB Concurrent Unordered Map for storing DataTableVersion objects. This decision gives a simple and easy to integrate solution for supporting concurrent insertions (new schemas) and lookups (all DML transactions) on the data tables. However, this will limit options when we start to implement compaction of obsolete schema versions because the data structure does not support concurrent erasures. This likely means we will have to take a heavy-weight approach for compaction such as duplicating the structure without the data to be erased and then use an unlink-delete staged method similar to how the GC already works on undo records. + +**Trade-off:** Our decision to manually mangle the headers for ProjectedRow and ProjectedColumn greatly increases the code complexity (manual recreation of the headers) in order to significantly improve performance for reading data across schema versions. Specifically, we avoid an unnecessary allocation and deallocation for temporary projections by allowing old schema versions to write directly into the final projection. + +## Future Work +### Pending tasks +#### Default values +- Populate the default values into the ProjectRow during Select and Scan operations. +- Handle changes to the default values. Should they be considered as a schema change or Catalog maintains the default values that can be queried by the SqlTable? + +### Stretch goals +- Rolling back schema change transactions. +- Implementing a Compactor to remove DataTables of older versions that don’t contain any tuples. +- Serializing transactions with unsafe/conflicting schema changes by using a central commit latch, allowing only one transaction to commit at a time. Rollback if the validation checks fail. diff --git a/benchmark/storage/sql_table_benchmark.cpp b/benchmark/storage/sql_table_benchmark.cpp new file mode 100644 index 0000000000..4837e26462 --- /dev/null +++ b/benchmark/storage/sql_table_benchmark.cpp @@ -0,0 +1,676 @@ +#include +#include + +#include "benchmark/benchmark.h" +#include "common/strong_typedef.h" +#include "loggers/main_logger.h" +#include "storage/data_table.h" +#include "storage/sql_table.h" +#include "storage/storage_defs.h" +#include "storage/storage_util.h" +#include "transaction/transaction_context.h" +#include "transaction/transaction_manager.h" +#include "util/catalog_test_util.h" +#include "util/multithread_test_util.h" +#include "util/storage_test_util.h" +#include "util/transaction_test_util.h" +namespace terrier { + +// This benchmark simulates a key-value store inserting a large number of tuples. This provides a good baseline and +// reference to other fast data structures (indexes) to compare against. We are interested in the SqlTable's raw +// performance, so the tuple's contents are intentionally left garbage and we don't verify correctness. That's the job +// of the Google Tests. + +class SqlTableBenchmark : public benchmark::Fixture { + public: + void SetUp(const benchmark::State &state) final { + common::WorkerPool thread_pool(num_threads_, {}); + + // create schema + catalog::col_oid_t col_oid(0); + for (uint32_t i = 0; i < column_num_; i++) { + columns_.emplace_back("", type::TypeId::BIGINT, false, col_oid++); + } + schema_ = new catalog::Schema(columns_, storage::layout_version_t(0)); + table_ = new storage::SqlTable(&block_store_, *schema_, catalog::table_oid_t(0)); + + // + std::vector all_col_oids; + for (auto &col : schema_->GetColumns()) all_col_oids.emplace_back(col.GetOid()); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(0)); + + initializer_ = new storage::ProjectedRowInitializer(std::get<0>(pair)); + map_ = new storage::ProjectionMap(std::get<1>(pair)); + // generate a random redo ProjectedRow to Insert + redo_buffer_ = common::AllocationUtil::AllocateAligned(initializer_->ProjectedRowSize()); + redo_ = initializer_->InitializeRow(redo_buffer_); + CatalogTestUtil::PopulateRandomRow(redo_, *schema_, pair.second, &generator_); + + // generate a ProjectedRow buffer to Read + read_buffer_ = common::AllocationUtil::AllocateAligned(initializer_->ProjectedRowSize()); + read_ = initializer_->InitializeRow(read_buffer_); + + // generate a vector of ProjectedRow buffers for concurrent reads + for (uint32_t i = 0; i < num_threads_; ++i) { + // Create read buffer + byte *read_buffer = common::AllocationUtil::AllocateAligned(initializer_->ProjectedRowSize()); + storage::ProjectedRow *read = initializer_->InitializeRow(read_buffer); + read_buffers_.emplace_back(read_buffer); + reads_.emplace_back(read); + } + } + + void TearDown(const benchmark::State &state) final { + delete[] redo_buffer_; + delete[] read_buffer_; + delete schema_; + delete initializer_; + delete map_; + delete table_; + for (uint32_t i = 0; i < num_threads_; ++i) delete[] read_buffers_[i]; + columns_.clear(); + read_buffers_.clear(); + reads_.clear(); + } + + // Sql Table + storage::SqlTable *table_ = nullptr; + + // Tuple properties + const storage::ProjectedRowInitializer *initializer_ = nullptr; + const storage::ProjectionMap *map_; + + // Workload + const uint32_t num_inserts_ = 10000000; + const uint32_t num_reads_ = 10000000; + const uint32_t num_updates_ = 10000000; + const uint32_t num_threads_ = 4; + const uint64_t buffer_pool_reuse_limit_ = 10000000; + const uint32_t scan_buffer_size_ = 1000; // maximum number of tuples in a buffer + + // Test infrastructure + std::default_random_engine generator_; + storage::BlockStore block_store_{1000, 1000}; + storage::RecordBufferSegmentPool buffer_pool_{num_inserts_, buffer_pool_reuse_limit_}; + transaction::TransactionManager txn_manager_ = {&buffer_pool_, true, LOGGING_DISABLED}; + + // Schema + const uint32_t column_num_ = 2; + std::vector columns_; + catalog::Schema *schema_ = nullptr; + + // Insert buffer pointers + byte *redo_buffer_; + storage::ProjectedRow *redo_; + + // Read buffer pointers; + byte *read_buffer_; + storage::ProjectedRow *read_; + + // Read buffers pointers for concurrent reads + std::vector read_buffers_; + std::vector reads_; +}; + +// Insert the num_inserts_ of tuples into a SqlTable in a single thread +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, SimpleInsert)(benchmark::State &state) { + // NOLINTNEXTLINE + for (auto _ : state) { + // Create a sql_table + storage::SqlTable table(&block_store_, *schema_, catalog::table_oid_t(0)); + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + for (uint32_t i = 0; i < num_inserts_; ++i) { + table.Insert(&txn, *redo_, storage::layout_version_t(0)); + } + } + + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Insert the num_inserts_ of tuples into a SqlTable concurrently +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, ConcurrentInsert)(benchmark::State &state) { + // NOLINTNEXTLINE + for (auto _ : state) { + // Create a sql_table + storage::SqlTable table(&block_store_, *schema_, catalog::table_oid_t(0)); + + auto workload = [&](uint32_t id) { + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + for (uint32_t i = 0; i < num_inserts_ / num_threads_; ++i) { + table.Insert(&txn, *redo_, storage::layout_version_t(0)); + } + }; + common::WorkerPool thread_pool(num_threads_, {}); + MultiThreadTestUtil::RunThreadsUntilFinish(&thread_pool, num_threads_, workload); + } + + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a sequential order from a SqlTable in a single thread +// The SqlTable has only one schema version +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, SingleVersionSequentialRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *redo_, storage::layout_version_t(0))); + } + + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Select(&txn, read_order[i], read_, *map_, storage::layout_version_t(0)); + } + } + + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a sequential order from a SqlTable in a single thread +// The SqlTable has multiple schema versions and the read version mismatches the one stored in the storage layer +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionMismatchSequentialRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *redo_, storage::layout_version_t(0))); + } + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a new read buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *read_pr = pair.first.InitializeRow(buffer); + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Select(&txn, read_order[i], read_pr, pair.second, storage::layout_version_t(1)); + } + } + delete[] buffer; + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a sequential order from a SqlTable in a single thread +// The SqlTable has multiple schema versions and the read version matches the one stored in the storage layer +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionMatchSequentialRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a new insert buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *insert_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *insert_pr = pair.first.InitializeRow(insert_buffer); + CatalogTestUtil::PopulateRandomRow(insert_pr, new_schema, pair.second, &generator_); + + // insert tuples + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *insert_pr, storage::layout_version_t(1))); + } + delete[] insert_buffer; + + // create a new read buffer + byte *read_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *read_pr = pair.first.InitializeRow(read_buffer); + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Select(&txn, read_order[i], read_pr, pair.second, storage::layout_version_t(1)); + } + } + delete[] read_buffer; + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a random order from a SqlTable in a single thread +// The SqlTable has only one schema version +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, SingleVersionRandomRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *redo_, storage::layout_version_t(0))); + } + // Create random reads + std::shuffle(read_order.begin(), read_order.end(), generator_); + + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Select(&txn, read_order[i], read_, *map_, storage::layout_version_t(0)); + } + } + + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a random order from a SqlTable in a single thread +// The SqlTable has multiple schema versions and the read version mismatches the one stored in the storage layer +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionMismatchRandomRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *redo_, storage::layout_version_t(0))); + } + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a new read buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *read_pr = pair.first.InitializeRow(buffer); + + // Create random reads + std::shuffle(read_order.begin(), read_order.end(), generator_); + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Select(&txn, read_order[i], read_pr, pair.second, storage::layout_version_t(1)); + } + } + delete[] buffer; + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a random order from a SqlTable in a single thread +// The SqlTable has multiple schema versions and the read version matches the one stored in the storage layer +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionMatchRandomRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a new insert buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *insert_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *insert_pr = pair.first.InitializeRow(insert_buffer); + CatalogTestUtil::PopulateRandomRow(insert_pr, new_schema, pair.second, &generator_); + + // insert tuples + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *insert_pr, storage::layout_version_t(1))); + } + delete[] insert_buffer; + + // create a new read buffer + byte *read_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *read_pr = pair.first.InitializeRow(read_buffer); + + // Create random reads + std::shuffle(read_order.begin(), read_order.end(), generator_); + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Select(&txn, read_order[i], read_pr, pair.second, storage::layout_version_t(1)); + } + } + delete[] read_buffer; + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a random order from a SqlTable concurrently +// The SqlTable has only a single versions +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, ConcurrentSingleVersionRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *redo_, storage::layout_version_t(0))); + } + // Generate random read orders and read buffer for each thread + std::shuffle(read_order.begin(), read_order.end(), generator_); + std::uniform_int_distribution rand_start(0, static_cast(read_order.size() - 1)); + std::vector rand_read_offsets; + for (uint32_t i = 0; i < num_threads_; ++i) { + // Create random reads + rand_read_offsets.emplace_back(rand_start(generator_)); + } + // NOLINTNEXTLINE + for (auto _ : state) { + auto workload = [&](uint32_t id) { + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + for (uint32_t i = 0; i < num_inserts_ / num_threads_; ++i) { + table_->Select(&txn, read_order[(rand_read_offsets[id] + i) % read_order.size()], reads_[id], *map_, + storage::layout_version_t(0)); + } + }; + common::WorkerPool thread_pool(num_threads_, {}); + MultiThreadTestUtil::RunThreadsUntilFinish(&thread_pool, num_threads_, workload); + } + + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Read the num_reads_ of tuples in a random order from a SqlTable concurrently +// The SqlTable has multiple schema versions and the read version mismatches the one stored in the storage layer +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, ConcurrentMultiVersionRead)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + std::vector read_order; + for (uint32_t i = 0; i < num_reads_; ++i) { + read_order.emplace_back(table_->Insert(&txn, *redo_, storage::layout_version_t(0))); + } + + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // update the vector of ProjectedRow buffers for concurrent reads + for (uint32_t i = 0; i < num_threads_; ++i) delete[] read_buffers_[i]; + read_buffers_.clear(); + reads_.clear(); + for (uint32_t i = 0; i < num_threads_; ++i) { + // Create read buffer + byte *read_buffer = common::AllocationUtil::AllocateAligned(initializer_->ProjectedRowSize()); + storage::ProjectedRow *read = initializer_->InitializeRow(read_buffer); + read_buffers_.emplace_back(read_buffer); + reads_.emplace_back(read); + } + + // Generate random read orders and read buffer for each thread + std::shuffle(read_order.begin(), read_order.end(), generator_); + std::uniform_int_distribution rand_start(0, static_cast(read_order.size() - 1)); + std::vector rand_read_offsets; + for (uint32_t i = 0; i < num_threads_; ++i) { + // Create random reads + rand_read_offsets.emplace_back(rand_start(generator_)); + } + // NOLINTNEXTLINE + for (auto _ : state) { + auto workload = [&](uint32_t id) { + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + for (uint32_t i = 0; i < num_inserts_ / num_threads_; ++i) { + table_->Select(&txn, read_order[(rand_read_offsets[id] + i) % read_order.size()], reads_[id], *map_, + storage::layout_version_t(1)); + } + }; + common::WorkerPool thread_pool(num_threads_, {}); + MultiThreadTestUtil::RunThreadsUntilFinish(&thread_pool, num_threads_, workload); + } + + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Update a tuple in a single-version SqlTable num_updates_ times in a single thread +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, SingleVersionUpdate)(benchmark::State &state) { + // Insert a tuple to the table + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + storage::TupleSlot slot = table_->Insert(&txn, *redo_, storage::layout_version_t(0)); + // Populate with random values for updates + CatalogTestUtil::PopulateRandomRow(redo_, *schema_, *map_, &generator_); + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_updates_; ++i) { + // update the tuple with the for benchmark purpose + table_->Update(&txn, slot, redo_, *map_, storage::layout_version_t(0)); + } + } + + state.SetItemsProcessed(state.iterations() * num_updates_); +} + +// Update a tuple in a SqlTable num_updates_ times in a single thread +// The SqlTable has multiple schema versions and the redo can be updated in place +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionMatchUpdate)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a new insert buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *insert_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *insert_pr = pair.first.InitializeRow(insert_buffer); + CatalogTestUtil::PopulateRandomRow(insert_pr, new_schema, pair.second, &generator_); + + // insert a tuple + storage::TupleSlot slot = table_->Insert(&txn, *insert_pr, storage::layout_version_t(1)); + + delete[] insert_buffer; + + // create a new update buffer + byte *update_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *update_pr = pair.first.InitializeRow(update_buffer); + CatalogTestUtil::PopulateRandomRow(update_pr, new_schema, pair.second, &generator_); + + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_updates_; ++i) { + table_->Update(&txn, slot, update_pr, pair.second, storage::layout_version_t(1)); + } + } + delete[] update_buffer; + state.SetItemsProcessed(state.iterations() * num_updates_); +} + +// Update a tuple in a SqlTable num_updates_ times in a single thread +// The SqlTable has multiple schema versions and the redo cannot be updated in place +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionMismatchUpdate)(benchmark::State &state) { + auto txn = txn_manager_.BeginTransaction(); + // insert a bunch of tuples + std::vector update_slots; + for (uint32_t i = 0; i < num_updates_; ++i) { + update_slots.emplace_back(table_->Insert(txn, *redo_, storage::layout_version_t(0))); + } + + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a update buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *update_buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedRowSize()); + storage::ProjectedRow *update_pr = pair.first.InitializeRow(update_buffer); + CatalogTestUtil::PopulateRandomRow(update_pr, new_schema, pair.second, &generator_); + + // NOLINTNEXTLINE + for (auto _ : state) { + for (uint32_t i = 0; i < num_updates_; ++i) { + table_->Update(txn, update_slots[i], update_pr, pair.second, storage::layout_version_t(1)); + } + } + delete[] update_buffer; + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; + state.SetItemsProcessed(state.iterations() * num_updates_); +} + +// Scan the num_insert_ of tuples from a SqlTable in a single thread +// The SqlTable has only one schema version +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, SingleVersionScan)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + for (uint32_t i = 0; i < num_inserts_; ++i) { + table_->Insert(&txn, *redo_, storage::layout_version_t(0)); + } + + // create a scan buffer + std::vector all_col_oids; + for (auto &col : schema_->GetColumns()) all_col_oids.emplace_back(col.GetOid()); + auto pair = table_->InitializerForProjectedColumns(all_col_oids, scan_buffer_size_, storage::layout_version_t(0)); + byte *buffer = common::AllocationUtil::AllocateAligned(pair.first.ProjectedColumnsSize()); + storage::ProjectedColumns *scan_pr = pair.first.Initialize(buffer); + + // NOLINTNEXTLINE + for (auto _ : state) { + auto start_pos = table_->begin(storage::layout_version_t(0)); + while (start_pos != table_->end()) { + table_->Scan(&txn, &start_pos, scan_pr, pair.second, storage::layout_version_t(0)); + scan_pr = pair.first.Initialize(buffer); + } + } + delete[] buffer; + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +// Scan the num_insert_ of tuples from a SqlTable in a single thread +// The SqlTable has two schema versions +// NOLINTNEXTLINE +BENCHMARK_DEFINE_F(SqlTableBenchmark, MultiVersionScan)(benchmark::State &state) { + // Populate read_table_ by inserting tuples + // We can use dummy timestamps here since we're not invoking concurrency control + transaction::TransactionContext txn(transaction::timestamp_t(0), transaction::timestamp_t(0), &buffer_pool_, + LOGGING_DISABLED, ACTION_FRAMEWORK_DISABLED); + + // insert tuples into old schema + for (uint32_t i = 0; i < num_inserts_ / 2; ++i) { + table_->Insert(&txn, *redo_, storage::layout_version_t(0)); + } + + // create new schema + catalog::col_oid_t col_oid(column_num_); + std::vector new_columns(columns_.begin(), columns_.end() - 1); + new_columns.emplace_back("", type::TypeId::BIGINT, false, col_oid); + catalog::Schema new_schema(new_columns, storage::layout_version_t(1)); + table_->UpdateSchema(new_schema); + + // create a new insert buffer + std::vector all_col_oids(new_columns.size()); + for (size_t i = 0; i < new_columns.size(); i++) all_col_oids[i] = new_columns[i].GetOid(); + auto row_pair = table_->InitializerForProjectedRow(all_col_oids, storage::layout_version_t(1)); + byte *insert_buffer = common::AllocationUtil::AllocateAligned(row_pair.first.ProjectedRowSize()); + storage::ProjectedRow *insert_pr = row_pair.first.InitializeRow(insert_buffer); + CatalogTestUtil::PopulateRandomRow(insert_pr, new_schema, row_pair.second, &generator_); + + // insert tuples into new schema + for (uint32_t i = 0; i < num_inserts_ / 2; ++i) { + table_->Insert(&txn, *insert_pr, storage::layout_version_t(1)); + } + + // create a new read buffer + auto col_pair = table_->InitializerForProjectedColumns(all_col_oids, scan_buffer_size_, storage::layout_version_t(1)); + byte *buffer = common::AllocationUtil::AllocateAligned(col_pair.first.ProjectedColumnsSize()); + storage::ProjectedColumns *scan_pr = col_pair.first.Initialize(buffer); + + // NOLINTNEXTLINE + for (auto _ : state) { + auto start_pos = table_->begin(storage::layout_version_t(1)); + while (start_pos != table_->end()) { + table_->Scan(&txn, &start_pos, scan_pr, col_pair.second, storage::layout_version_t(1)); + scan_pr = col_pair.first.Initialize(buffer); + } + } + delete[] buffer; + state.SetItemsProcessed(state.iterations() * num_inserts_); +} + +BENCHMARK_REGISTER_F(SqlTableBenchmark, SimpleInsert)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, ConcurrentInsert)->Unit(benchmark::kMillisecond)->UseRealTime(); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, SingleVersionSequentialRead)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionMismatchSequentialRead)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionMatchSequentialRead)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, SingleVersionRandomRead)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionMismatchRandomRead)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionMatchRandomRead)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, ConcurrentSingleVersionRead)->Unit(benchmark::kMillisecond)->UseRealTime(); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, ConcurrentMultiVersionRead)->Unit(benchmark::kMillisecond)->UseRealTime(); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, SingleVersionUpdate)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionMatchUpdate)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionMismatchUpdate)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, SingleVersionScan)->Unit(benchmark::kMillisecond); + +BENCHMARK_REGISTER_F(SqlTableBenchmark, MultiVersionScan)->Unit(benchmark::kMillisecond); + +} // namespace terrier diff --git a/script/micro_bench/run_micro_bench.py b/script/micro_bench/run_micro_bench.py index f98a323f9b..e988a24bd6 100755 --- a/script/micro_bench/run_micro_bench.py +++ b/script/micro_bench/run_micro_bench.py @@ -30,7 +30,8 @@ class TestConfig(object): """ def __init__(self): # benchmark executables to run - self.benchmark_list = ["data_table_benchmark", + self.benchmark_list = ["sql_table_benchmark", + "data_table_benchmark", "garbage_collector_benchmark", "large_transaction_benchmark", "logging_benchmark", @@ -710,7 +711,8 @@ class RunMicroBenchmarks(object): """ def __init__(self, verbose=False, debug=False): # list of benchmarks to run - self.benchmark_list = ["data_table_benchmark", + self.benchmark_list = ["sql_table_benchmark", + "data_table_benchmark", "garbage_collector_benchmark", "large_transaction_benchmark", "logging_benchmark", diff --git a/src/include/catalog/schema.h b/src/include/catalog/schema.h index cabd5b0140..6e8bb6772b 100644 --- a/src/include/catalog/schema.h +++ b/src/include/catalog/schema.h @@ -26,6 +26,7 @@ class Schema { * the reliance on these classes */ class Column { + // TODO(Sai): DON'T use default parameters for function arguments. Everyone should explicitly pass in nullptr. public: /** * Instantiates a Column object, primary to be used for building a Schema object (non VARLEN attributes) @@ -33,8 +34,10 @@ class Schema { * @param type SQL type for this column * @param nullable true if the column is nullable, false otherwise * @param oid internal unique identifier for this column + * @param default_value default value for this column. Null by default */ - Column(std::string name, const type::TypeId type, const bool nullable, const col_oid_t oid) + Column(std::string name, const type::TypeId type, const bool nullable, const col_oid_t oid, + byte *default_value = nullptr) : name_(std::move(name)), type_(type), attr_size_(type::TypeUtil::GetTypeSize(type_)), @@ -43,6 +46,10 @@ class Schema { TERRIER_ASSERT(attr_size_ == 1 || attr_size_ == 2 || attr_size_ == 4 || attr_size_ == 8, "This constructor is meant for non-VARLEN columns."); TERRIER_ASSERT(type_ != type::TypeId::INVALID, "Attribute type cannot be INVALID."); + + // ASSUMPTION: The default_value passed in is of size attr_size_ + // Copy the passed in default value (if exists) + SetDefault(default_value); } /** @@ -52,17 +59,20 @@ class Schema { * @param max_varlen_size the maximum length of the varlen entry * @param nullable true if the column is nullable, false otherwise * @param oid internal unique identifier for this column + * @param default_value default value for this column. Null by default */ Column(std::string name, const type::TypeId type, const uint16_t max_varlen_size, const bool nullable, - const col_oid_t oid) + const col_oid_t oid, byte *default_value = nullptr) : name_(std::move(name)), type_(type), attr_size_(type::TypeUtil::GetTypeSize(type_)), max_varlen_size_(max_varlen_size), nullable_(nullable), oid_(oid) { + // TODO(Sai): How to handle default values for VARLEN? TERRIER_ASSERT(attr_size_ == VARLEN_COLUMN, "This constructor is meant for VARLEN columns."); TERRIER_ASSERT(type_ != type::TypeId::INVALID, "Attribute type cannot be INVALID."); + SetDefault(default_value); } /** @@ -94,6 +104,20 @@ class Schema { * @return internal unique identifier for this column */ col_oid_t GetOid() const { return oid_; } + /** + * @return default value for this column + */ + const byte *GetDefault() const { return (default_is_null_) ? nullptr : reinterpret_cast(&default_); } + + /** + * Set the default value of the column + * @param default_value default_value as a bytes array. Could be nullptr + */ + void SetDefault(byte *default_value) { + default_is_null_ = (default_value == nullptr); + // If explicitly setting the default value to null + if (!default_is_null_) std::memcpy(default_, default_value, attr_size_); + } /** * Default constructor for deserialization @@ -134,15 +158,19 @@ class Schema { uint16_t max_varlen_size_; bool nullable_; col_oid_t oid_; - // TODO(Matt): default value would go here - // Value default_; + // TODO(Sai): Consider having a DefaultValueObject containing isNull, 16-byte variable and attribute size + // This avoids handling memory explicitly for default values + byte default_[16]; + bool default_is_null_; }; /** * Instantiates a Schema object from a vector of previously-defined Columns * @param columns description of this SQL table's schema as a collection of Columns + * @param version the schema version number */ - explicit Schema(std::vector columns) : columns_(std::move(columns)) { + explicit Schema(std::vector columns, storage::layout_version_t version = storage::layout_version_t(0)) + : version_(version), columns_(std::move(columns)) { TERRIER_ASSERT(!columns_.empty() && columns_.size() <= common::Constants::MAX_COL, "Number of columns must be between 1 and MAX_COL."); for (uint32_t i = 0; i < columns_.size(); i++) { @@ -176,6 +204,10 @@ class Schema { * @return description of this SQL table's schema as a collection of Columns */ const std::vector &GetColumns() const { return columns_; } + /** + * @return version number for this schema + */ + const storage::layout_version_t GetVersion() const { return version_; } /** * @return serialized schema @@ -205,6 +237,7 @@ class Schema { } private: + storage::layout_version_t version_; const std::vector columns_; std::unordered_map col_oid_to_offset; }; diff --git a/src/include/storage/data_table.h b/src/include/storage/data_table.h index 404698ca6d..56c5065fe2 100644 --- a/src/include/storage/data_table.h +++ b/src/include/storage/data_table.h @@ -213,9 +213,9 @@ class DataTable { // The TransactionManager needs to modify VersionPtrs when rolling back aborts friend class transaction::TransactionManager; + const TupleAccessStrategy accessor_; BlockStore *const block_store_; const layout_version_t layout_version_; - const TupleAccessStrategy accessor_; // TODO(Tianyu): For now, on insertion, we simply sequentially go through a block and allocate a // new one when the current one is full. Needless to say, we will need to revisit this when extending GC to handle diff --git a/src/include/storage/projected_row.h b/src/include/storage/projected_row.h index 5853ce4b65..24319891a2 100644 --- a/src/include/storage/projected_row.h +++ b/src/include/storage/projected_row.h @@ -172,6 +172,13 @@ class ProjectedRowInitializer { */ uint32_t ProjectedRowSize() const { return size_; } + /** + * Populates the ProjectedRow's members but with only the header, doesn't clear space for data + * @param head pointer to the byte buffer to populate for header + * @return pointer to the populated ProjectedRow header + */ + ProjectedRow *InitializeHeader(void *head) const; + /** * @return number of columns in the projection list */ diff --git a/src/include/storage/sql_table.h b/src/include/storage/sql_table.h index 6195ed0efb..42c7e8b20d 100644 --- a/src/include/storage/sql_table.h +++ b/src/include/storage/sql_table.h @@ -1,9 +1,13 @@ #pragma once #include #include +#include #include #include #include "catalog/schema.h" +#include "common/container/concurrent_map.h" +#include "common/macros.h" +#include "loggers/storage_logger.h" #include "storage/data_table.h" #include "storage/projected_columns.h" #include "storage/projected_row.h" @@ -27,6 +31,110 @@ class SqlTable { DataTable *data_table; BlockLayout layout; ColumnMap column_map; + InverseColumnMap inverse_column_map; + }; + + /** + * Iterator for all the slots, claimed or otherwise, in the data table. This is useful for sequential scans. + */ + class SlotIterator { + // TODO(Yashwanth): Slot iterator currently flawed, for a scan on a certain version, it MUST begin on the latest + // version it sees + public: + /** + * @return reference to the underlying tuple slot + */ + const TupleSlot &operator*() const { return *current_it_; } + + /** + * @return pointer to the underlying tuple slot + */ + const TupleSlot *operator->() const { return &(*current_it_); } + + /** + * pre-fix increment. + * @return self-reference after the iterator is advanced + */ + SlotIterator &operator++() { + TERRIER_ASSERT(!is_end_, "Cannot increment an end iterator"); + current_it_++; + AdvanceOnEndOfDatatable_(); + return *this; + } + + /** + * post-fix increment. + * @return copy of the iterator equal to this before increment + */ + const SlotIterator operator++(int) { + SlotIterator copy = *this; + operator++(); + return copy; + } + + /** + * Equality check. Either both iterators are "end" or they must match + * on both their observed version and the underlying tuple. + * @param other other iterator to compare to + * @return if the two iterators point to the same slotcolumn_ids + */ + bool operator==(const SlotIterator &other) const { + // First "is_end" check is that both are end, the second protects the iterator check through short-circuit + return (is_end_ && other.is_end_) || + (is_end_ == other.is_end_ && txn_version_ == other.txn_version_ && current_it_ == other.current_it_); + } + + /** + * Inequality check. + * @param other other iterator to compare to + * @return if the two iterators are not equal + */ + bool operator!=(const SlotIterator &other) const { return !this->operator==(other); } + + DataTable::SlotIterator *GetDataTableSlotIterator() { return ¤t_it_; } + + private: + friend class SqlTable; + + SlotIterator(const common::ConcurrentMap *tables, + const layout_version_t txn_version, bool is_end) + : tables_(tables), current_it_(tables_->Find(txn_version)->second.data_table->begin()) { + txn_version_ = txn_version; + curr_version_ = txn_version; + is_end_ = is_end; + } + + /** + * Checks if DataTable::SlotIterator is at the end of the table and advances + * the iterator to the next table or sets is_end flag as appropriate. This + * refactor is necessary to keep iterator in correct state when used in two + * distinct ways: increment called directly on this object or implicitly + * through advancement of the DataTable iterator in SqlTable::Scan. + */ + void AdvanceOnEndOfDatatable_() { + TERRIER_ASSERT(curr_version_ <= txn_version_, "Current version cannot be newer than transaction"); + while (current_it_ == tables_->Find(curr_version_)->second.data_table->end()) { + // layout_version_t is uint32_t so we need to protect against underflow. + if ((!curr_version_) == 0) { + is_end_ = true; + return; + } + curr_version_--; + auto next_table = tables_->Find(curr_version_); + if (next_table == tables_->CEnd()) { // next_table does not exist (at end) + is_end_ = true; + break; + } + current_it_ = next_table->second.data_table->begin(); + } + TERRIER_ASSERT(current_it_->GetBlock() != nullptr && !is_end_, "Invalid iterator"); + } + + const common::ConcurrentMap *tables_; + layout_version_t txn_version_; + layout_version_t curr_version_; + DataTable::SlotIterator current_it_; + bool is_end_; }; public: @@ -43,7 +151,16 @@ class SqlTable { /** * Destructs a SqlTable, frees all its members. */ - ~SqlTable() { delete table_.data_table; } + ~SqlTable(); + + /** + * Adds the new schema to set of active data tables. This functions should + * only be called upon commit because there is no mechanism to rollback an + * abort (which shouldn't be needed). + * + * @param schema the new Schema for the SqlTable (version must be unique) + */ + void UpdateSchema(const catalog::Schema &schema); /** * Materializes a single tuple from the given slot, as visible at the timestamp of the calling txn. @@ -51,11 +168,12 @@ class SqlTable { * @param txn the calling transaction * @param slot the tuple slot to read * @param out_buffer output buffer. The object should already contain projection list information. @see ProjectedRow. + * @param pr_map the ProjectionMap of the out_buffer + * @param version_num the schema version which the transaction sees * @return true if tuple is visible to this txn and ProjectedRow has been populated, false otherwise */ - bool Select(transaction::TransactionContext *const txn, const TupleSlot slot, ProjectedRow *const out_buffer) const { - return table_.data_table->Select(txn, slot, out_buffer); - } + bool Select(transaction::TransactionContext *txn, TupleSlot slot, ProjectedRow *out_buffer, + const ProjectionMap &pr_map, layout_version_t version_num) const; /** * Update the tuple according to the redo buffer given. @@ -63,32 +181,43 @@ class SqlTable { * @param txn the calling transaction * @param slot the slot of the tuple to update. * @param redo the desired change to be applied. This should be the after-image of the attributes of interest. - * @return true if successful, false otherwise + * @param map the ProjectionMap of the ProjectedRow + * @param version_num the schema version which the transaction sees + * @return true if successful, false otherwise; If the update changed the location of the TupleSlot, a new TupleSlot + * is returned. Otherwise, the same TupleSlot is returned. */ - bool Update(transaction::TransactionContext *const txn, const TupleSlot slot, const ProjectedRow &redo) const { - return table_.data_table->Update(txn, slot, redo); - } + std::pair Update(transaction::TransactionContext *txn, TupleSlot slot, ProjectedRow *redo, + const ProjectionMap &map, layout_version_t version_num); /** * Inserts a tuple, as given in the redo, and return the slot allocated for the tuple. * * @param txn the calling transaction * @param redo after-image of the inserted tuple. + * @param version_num the schema version which the transaction sees * @return the TupleSlot allocated for this insert, used to identify this tuple's physical location for indexes and * such. */ - TupleSlot Insert(transaction::TransactionContext *const txn, const ProjectedRow &redo) const { - return table_.data_table->Insert(txn, redo); + TupleSlot Insert(transaction::TransactionContext *const txn, const ProjectedRow &redo, + layout_version_t version_num) const { + // TODO(Matt): check constraints? Discuss if that happens in execution layer or not + // always insert into the new DataTable + TERRIER_ASSERT(tables_.Find(version_num) != tables_.CEnd(), "Table version must exist before insert"); + return tables_.Find(version_num)->second.data_table->Insert(txn, redo); } /** * Deletes the given TupleSlot, this will call StageWrite on the provided txn to generate the RedoRecord for delete. * @param txn the calling transaction * @param slot the slot of the tuple to delete + * @param version_num the schema version which the transaction sees * @return true if successful, false otherwise */ - bool Delete(transaction::TransactionContext *const txn, const TupleSlot slot) { - return table_.data_table->Delete(txn, slot); + bool Delete(transaction::TransactionContext *const txn, const TupleSlot slot, layout_version_t version_num) const { + // TODO(Matt): check constraints? Discuss if that happens in execution layer or not + layout_version_t old_version = slot.GetBlock()->layout_version_; + // always delete the tuple in the old block + return tables_.Find(old_version)->second.data_table->Delete(txn, slot); } /** @@ -102,45 +231,59 @@ class SqlTable { * @param start_pos iterator to the starting location for the sequential scan * @param out_buffer output buffer. The object should already contain projection list information. This buffer is * always cleared of old values. + * @param pr_map the ProjectionMap for the Projected Columns + * @param version_num the schema version which the transaction sees */ - void Scan(transaction::TransactionContext *const txn, DataTable::SlotIterator *const start_pos, - ProjectedColumns *const out_buffer) const { - return table_.data_table->Scan(txn, start_pos, out_buffer); - } + void Scan(transaction::TransactionContext *txn, SqlTable::SlotIterator *start_pos, ProjectedColumns *out_buffer, + const ProjectionMap &pr_map, layout_version_t version_num) const; /** - * @return table's unique identifier + * @return the first tuple slot contained in the data table */ - catalog::table_oid_t Oid() const { return oid_; } + SlotIterator begin(layout_version_t txn_version) const { + // common::SpinLatch::ScopedSpinLatch guard(&tables_latch_); + auto ret = SlotIterator(&tables_, txn_version, false); + ret.AdvanceOnEndOfDatatable_(); + return ret; + } /** - * @return the first tuple slot contained in the underlying DataTable + * Returns one past the last tuple slot contained in the last data table. Note that this is not an accurate number + * when concurrent accesses are happening, as inserts maybe in flight. However, the number given is always + * transactionally correct, as any inserts that might have happened is not going to be visible to the calling + * transaction. + * + * @return one past the last tuple slot contained in the data table. */ - DataTable::SlotIterator begin() const { return table_.data_table->begin(); } + SlotIterator end() const { + // common::SpinLatch::ScopedSpinLatch guard(&tables_latch_); + return SlotIterator(&tables_, tables_.CBegin()->first, true); + } /** - * @return one past the last tuple slot contained in the underlying DataTable + * @return table's unique identifier */ - DataTable::SlotIterator end() const { return table_.data_table->end(); } + catalog::table_oid_t Oid() const { return oid_; } /** * Generates an ProjectedColumnsInitializer for the execution layer to use. This performs the translation from col_oid * to col_id for the Initializer's constructor so that the execution layer doesn't need to know anything about col_id. * @param col_oids set of col_oids to be projected * @param max_tuples the maximum number of tuples to store in the ProjectedColumn + * @param version_num the schema version * @return pair of: initializer to create ProjectedColumns, and a mapping between col_oid and the offset within the * ProjectedColumn * @warning col_oids must be a set (no repeats) */ std::pair InitializerForProjectedColumns( - const std::vector &col_oids, const uint32_t max_tuples) const { + const std::vector &col_oids, const uint32_t max_tuples, layout_version_t version_num) const { TERRIER_ASSERT((std::set(col_oids.cbegin(), col_oids.cend())).size() == col_oids.size(), "There should not be any duplicated in the col_ids!"); - auto col_ids = ColIdsForOids(col_oids); + auto col_ids = ColIdsForOids(col_oids, version_num); TERRIER_ASSERT(col_ids.size() == col_oids.size(), "Projection should be the same number of columns as requested col_oids."); - ProjectedColumnsInitializer initializer(table_.layout, col_ids, max_tuples); - auto projection_map = ProjectionMapForInitializer(initializer); + ProjectedColumnsInitializer initializer(tables_.Find(version_num)->second.layout, col_ids, max_tuples); + auto projection_map = ProjectionMapForInitializer(initializer, version_num); TERRIER_ASSERT(projection_map.size() == col_oids.size(), "ProjectionMap be the same number of columns as requested col_oids."); return {initializer, projection_map}; @@ -150,21 +293,23 @@ class SqlTable { * Generates an ProjectedRowInitializer for the execution layer to use. This performs the translation from col_oid to * col_id for the Initializer's constructor so that the execution layer doesn't need to know anything about col_id. * @param col_oids set of col_oids to be projected + * @param version_num the schema version * @return pair of: initializer to create ProjectedRow, and a mapping between col_oid and the offset within the * ProjectedRow to create ProjectedColumns, and a mapping between col_oid and the offset within the * ProjectedColumn * @warning col_oids must be a set (no repeats) */ std::pair InitializerForProjectedRow( - const std::vector &col_oids) const { + const std::vector &col_oids, layout_version_t version_num) const { TERRIER_ASSERT((std::set(col_oids.cbegin(), col_oids.cend())).size() == col_oids.size(), "There should not be any duplicated in the col_ids!"); - auto col_ids = ColIdsForOids(col_oids); + auto col_ids = ColIdsForOids(col_oids, version_num); TERRIER_ASSERT(col_ids.size() == col_oids.size(), "Projection should be the same number of columns as requested col_oids."); + TERRIER_ASSERT(tables_.Find(version_num) != tables_.CEnd(), "Table version must exist before insert"); ProjectedRowInitializer initializer = - ProjectedRowInitializer::CreateProjectedRowInitializer(table_.layout, col_ids); - auto projection_map = ProjectionMapForInitializer(initializer); + ProjectedRowInitializer::CreateProjectedRowInitializer(tables_.Find(version_num)->second.layout, col_ids); + auto projection_map = ProjectionMapForInitializer(initializer, version_num); TERRIER_ASSERT(projection_map.size() == col_oids.size(), "ProjectionMap be the same number of columns as requested col_oids."); return {initializer, projection_map}; @@ -174,15 +319,19 @@ class SqlTable { BlockStore *const block_store_; const catalog::table_oid_t oid_; - // Eventually we'll support adding more tables when schema changes. For now we'll always access the one DataTable. - DataTableVersion table_; + common::ConcurrentMap tables_; + // NOTE: This map only keeps track of the default values specified at column creation + // For columns which don't have default value or added later, just set to null + // Populating default values into the ProjectedRow inserted later is taken care of by the execution engine + DefaultValueMap default_value_map_; /** * Given a set of col_oids, return a vector of corresponding col_ids to use for ProjectionInitialization * @param col_oids set of col_oids, they must be in the table's ColumnMap + * @param version the version of DataTable * @return vector of col_ids for these col_oids */ - std::vector ColIdsForOids(const std::vector &col_oids) const; + std::vector ColIdsForOids(const std::vector &col_oids, layout_version_t version) const; /** * Given a ProjectionInitializer, returns a map between col_oid and the offset within the projection to access that @@ -192,6 +341,39 @@ class SqlTable { * @return the projection map for this initializer */ template - ProjectionMap ProjectionMapForInitializer(const ProjectionInitializerType &initializer) const; + ProjectionMap ProjectionMapForInitializer(const ProjectionInitializerType &initializer, + layout_version_t version) const; + + /** + * Given a projected row/col translates the column id of each column to the column id of the version passed in + * If a column doesn't exist in that version sets the column id to VERSION_POINTER_COLUMN_ID + * @param out_buffer - projected row/col whose header to modify + * @param curr_dt_version - version number of schema of the passed in projected row/col + * @param old_dt_version - version number of schema that is desired + * @param original_col_id_store - array to store the original column id's on. Should have space to fill all column_ids + */ + template + void ModifyProjectionHeaderForVersion(RowType *out_buffer, const DataTableVersion &curr_dt_version, + const DataTableVersion &old_dt_version, col_id_t *original_col_id_store) const; + + /** + * Calculate the columns of the ProjectionMap that are missing from the old version of datatable + * Used to find out the columns for which default values must be filled in + * @param pr_map ProjectionMap of the ProjectedRow passed into Select/Scan + * @param old_dt_version old version of the datatable + * @return Unordered set of missing column oids + */ + std::unordered_set GetMissingColumnOidsForVersion(const ProjectionMap &pr_map, + const DataTableVersion &old_dt_version) const; + + /** + * Fill in the default value for the given column i.e. col_oid. The default value being filled in can be null + * @tparam RowType ProjectedRow or ProjectedColumns::RowView + * @param out_buffer ProjectedRow or ProjectedColumns::RowView + * @param col_oid OID of the column + * @param pr_map ProjectionMap of the RowType + */ + template + void FillDefaultValue(RowType *out_buffer, catalog::col_oid_t col_oid, const ProjectionMap &pr_map) const; }; } // namespace terrier::storage diff --git a/src/include/storage/storage_defs.h b/src/include/storage/storage_defs.h index c47a746560..401d9ce874 100644 --- a/src/include/storage/storage_defs.h +++ b/src/include/storage/storage_defs.h @@ -10,6 +10,7 @@ #include "catalog/catalog_defs.h" #include "common/constants.h" #include "common/container/bitmap.h" +#include "common/container/concurrent_map.h" #include "common/hash_util.h" #include "common/macros.h" #include "common/object_pool.h" @@ -176,6 +177,11 @@ using ColumnMap = std::unordered_map; * Used by execution and storage layers to map between col_oids and offsets within a ProjectedRow */ using ProjectionMap = std::unordered_map; +using InverseColumnMap = std::unordered_map; +/** + * Used by SqlTable to map between col_oids in Schema and their {default_value, attribute_size} + */ +using DefaultValueMap = common::ConcurrentMap>; /** * Denote whether a record modifies the logical delete column, used when DataTable inspects deltas diff --git a/src/include/storage/storage_util.h b/src/include/storage/storage_util.h index 2d90d11d3a..dac5983f4f 100644 --- a/src/include/storage/storage_util.h +++ b/src/include/storage/storage_util.h @@ -66,6 +66,25 @@ class StorageUtil { static void CopyAttrFromProjection(const TupleAccessStrategy &accessor, TupleSlot to, const RowType &from, uint16_t projection_list_offset); + /** + * Copy from a projection to another projection from a different data-table block, which has different block layout. + * + * Note that this function understand the notion of col_oid, so it knows Sql concepts. + * + * 1) It only copies the columns that exist in both ProjectedRows + * 2) For columns that only exist in the destination row, it is set to NULL. (This should be changed to default values + * in the future). + * @tparam RowType1 ProjectedRow or ProjectedColumns::RowView + * @tparam RowType2 ProjectedRow or ProjectedColumns::RowView + * @param from the source row + * @param from_map the source ProjectionMap + * @param layout the block layout of the source row + * @param to the destination row + * @param to_map the destination ProjectionMap + */ + template + static void CopyProjectionIntoProjection(const RowType1 &from, const ProjectionMap &from_map, + const BlockLayout &layout, RowType2 *to, const ProjectionMap &to_map); /** * Applies delta into the given buffer. * @@ -137,5 +156,12 @@ class StorageUtil { */ static std::vector ComputeBaseAttributeOffsets(const std::vector &attr_sizes, uint16_t num_reserved_columns); + + /** + * Given a layout, it generations a projection list that includes all the columns. + * @param layout the given layout + * @return a vector of all column ids + */ + static std::vector ProjectionListAllColumns(const storage::BlockLayout &layout); }; } // namespace terrier::storage diff --git a/src/storage/data_table.cpp b/src/storage/data_table.cpp index 4897724b45..74f865358c 100644 --- a/src/storage/data_table.cpp +++ b/src/storage/data_table.cpp @@ -8,7 +8,7 @@ namespace terrier::storage { DataTable::DataTable(BlockStore *const store, const BlockLayout &layout, const layout_version_t layout_version) - : block_store_(store), layout_version_(layout_version), accessor_(layout) { + : accessor_(layout), block_store_(store), layout_version_(layout_version) { TERRIER_ASSERT(layout.AttrSize(VERSION_POINTER_COLUMN_ID) == 8, "First column must have size 8 for the version chain."); TERRIER_ASSERT(layout.NumColumns() > NUM_RESERVED_COLUMNS, @@ -43,6 +43,7 @@ void DataTable::Scan(transaction::TransactionContext *const txn, SlotIterator *c out_buffer->TupleSlots()[filled] = slot; filled++; } + ++(*start_pos); } out_buffer->SetNumTuples(filled); @@ -187,9 +188,6 @@ bool DataTable::Delete(transaction::TransactionContext *const txn, const TupleSl template bool DataTable::SelectIntoBuffer(transaction::TransactionContext *const txn, const TupleSlot slot, RowType *const out_buffer) const { - TERRIER_ASSERT(out_buffer->NumColumns() <= accessor_.GetBlockLayout().NumColumns() - NUM_RESERVED_COLUMNS, - "The output buffer never returns the version pointer columns, so it should have " - "fewer attributes."); TERRIER_ASSERT(out_buffer->NumColumns() > 0, "The output buffer should return at least one attribute."); // This cannot be visible if it's already deallocated. if (!accessor_.Allocated(slot)) return false; @@ -202,8 +200,9 @@ bool DataTable::SelectIntoBuffer(transaction::TransactionContext *const txn, con // because so long as we set the version ptr before updating in place, the reader will know if a conflict // can potentially happen, and chase the version chain before returning anyway, for (uint16_t i = 0; i < out_buffer->NumColumns(); i++) { - TERRIER_ASSERT(out_buffer->ColumnIds()[i] != VERSION_POINTER_COLUMN_ID, - "Output buffer should not read the version pointer column."); + if (out_buffer->ColumnIds()[i] == VERSION_POINTER_COLUMN_ID) { + continue; + } StorageUtil::CopyAttrIntoProjection(accessor_, slot, out_buffer, i); } diff --git a/src/storage/projected_row.cpp b/src/storage/projected_row.cpp index 3e1ff05285..08314f42c4 100644 --- a/src/storage/projected_row.cpp +++ b/src/storage/projected_row.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -49,6 +50,15 @@ ProjectedRowInitializer::ProjectedRowInitializer(const std::vector &at } ProjectedRow *ProjectedRowInitializer::InitializeRow(void *const head) const { + TERRIER_ASSERT(reinterpret_cast(head) % sizeof(uint64_t) == 0, + "start of ProjectedRow needs to be aligned to 8 bytes to" + "ensure correctness of alignment of its members"); + ProjectedRow *result = InitializeHeader(head); + result->Bitmap().Clear(result->num_cols_); + return result; +} + +ProjectedRow *ProjectedRowInitializer::InitializeHeader(void *head) const { TERRIER_ASSERT(reinterpret_cast(head) % sizeof(uint64_t) == 0, "start of ProjectedRow needs to be aligned to 8 bytes to" "ensure correctness of alignment of its members"); @@ -57,7 +67,7 @@ ProjectedRow *ProjectedRowInitializer::InitializeRow(void *const head) const { result->num_cols_ = static_cast(col_ids_.size()); for (uint32_t i = 0; i < col_ids_.size(); i++) result->ColumnIds()[i] = col_ids_[i]; for (uint32_t i = 0; i < col_ids_.size(); i++) result->AttrValueOffsets()[i] = offsets_[i]; - result->Bitmap().Clear(result->num_cols_); + return result; } diff --git a/src/storage/sql_table.cpp b/src/storage/sql_table.cpp index 2e8af1e1b0..f9cdc4e04c 100644 --- a/src/storage/sql_table.cpp +++ b/src/storage/sql_table.cpp @@ -1,6 +1,9 @@ #include "storage/sql_table.h" #include +#include +#include #include + #include "common/macros.h" #include "storage/storage_util.h" @@ -8,6 +11,27 @@ namespace terrier::storage { SqlTable::SqlTable(BlockStore *const store, const catalog::Schema &schema, const catalog::table_oid_t oid) : block_store_(store), oid_(oid) { + UpdateSchema(schema); +} + +SqlTable::~SqlTable() { + while (default_value_map_.CBegin() != default_value_map_.CEnd()) { + auto pair = *(default_value_map_.CBegin()); + delete[] pair.second.first; + default_value_map_.UnsafeErase(pair.first); + } + + while (tables_.CBegin() != tables_.CEnd()) { + auto pair = *(tables_.CBegin()); + delete (pair.second.data_table); // Delete the data_table object on the heap + tables_.UnsafeErase(pair.first); + } +} + +void SqlTable::UpdateSchema(const catalog::Schema &schema) { + STORAGE_LOG_DEBUG("Update schema version: {}", uint32_t(schema.GetVersion())); + TERRIER_ASSERT(tables_.Find(schema.GetVersion()) == tables_.End(), "schema versions for an SQL table must be unique"); + // Begin with the NUM_RESERVED_COLUMNS in the attr_sizes std::vector attr_sizes; attr_sizes.reserve(NUM_RESERVED_COLUMNS + schema.GetColumns().size()); @@ -19,48 +43,311 @@ SqlTable::SqlTable(BlockStore *const store, const catalog::Schema &schema, const TERRIER_ASSERT(attr_sizes.size() == NUM_RESERVED_COLUMNS, "attr_sizes should be initialized with NUM_RESERVED_COLUMNS elements."); + // First pass through to accumulate the counts of each attr_size for (const auto &column : schema.GetColumns()) { attr_sizes.push_back(column.GetAttrSize()); } auto offsets = storage::StorageUtil::ComputeBaseAttributeOffsets(attr_sizes, NUM_RESERVED_COLUMNS); - ColumnMap col_oid_to_id; - // Build the map from Schema columns to underlying columns + ColumnMap col_map; + InverseColumnMap inv_col_map; + + // Build the maps between Schema column OIDs and underlying column IDs for (const auto &column : schema.GetColumns()) { switch (column.GetAttrSize()) { case VARLEN_COLUMN: - col_oid_to_id[column.GetOid()] = col_id_t(offsets[0]++); + inv_col_map[col_id_t(offsets[0])] = column.GetOid(); + col_map[column.GetOid()] = col_id_t(offsets[0]++); break; case 8: - col_oid_to_id[column.GetOid()] = col_id_t(offsets[1]++); + inv_col_map[col_id_t(offsets[1])] = column.GetOid(); + col_map[column.GetOid()] = col_id_t(offsets[1]++); break; case 4: - col_oid_to_id[column.GetOid()] = col_id_t(offsets[2]++); + inv_col_map[col_id_t(offsets[2])] = column.GetOid(); + col_map[column.GetOid()] = col_id_t(offsets[2]++); break; case 2: - col_oid_to_id[column.GetOid()] = col_id_t(offsets[3]++); + inv_col_map[col_id_t(offsets[3])] = column.GetOid(); + col_map[column.GetOid()] = col_id_t(offsets[3]++); break; case 1: - col_oid_to_id[column.GetOid()] = col_id_t(offsets[4]++); + inv_col_map[col_id_t(offsets[4])] = column.GetOid(); + col_map[column.GetOid()] = col_id_t(offsets[4]++); break; default: throw std::runtime_error("unexpected switch case value"); } } - auto layout = storage::BlockLayout(attr_sizes); - table_ = {new DataTable(block_store_, layout, layout_version_t(0)), layout, col_oid_to_id}; + // Populate the default value map + // + // clang's memory analysis gets confused by the iterator here. This is tied + // to the same false positive identified inside of the loop and is caused. + // NOLINTNEXTLINE + for (const auto &column : schema.GetColumns()) { + auto col_oid = column.GetOid(); + auto *default_value = column.GetDefault(); + // Only populate the default values of the columns which are new and have a default value + if (default_value_map_.Find(col_oid) == default_value_map_.End()) { + uint8_t attr_size = column.GetAttrSize(); + byte *temp = nullptr; + if (default_value != nullptr) { + // clang's memory analysis has a false positive on this allocation. All memory + // allocated here is placed into 'default_value_map_' and eventually freed + // with the rest of the SqlTable object in the destructor (see line 20). + // NOLINTNEXTLINE + temp = new byte[attr_size]; + std::memcpy(temp, default_value, attr_size); + } + default_value_map_.Insert(column.GetOid(), {temp, attr_size}); + } + } + + auto layout = BlockLayout(attr_sizes); + + auto dt = new DataTable(block_store_, layout, schema.GetVersion()); + // clang's memory analysis has a false positive on this allocation. The TERRIER_ASSERT on the second line of this + // function prevents the insert below from failing (can only fail when key is not unique). The write-lock on the + // catalog prevents any other transaction from being in a race condition with this one. The corresponding delete + // for this allocation is in the destructor for SqlTable. clang-analyzer-cplusplus.NewDeleteLeaks identifies this + // as a potential leak and throws an error incorrectly. + // NOLINTNEXTLINE + tables_.Insert(schema.GetVersion(), {dt, layout, col_map, inv_col_map}); +} + +bool SqlTable::Select(transaction::TransactionContext *const txn, const TupleSlot slot, ProjectedRow *const out_buffer, + const ProjectionMap &pr_map, layout_version_t version_num) const { + TERRIER_ASSERT(slot.GetBlock() != nullptr, "Slot does not exist"); + STORAGE_LOG_DEBUG("slot version: {}, current version: {}", !slot.GetBlock()->layout_version_, !version_num); + + layout_version_t old_version_num = slot.GetBlock()->layout_version_; + + TERRIER_ASSERT(out_buffer->NumColumns() <= tables_.Find(version_num)->second.column_map.size(), + "The output buffer never returns the version pointer columns, so it should have " + "fewer attributes."); + + // The version of the current slot is the same as the version num + auto &curr_dt_version = tables_.Find(version_num)->second; + + if (old_version_num == version_num) { + return curr_dt_version.data_table->Select(txn, slot, out_buffer); + } + + auto &old_dt_version = tables_.Find(old_version_num)->second; + + // The slot version is not the same as the version_num + col_id_t original_column_ids[out_buffer->NumColumns()]; + ModifyProjectionHeaderForVersion(out_buffer, curr_dt_version, old_dt_version, original_column_ids); + + // Get the result and copy back the old header + bool result = old_dt_version.data_table->Select(txn, slot, out_buffer); + std::memcpy(out_buffer->ColumnIds(), original_column_ids, sizeof(col_id_t) * out_buffer->NumColumns()); + + // Default values should only be filled into the columns that are missing in the old_dt_version + // Do not fill default values for the columns which already exist in the old_dt_version + + // Calculate the col_oids of out_buffer which are missing from the old version of datatable + std::unordered_set missing_col_oids = GetMissingColumnOidsForVersion(pr_map, old_dt_version); + + // Fill in the default values for the missing columns + for (catalog::col_oid_t col_oid : missing_col_oids) { + FillDefaultValue(out_buffer, col_oid, pr_map); + } + + return result; +} + +/** + * Update the tuple according to the redo buffer given. + * + * @param txn the calling transaction + * @param slot the slot of the tuple to update. + * @param redo the desired change to be applied. This should be the after-image of the attributes of interest. + * @param map the ProjectionMap of the ProjectedRow + * @param version_num the schema version which the transaction sees + * @return true if successful, false otherwise; If the update changed the location of the TupleSlot, a new TupleSlot + * is returned. Otherwise, the same TupleSlot is returned. + */ +std::pair SqlTable::Update(transaction::TransactionContext *const txn, const TupleSlot slot, + ProjectedRow *redo, const ProjectionMap &map, + layout_version_t version_num) { + // TODO(Matt): check constraints? Discuss if that happens in execution layer or not + // TODO(Matt): update indexes + STORAGE_LOG_DEBUG("Update slot version : {}, current version: {}", !slot.GetBlock()->layout_version_, !version_num); + + layout_version_t old_version = slot.GetBlock()->layout_version_; + + // The version of the current slot is the same as the version num + TERRIER_ASSERT(old_version <= version_num, "Transaction should not be seeing this tuple"); + if (old_version == version_num) { + return {tables_.Find(version_num)->second.data_table->Update(txn, slot, *redo), slot}; + } + + // The versions are different + // 1. Check if we can just update the old version + // 2. If Yes: + // 2.a) Convert ProjectedRow into old ProjectedRow + // 2.b) Update the old DataTable using the old ProjectedRow + // 3. Else: + // 3.a) Get the old row + // 3.b) Convert it into new row + // 3.c) Delete old row + // 3.d) Update the new row before insert + // 3.e) Insert new row into new table + + // Check if the Redo's attributes are a subset of old schema so that we can update old version in place + bool is_subset = true; + + std::vector redo_col_oids; // the set of col oids the redo touches + for (auto &it : map) { + redo_col_oids.emplace_back(it.first); + // check if the col_oid exists in the old schema + if (tables_.Find(old_version)->second.column_map.count(it.first) == 0) { + is_subset = false; + break; + } + } + + storage::TupleSlot ret_slot; + if (is_subset) { + // we can update in place + auto old_dt_version = tables_.Find(old_version)->second; + + // The slot version is not the same as the version_num + col_id_t original_column_ids[redo->NumColumns()]; + ModifyProjectionHeaderForVersion(redo, tables_.Find(version_num)->second, old_dt_version, original_column_ids); + + // Get the result and copy back the old header + bool result = old_dt_version.data_table->Update(txn, slot, *redo); + + // We should create a buffer of old Projected Row and update in place. We can't just + // directly erase the data without creating a redo and update the chain. + + // auto old_pair = InitializerForProjectedRow(redo_col_oids, old_version); + + // // 1. Create a ProjectedRow Buffer for the old version + // byte *buffer = common::AllocationUtil::AllocateAligned(old_pair.first.ProjectedRowSize()); + // storage::ProjectedRow *pr = old_pair.first.InitializeRow(buffer); + + // // 2. Copy from new ProjectedRow to old ProjectedRow + // StorageUtil::CopyProjectionIntoProjection(redo, map, tables_.Find(old_version)->second.layout, pr, + // old_pair.second); + + // // 3. Update the old data-table + // bool result = tables_.Find(old_version)->second.data_table->Update(txn, slot, *pr); + // delete[] buffer; + if (!result) { + return {false, slot}; + } + ret_slot = slot; + } else { + STORAGE_LOG_DEBUG("have to delete and insert ... "); + + // need to create a new ProjectedRow of all columns + // 1. Get the old row + // 2. Convert it into new row + // 3. Delete old row + // 4. Update the new row before insert + // 5. Insert new row into new table + + // 1. Get old row + // 2. Convert it into new row + std::vector new_col_oids; // the set of col oids which the new schema has + for (auto &it : tables_.Find(version_num)->second.column_map) new_col_oids.emplace_back(it.first); + auto new_pair = InitializerForProjectedRow(new_col_oids, version_num); + auto new_buffer = common::AllocationUtil::AllocateAligned(new_pair.first.ProjectedRowSize()); + ProjectedRow *new_pr = new_pair.first.InitializeRow(new_buffer); + bool valid = Select(txn, slot, new_pr, new_pair.second, version_num); + if (!valid) { + delete[] new_buffer; + return {false, slot}; + } + // 3. Delete the old row + bool succ = tables_.Find(old_version)->second.data_table->Delete(txn, slot); + + // 4. Update the new row before insert + StorageUtil::CopyProjectionIntoProjection(*redo, map, tables_.Find(version_num)->second.layout, new_pr, + new_pair.second); + + // 5. Insert the row into new table + storage::TupleSlot new_slot; + if (succ) { + new_slot = tables_.Find(version_num)->second.data_table->Insert(txn, *new_pr); + } else { + // someone else deleted the old row, write-write conflict + delete[] new_buffer; + return {false, slot}; + } + + delete[] new_buffer; + + ret_slot = new_slot; + } + return {true, ret_slot}; +} + +void SqlTable::Scan(transaction::TransactionContext *const txn, SqlTable::SlotIterator *start_pos, + ProjectedColumns *const out_buffer, const ProjectionMap &pr_map, + layout_version_t version_num) const { + layout_version_t old_version_num = start_pos->curr_version_; + + TERRIER_ASSERT(out_buffer->NumColumns() <= tables_.Find(version_num)->second.column_map.size(), + "The output buffer never returns the version pointer columns, so it should have " + "fewer attributes."); + + DataTable::SlotIterator *dt_slot = start_pos->GetDataTableSlotIterator(); + + // Check for version match + if (old_version_num == version_num) { + tables_.Find(version_num)->second.data_table->Scan(txn, dt_slot, out_buffer); + start_pos->AdvanceOnEndOfDatatable_(); + return; + } + + col_id_t original_column_ids[out_buffer->NumColumns()]; + ModifyProjectionHeaderForVersion(out_buffer, tables_.Find(version_num)->second, tables_.Find(old_version_num)->second, + original_column_ids); + + tables_.Find(old_version_num)->second.data_table->Scan(txn, dt_slot, out_buffer); + start_pos->AdvanceOnEndOfDatatable_(); + + uint32_t filled = out_buffer->NumTuples(); + std::memcpy(out_buffer->ColumnIds(), original_column_ids, sizeof(col_id_t) * out_buffer->NumColumns()); + out_buffer->SetNumTuples(filled); + + auto old_dt_version = tables_.Find(old_version_num)->second; + auto curr_dt_version = tables_.Find(version_num)->second; + + // Populate the default values to all the scanned tuples + if (filled > 0) { + // Calculate the col_oids of out_buffer which are missing from the old version of datatable + std::unordered_set missing_col_oids = GetMissingColumnOidsForVersion(pr_map, old_dt_version); + + // TODO(Sai): The default values can be populated directly into the ProjectedColumns, making it faster than the + // case of column being present. Need to handle the bitmask operations correctly for null default values. + // Fill in the default values for the missing columns + for (uint32_t row_idx = 0; row_idx < filled; row_idx++) { + ProjectedColumns::RowView row = out_buffer->InterpretAsRow(row_idx); + for (catalog::col_oid_t col_oid : missing_col_oids) { + FillDefaultValue(&row, col_oid, pr_map); + } + } + } } -std::vector SqlTable::ColIdsForOids(const std::vector &col_oids) const { +std::vector SqlTable::ColIdsForOids(const std::vector &col_oids, + layout_version_t version) const { TERRIER_ASSERT(!col_oids.empty(), "Should be used to access at least one column."); std::vector col_ids; // Build the input to the initializer constructor for (const catalog::col_oid_t col_oid : col_oids) { - TERRIER_ASSERT(table_.column_map.count(col_oid) > 0, "Provided col_oid does not exist in the table."); - const col_id_t col_id = table_.column_map.at(col_oid); + TERRIER_ASSERT(tables_.Find(version) != tables_.CEnd(), "Table version must exist before insert"); + TERRIER_ASSERT(tables_.Find(version)->second.column_map.count(col_oid) > 0, + "Provided col_oid does not exist in the table."); + const col_id_t col_id = tables_.Find(version)->second.column_map.at(col_oid); col_ids.push_back(col_id); } @@ -68,15 +355,18 @@ std::vector SqlTable::ColIdsForOids(const std::vector -ProjectionMap SqlTable::ProjectionMapForInitializer(const ProjectionInitializerType &initializer) const { +ProjectionMap SqlTable::ProjectionMapForInitializer(const ProjectionInitializerType &initializer, + layout_version_t version) const { ProjectionMap projection_map; // for every attribute in the initializer for (uint16_t i = 0; i < initializer.NumColumns(); i++) { // extract the underlying col_id it refers to const col_id_t col_id_at_offset = initializer.ColId(i); // find the key (col_oid) in the table's map corresponding to the value (col_id) + + TERRIER_ASSERT(tables_.Find(version) != tables_.CEnd(), "Table version must exist"); const auto oid_to_id = - std::find_if(table_.column_map.cbegin(), table_.column_map.cend(), + std::find_if(tables_.Find(version)->second.column_map.cbegin(), tables_.Find(version)->second.column_map.cend(), [&](const auto &oid_to_id) -> bool { return oid_to_id.second == col_id_at_offset; }); // insert the mapping from col_oid to projection offset projection_map[oid_to_id->first] = i; @@ -86,8 +376,75 @@ ProjectionMap SqlTable::ProjectionMapForInitializer(const ProjectionInitializerT } template ProjectionMap SqlTable::ProjectionMapForInitializer( - const ProjectedColumnsInitializer &initializer) const; + const ProjectedColumnsInitializer &initializer, layout_version_t version) const; template ProjectionMap SqlTable::ProjectionMapForInitializer( - const ProjectedRowInitializer &initializer) const; + const ProjectedRowInitializer &initializer, layout_version_t version) const; + +// TODO(Yashwanth): don't copy the entire header, no need for template only take in ColumnIds() and then just modify +// that when resetting header only have memc py ColumnIds() +template +void SqlTable::ModifyProjectionHeaderForVersion(RowType *out_buffer, const DataTableVersion &curr_dt_version, + const DataTableVersion &old_dt_version, + col_id_t *original_col_id_store) const { + // The slot version is not the same as the version_num + // 1. Copy the old header (excluding bitmap) + std::memcpy(original_col_id_store, out_buffer->ColumnIds(), sizeof(col_id_t) * out_buffer->NumColumns()); + + // 2. For each column present in the old version, change the column id to the col id of that version + // For each column not present in the old version, change the column id to the sentinel value + // VERSION_POINTER_COLUMN_ID + for (uint16_t i = 0; i < out_buffer->NumColumns(); i++) { + TERRIER_ASSERT(out_buffer->ColumnIds()[i] != VERSION_POINTER_COLUMN_ID, + "Output buffer should not read the version pointer column."); + catalog::col_oid_t col_oid = curr_dt_version.inverse_column_map.at(out_buffer->ColumnIds()[i]); + if (old_dt_version.column_map.count(col_oid) > 0) { + out_buffer->ColumnIds()[i] = old_dt_version.column_map.at(col_oid); + } else { + // TODO(Yashwanth): consider renaming VERSION_POINTER_COLUMN_ID, since we're using it for more than just that now + out_buffer->ColumnIds()[i] = VERSION_POINTER_COLUMN_ID; + } + } +} + +template void SqlTable::ModifyProjectionHeaderForVersion(ProjectedRow *out_buffer, + const DataTableVersion &curr_dt_version, + const DataTableVersion &old_dt_version, + col_id_t *original_col_id_store) const; +template void SqlTable::ModifyProjectionHeaderForVersion(ProjectedColumns *out_buffer, + const DataTableVersion &curr_dt_version, + const DataTableVersion &old_dt_version, + col_id_t *original_col_id_store) const; + +std::unordered_set SqlTable::GetMissingColumnOidsForVersion( + const ProjectionMap &pr_map, const DataTableVersion &old_dt_version) const { + // Calculate the col_oids of out_buffer which are missing from the old version of datatable + std::unordered_set missing_col_oids; + for (const auto &it : pr_map) { + missing_col_oids.emplace(it.first); + } + for (const auto &it : old_dt_version.column_map) { + // Remove the col_oid from the missing_col_oids + missing_col_oids.erase(it.first); + } + return missing_col_oids; +} + +template +void SqlTable::FillDefaultValue(RowType *out_buffer, const catalog::col_oid_t col_oid, + const ProjectionMap &pr_map) const { + TERRIER_ASSERT(default_value_map_.Find(col_oid) != default_value_map_.CEnd(), + "Every column in schema must exist in default_value_map"); + auto pair = default_value_map_.Find(col_oid)->second; + auto default_value = pair.first; + auto attr_size = pair.second; + // TODO(Sai): If this becomes a performance bottleneck, we can move this logic to ModifyProjectionHeaderForVersion + storage::StorageUtil::CopyWithNullCheck(default_value, out_buffer, attr_size, pr_map.at(col_oid)); +} + +template void SqlTable::FillDefaultValue(ProjectedRow *, const catalog::col_oid_t, + const ProjectionMap &) const; +template void SqlTable::FillDefaultValue(ProjectedColumns::RowView *, + const catalog::col_oid_t, + const ProjectionMap &) const; } // namespace terrier::storage diff --git a/src/storage/storage_util.cpp b/src/storage/storage_util.cpp index 9d41f3da39..61e7fdca7a 100644 --- a/src/storage/storage_util.cpp +++ b/src/storage/storage_util.cpp @@ -5,6 +5,7 @@ #include #include "catalog/schema.h" #include "storage/projected_columns.h" +#include "storage/sql_table.h" #include "storage/tuple_access_strategy.h" #include "storage/undo_record.h" namespace terrier::storage { @@ -58,6 +59,50 @@ template void StorageUtil::CopyAttrFromProjection(con const ProjectedColumns::RowView &, uint16_t); +template +void StorageUtil::CopyProjectionIntoProjection(const RowType1 &from, const ProjectionMap &from_map, + const BlockLayout &layout, RowType2 *const to, + const ProjectionMap &to_map) { + // copy values + for (auto &it : from_map) { + if (to_map.count(it.first) > 0) { + // get the data bytes + const byte *value = from.AccessWithNullCheck(from_map.at(it.first)); + + // get the offset where we copy into + uint16_t offset = to_map.at(it.first); + if (value == nullptr) { + to->SetNull(offset); + } else { + to->SetNotNull(offset); + // get the size of the attribute + uint8_t attr_size = layout.AttrSize(from.ColumnIds()[it.second]); + + byte *addr = to->AccessForceNotNull(offset); + // Copy things over + std::memcpy(addr, value, attr_size); + } + } + } + // Fill with default values + // TODO(yangjuns): fill will default values instead of setting it to be null + for (auto &it : to_map) { + if (from_map.count(it.first) == 0) { + to->SetNull(it.second); + } + } +} + +template void StorageUtil::CopyProjectionIntoProjection(const ProjectedRow &from, + const ProjectionMap &from_map, + const BlockLayout &from_tas, + ProjectedRow *const to, + const ProjectionMap &to_map); + +template void StorageUtil::CopyProjectionIntoProjection( + const ProjectedColumns::RowView &from, const ProjectionMap &from_map, const BlockLayout &from_tas, + ProjectedColumns::RowView *const to, const ProjectionMap &to_map); + template void StorageUtil::ApplyDelta(const BlockLayout &layout, const ProjectedRow &delta, RowType *const buffer) { // the projection list in delta and buffer have to be sorted in the same way for this to work, @@ -141,4 +186,13 @@ std::vector StorageUtil::ComputeBaseAttributeOffsets(const std::vector return offsets; } +std::vector StorageUtil::ProjectionListAllColumns(const storage::BlockLayout &layout) { + std::vector col_ids(layout.NumColumns() - NUM_RESERVED_COLUMNS); + // Add all of the column ids from the layout to the projection list + // 0 is version vector so we skip it + for (uint16_t col = NUM_RESERVED_COLUMNS; col < layout.NumColumns(); col++) { + col_ids[col - NUM_RESERVED_COLUMNS] = storage::col_id_t(col); + } + return col_ids; +} } // namespace terrier::storage diff --git a/test/include/util/catalog_test_util.h b/test/include/util/catalog_test_util.h index d6e9275ab6..7cb68c9b88 100644 --- a/test/include/util/catalog_test_util.h +++ b/test/include/util/catalog_test_util.h @@ -3,6 +3,7 @@ #include #include "catalog/schema.h" #include "common/strong_typedef.h" +#include "storage/sql_table.h" #include "type/type_id.h" #include "util/multithread_test_util.h" #include "util/random_test_util.h" @@ -33,5 +34,99 @@ struct CatalogTestUtil { } return catalog::Schema(columns); } + + // The same as RandomSchem but it won't generate Varchar Column + template + static catalog::Schema RandomSchemaNoVarchar(const uint16_t max_cols, Random *const generator, + storage::layout_version_t version = storage::layout_version_t(0)) { + TERRIER_ASSERT(max_cols > 0, "There should be at least 1 columm."); + catalog::col_oid_t col_oid(0); + const uint16_t num_attrs = std::uniform_int_distribution(1, max_cols)(*generator); + std::vector possible_attr_types{ + type::TypeId::BOOLEAN, type::TypeId::TINYINT, type::TypeId::SMALLINT, type::TypeId::INTEGER, + type::TypeId::BIGINT, type::TypeId::DECIMAL, type::TypeId::TIMESTAMP, type::TypeId::DATE}; + std::vector possible_attr_nullable{true, false}; + std::vector columns; + for (uint16_t i = 0; i < num_attrs; i++) { + type::TypeId attr_type = *RandomTestUtil::UniformRandomElement(&possible_attr_types, generator); + bool attr_nullable = *RandomTestUtil::UniformRandomElement(&possible_attr_nullable, generator); + columns.emplace_back("col_name", attr_type, attr_nullable, col_oid++); + } + return catalog::Schema(columns, version); + } + + // Return a random ProjectedRow for the given schema. It always include all columns. + // The caller needs to free the ProjectedRow later by using + // delete[] reinterpret_cast(pr) + template + static storage::ProjectedRow *RandomInsertRow(storage::SqlTable *sql_table, const catalog::Schema &schema, + storage::layout_version_t version, Random *const generator) { + // insertion rows need to have all columns + std::vector col_oids; + + for (auto &col : schema.GetColumns()) { + col_oids.emplace_back(col.GetOid()); + } + // populate the row with random bytes + auto pr_pair = sql_table->InitializerForProjectedRow(col_oids, version); + byte *buffer = common::AllocationUtil::AllocateAligned(pr_pair.first.ProjectedRowSize()); + storage::ProjectedRow *pr = pr_pair.first.InitializeRow(buffer); + PopulateRandomRow(pr, schema, pr_pair.second, generator); + + // if a columns is nullable, flip a coin to set it to null + std::vector include{true, false}; + for (auto &col : schema.GetColumns()) { + if (col.GetNullable()) { + // flip a coin + if (*RandomTestUtil::UniformRandomElement(&include, generator)) { + pr->SetNull(pr_pair.second.at(col.GetOid())); + } + } + } + return pr; + } + + // It populates a projected row with random bytes. The schema must not contain VarChar + // The projected row must contain all the columns + template + static void PopulateRandomRow(storage::ProjectedRow *row, const catalog::Schema &schema, + const storage::ProjectionMap &pr_map, Random *const generator) { + for (auto &col : schema.GetColumns()) { + auto col_id = pr_map.at(col.GetOid()); + row->SetNotNull(col_id); + byte *addr = row->AccessWithNullCheck(col_id); + uint8_t size = col.GetAttrSize(); + StorageTestUtil::FillWithRandomBytes(size, addr, generator); + } + } + + // Check if two rows have the same content. It assumes they have the same ProjectionMap. + template + static bool ProjectionListEqual(const catalog::Schema &schema, const RowType1 *const one, const RowType2 *const other, + const storage::ProjectionMap &map) { + if (one->NumColumns() != other->NumColumns()) return false; + for (auto &it : map) { + catalog::col_oid_t col_oid = it.first; + uint8_t attr_size = 0; + for (auto &col : schema.GetColumns()) { + TERRIER_ASSERT(col.GetAttrSize() < 128, "VARLEN columns not supported byt this framework"); + if (col.GetOid() == col_oid) { + attr_size = col.GetAttrSize(); + break; + } + } + + const byte *one_content = one->AccessWithNullCheck(it.second); + const byte *other_content = other->AccessWithNullCheck(it.second); + // Either both are null or neither is null. + if (one_content == nullptr || other_content == nullptr) { + if (one_content == other_content) continue; + return false; + } + // Otherwise, they should be bit-wise identical. + if (memcmp(one_content, other_content, attr_size) != 0) return false; + } + return true; + } }; } // namespace terrier diff --git a/test/storage/sql_table_concurrent_test.cpp b/test/storage/sql_table_concurrent_test.cpp new file mode 100644 index 0000000000..33838187b2 --- /dev/null +++ b/test/storage/sql_table_concurrent_test.cpp @@ -0,0 +1,470 @@ +#include +#include +#include +#include +#include +#include +#include +#include "storage/garbage_collector.h" +#include "storage/sql_table.h" +#include "transaction/transaction_manager.h" +#include "type/type_id.h" +#include "util/catalog_test_util.h" +#include "util/test_harness.h" +#include "util/transaction_test_util.h" +namespace terrier { +struct SqlTableConcurrentTests : public TerrierTest { + SqlTableConcurrentTests() : gc_(&txn_manager_) {} + + void SetUp() override { TerrierTest::SetUp(); } + + void TearDown() override { TerrierTest::TearDown(); } + + std::vector GenerateColumnsVector(storage::layout_version_t v) { + std::vector cols; + + cols.emplace_back("version", type::TypeId::INTEGER, false, catalog::col_oid_t(100)); + cols.emplace_back("sentinel", type::TypeId::INTEGER, false, catalog::col_oid_t(1000)); + if ((!v) < 1) cols.emplace_back("bigint", type::TypeId::BIGINT, false, catalog::col_oid_t(1001)); + if ((!v) < 2) cols.emplace_back("integer", type::TypeId::INTEGER, false, catalog::col_oid_t(1002)); + if ((!v) < 3) cols.emplace_back("smallint", type::TypeId::SMALLINT, false, catalog::col_oid_t(1003)); + if ((!v) < 4) cols.emplace_back("tinyint", type::TypeId::TINYINT, false, catalog::col_oid_t(1004)); + if ((!v) >= 5) cols.emplace_back("tinyint", type::TypeId::TINYINT, false, catalog::col_oid_t(1005)); + if ((!v) >= 6) cols.emplace_back("smallint", type::TypeId::SMALLINT, false, catalog::col_oid_t(1006)); + if ((!v) >= 7) cols.emplace_back("integer", type::TypeId::INTEGER, false, catalog::col_oid_t(1007)); + if ((!v) >= 8) cols.emplace_back("bigint", type::TypeId::BIGINT, false, catalog::col_oid_t(1008)); + + auto *col_oids = new std::vector; + for (const auto &c : cols) { + col_oids->emplace_back(c.GetOid()); + } + versioned_col_oids.emplace_back(col_oids); + + return cols; + } + + void PopulateProjectedRow(storage::layout_version_t v, int32_t base_val, storage::ProjectedRow *pr, + storage::ProjectionMap *pr_map) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(100)), pr_map->end()); + uint32_t *version = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(100)))); + *version = static_cast(v); + + EXPECT_NE(pr_map->find(catalog::col_oid_t(1000)), pr_map->end()); + int32_t *sentinel = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1000)))); + *sentinel = base_val; + + if ((!v) < 1) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1001)), pr_map->end()); + int64_t *bigint = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1001)))); + *bigint = base_val << 31; + } + + if ((!v) < 2) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1002)), pr_map->end()); + int32_t *integer = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1002)))); + *integer = base_val; + } + + if ((!v) < 3) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1003)), pr_map->end()); + int16_t *smallint = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1003)))); + *smallint = static_cast(base_val % 1 << 15); + } + + if ((!v) < 4) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1004)), pr_map->end()); + int8_t *tinyint = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1004)))); + *tinyint = static_cast(base_val % 1 << 7); + } + + if ((!v) >= 5) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1005)), pr_map->end()); + int8_t *tinyint = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1005)))); + *tinyint = static_cast(base_val % 1 << 7); + } + + if ((!v) >= 6) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1006)), pr_map->end()); + int16_t *smallint = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1006)))); + *smallint = static_cast(base_val % 1 << 15); + } + + if ((!v) >= 7) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1007)), pr_map->end()); + int32_t *integer = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1007)))); + *integer = base_val; + } + + if ((!v) >= 8) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1008)), pr_map->end()); + int64_t *bigint = reinterpret_cast(pr->AccessForceNotNull(pr_map->at(catalog::col_oid_t(1008)))); + *bigint = base_val << 31; + } + } + + template + void ValidateTuple(RowType *pr, storage::ProjectionMap *pr_map, storage::layout_version_t v, int base_val) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(100)), pr_map->end()); + uint32_t *version = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(100)))); + EXPECT_NE(version, nullptr); + ASSERT_TRUE(*version <= (!v)); + + EXPECT_NE(pr_map->find(catalog::col_oid_t(1000)), pr_map->end()); + int32_t *sentinel = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1000)))); + EXPECT_EQ(*sentinel, base_val); + + if ((!v) < 1) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1001)), pr_map->end()); + int64_t *bigint = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1001)))); + EXPECT_EQ(*bigint, base_val << 31); + } + + if ((!v) < 2) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1002)), pr_map->end()); + int32_t *integer = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1002)))); + EXPECT_EQ(*integer, base_val); + } + + if ((!v) < 3) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1003)), pr_map->end()); + int16_t *smallint = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1003)))); + EXPECT_EQ(*smallint, static_cast(base_val % 1 << 15)); + } + + if ((!v) < 4) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1004)), pr_map->end()); + int8_t *tinyint = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1004)))); + EXPECT_EQ(*tinyint, static_cast(base_val % 1 << 7)); + } + + if ((!v) >= 5) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1005)), pr_map->end()); + int8_t *tinyint = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1005)))); + if (*version < 5) + EXPECT_EQ(tinyint, nullptr); + else + EXPECT_EQ(*tinyint, static_cast(base_val % 1 << 7)); + } + + if ((!v) >= 6) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1006)), pr_map->end()); + int16_t *smallint = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1006)))); + if (*version < 6) + EXPECT_EQ(smallint, nullptr); + else + EXPECT_EQ(*smallint, static_cast(base_val % 1 << 15)); + } + + if ((!v) >= 7) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1007)), pr_map->end()); + int32_t *integer = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1007)))); + if (*version < 7) + EXPECT_EQ(integer, nullptr); + else + EXPECT_EQ(*integer, base_val); + } + + if ((!v) >= 8) { + EXPECT_NE(pr_map->find(catalog::col_oid_t(1008)), pr_map->end()); + int64_t *bigint = reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(1008)))); + if (*version < 8) + EXPECT_EQ(bigint, nullptr); + else + EXPECT_EQ(*bigint, base_val << 31); + } + } + + void ValidateTable(const storage::SqlTable &table) { + auto txn = txn_manager_.BeginTransaction(); + + auto row_pair = table.InitializerForProjectedColumns(*versioned_col_oids[!schema_version_], 100, schema_version_); + auto pci = new storage::ProjectedColumnsInitializer(std::get<0>(row_pair)); + auto pc_map = new storage::ProjectionMap(std::get<1>(row_pair)); + byte *buffer = common::AllocationUtil::AllocateAligned(pci->ProjectedColumnsSize()); + auto table_iter = table.begin(schema_version_); + + while (table_iter != table.end()) { + auto pc = pci->Initialize(buffer); + table.Scan(txn, &table_iter, pc, *pc_map, schema_version_); + + for (uint i : {0u, pc->NumTuples() - 1u}) { + auto pr = pc->InterpretAsRow(i); + EXPECT_NE(pc_map->find(catalog::col_oid_t(1000)), pc_map->end()); + int32_t *base_val = reinterpret_cast(pr.AccessWithNullCheck(pc_map->at(catalog::col_oid_t(1000)))); + ValidateTuple(&pr, pc_map, schema_version_, *base_val); + } + } + + delete[] buffer; + delete pci; + delete pc_map; + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + } + + storage::RecordBufferSegmentPool buffer_pool_{100000, 100000}; + transaction::TransactionManager txn_manager_ = {&buffer_pool_, true, LOGGING_DISABLED}; + + std::default_random_engine generator_; + + catalog::table_oid_t table_oid_ = catalog::table_oid_t(42); + storage::layout_version_t schema_version_ = storage::layout_version_t(0); + storage::BlockStore block_store_{100, 100}; + std::vector cols_; + std::vector *> versioned_col_oids; + storage::GarbageCollector gc_; + + private: +}; + +// Execute a large number of inserts concurrently with schema changes and validate +// all of the data is consistent with what we expect in the end by embedding an id +// and insertion time version into the tuple. +// NOLINTNEXTLINE +TEST_F(SqlTableConcurrentTests, ConcurrentInsertsWithSchemaChanges) { + const uint32_t num_iterations = 100; + const uint32_t txns_per_thread = 10; + const uint32_t num_threads = MultiThreadTestUtil::HardwareConcurrency(); + common::WorkerPool thread_pool(num_threads, {}); + + for (uint32_t iteration = 0; iteration < num_iterations; iteration++) { + schema_version_ = storage::layout_version_t(0); + catalog::Schema schema(GenerateColumnsVector(schema_version_), schema_version_); + storage::SqlTable table(&block_store_, schema, table_oid_); + + // Begin concurrent section + auto workload = [&](uint32_t id) { + for (uint32_t t = 0; t < txns_per_thread; t++) { + storage::layout_version_t working_version = schema_version_; + auto *txn = txn_manager_.BeginTransaction(); + if (id == 0) { + if (t < 8) { + catalog::Schema schema(GenerateColumnsVector(working_version + 1), working_version + 1); + table.UpdateSchema(schema); + } + } else { + auto row_pair = table.InitializerForProjectedRow(*versioned_col_oids[!working_version], working_version); + auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); + auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); + int32_t base_val = id * txns_per_thread + t; + byte *buffer = common::AllocationUtil::AllocateAligned(pri->ProjectedRowSize()); + auto pr = pri->InitializeRow(buffer); + + PopulateProjectedRow(working_version, base_val, pr, pr_map); + + table.Insert(txn, *pr, working_version); + pr = nullptr; + delete[] buffer; + delete pri; + delete pr_map; + } + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + if (id == 0 && t < 8) schema_version_++; + } + }; + + MultiThreadTestUtil::RunThreadsUntilFinish(&thread_pool, num_threads, workload); + // End concurrent section + + ValidateTable(table); + + for (auto &version : versioned_col_oids) delete version; + versioned_col_oids.clear(); + + gc_.PerformGarbageCollection(); + gc_.PerformGarbageCollection(); + } +} + +// Initialize the table with predetermined tuples and then concurrently read +// (via Select) the tuples and verify that the data they contain is consistent +// with the write and read versions. +// NOLINTNEXTLINE +TEST_F(SqlTableConcurrentTests, ConcurrentSelectsWithSchemaChanges) { + const uint32_t num_iterations = 100; + const uint32_t txns_per_thread = 10; + const uint32_t num_threads = MultiThreadTestUtil::HardwareConcurrency(); + common::WorkerPool thread_pool(num_threads, {}); + + for (uint32_t iteration = 0; iteration < num_iterations; iteration++) { + schema_version_ = storage::layout_version_t(0); + catalog::Schema schema(GenerateColumnsVector(schema_version_), schema_version_); + storage::SqlTable table(&block_store_, schema, table_oid_); + std::vector tuples; + + // Setup table + transaction::TransactionContext *init_txn = txn_manager_.BeginTransaction(); + + auto row_pair = table.InitializerForProjectedRow(*versioned_col_oids[!schema_version_], schema_version_); + auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); + auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); + + for (uint32_t i = 0; i < txns_per_thread * num_threads; i++) { + byte *buffer = common::AllocationUtil::AllocateAligned(pri->ProjectedRowSize()); + auto pr = pri->InitializeRow(buffer); + + PopulateProjectedRow(schema_version_, i, pr, pr_map); + + tuples.emplace_back(table.Insert(init_txn, *pr, schema_version_)); + pr = nullptr; + delete[] buffer; + } + + txn_manager_.Commit(init_txn, TestCallbacks::EmptyCallback, nullptr); + + delete pri; + delete pr_map; + + // Begin concurrent section + auto workload = [&](uint32_t id) { + for (uint32_t t = 0; t < txns_per_thread; t++) { + storage::layout_version_t working_version = schema_version_; + auto *txn = txn_manager_.BeginTransaction(); + + if (id == 0) { + // Update schema if there are still more schemas + if (t < 8) { + catalog::Schema schema(GenerateColumnsVector(working_version + 1), working_version + 1); + table.UpdateSchema(schema); + } + } else { + // Select a tuple + auto row_pair = table.InitializerForProjectedRow(*versioned_col_oids[!working_version], working_version); + auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); + auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); + int32_t base_val = id * txns_per_thread + t; + byte *buffer = common::AllocationUtil::AllocateAligned(pri->ProjectedRowSize()); + auto pr = pri->InitializeRow(buffer); + + table.Select(txn, tuples[base_val], pr, *pr_map, working_version); + ValidateTuple(pr, pr_map, working_version, base_val); + + pr = nullptr; + delete[] buffer; + delete pri; + delete pr_map; + } + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + if (id == 0 && t < 8) schema_version_++; + } + }; + + MultiThreadTestUtil::RunThreadsUntilFinish(&thread_pool, num_threads, workload); + // End concurrent section + + for (auto &version : versioned_col_oids) delete version; + versioned_col_oids.clear(); + + gc_.PerformGarbageCollection(); + gc_.PerformGarbageCollection(); + } +} + +// Initialize the table with tuples and then update them to a checkable +// state that forces migration on later schemas and validate consistency. +// NOLINTNEXTLINE +TEST_F(SqlTableConcurrentTests, ConcurrentUpdatesWithSchemaChanges) { + const uint32_t num_iterations = 100; + const uint32_t txns_per_thread = 10; + const uint32_t num_threads = MultiThreadTestUtil::HardwareConcurrency(); + common::WorkerPool thread_pool(num_threads, {}); + + for (uint32_t iteration = 0; iteration < num_iterations; iteration++) { + schema_version_ = storage::layout_version_t(0); + catalog::Schema schema(GenerateColumnsVector(schema_version_), schema_version_); + storage::SqlTable table(&block_store_, schema, table_oid_); + std::vector tuples; + + // Setup table + auto *init_txn = txn_manager_.BeginTransaction(); + + auto row_pair = table.InitializerForProjectedRow(*versioned_col_oids[!schema_version_], schema_version_); + auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); + auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); + + for (uint32_t i = 0; i < txns_per_thread * num_threads; i++) { + byte *buffer = common::AllocationUtil::AllocateAligned(pri->ProjectedRowSize()); + auto pr = pri->InitializeRow(buffer); + + PopulateProjectedRow(schema_version_, i, pr, pr_map); + + tuples.emplace_back(table.Insert(init_txn, *pr, schema_version_)); + pr = nullptr; + delete[] buffer; + } + txn_manager_.Commit(init_txn, TestCallbacks::EmptyCallback, nullptr); + + delete pri; + delete pr_map; + + // Begin concurrent section + auto workload = [&](uint32_t id) { + for (uint32_t t = 0; t < txns_per_thread; t++) { + storage::layout_version_t working_version = schema_version_; + auto *txn = txn_manager_.BeginTransaction(); + if (id == 0) { + // Execute a schema change if there are still schemas left + if (t < 8) { + catalog::Schema schema(GenerateColumnsVector(working_version + 1), working_version + 1); + table.UpdateSchema(schema); + } + } else { + // Update a tuple and track if it moved slots + auto row_pair = table.InitializerForProjectedRow(*versioned_col_oids[!working_version], working_version); + auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); + auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); + int32_t base_val = id * txns_per_thread + t; + byte *buffer = common::AllocationUtil::AllocateAligned(pri->ProjectedRowSize()); + auto pr = pri->InitializeRow(buffer); + + table.Select(txn, tuples[base_val], pr, *pr_map, working_version); + ValidateTuple(pr, pr_map, working_version, base_val); + + EXPECT_NE(pr_map->find(catalog::col_oid_t(100)), pr_map->end()); + uint32_t *version = + reinterpret_cast(pr->AccessWithNullCheck(pr_map->at(catalog::col_oid_t(100)))); + + EXPECT_EQ(*version, (!tuples[base_val].GetBlock()->layout_version_)); + auto old_version = tuples[base_val].GetBlock()->layout_version_; + PopulateProjectedRow(working_version, base_val, pr, pr_map); + + auto result = table.Update(txn, tuples[base_val], pr, *pr_map, working_version); + + auto new_version = result.second.GetBlock()->layout_version_; + EXPECT_LE(old_version, new_version); + EXPECT_LE(new_version, working_version); + EXPECT_TRUE(result.first); + EXPECT_EQ(*version, !working_version); + if (old_version != working_version && (!working_version) >= 5) + EXPECT_NE(tuples[base_val], result.second); + else if (old_version == working_version) + EXPECT_EQ(tuples[base_val], result.second); + else + EXPECT_EQ(tuples[base_val], result.second); + + tuples[base_val] = result.second; + + pr = nullptr; + delete[] buffer; + delete pri; + delete pr_map; + } + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + if (id == 0 && t < 8) schema_version_++; + } + }; + + MultiThreadTestUtil::RunThreadsUntilFinish(&thread_pool, num_threads, workload); + // End concurrent section + + ValidateTable(table); + + for (auto &version : versioned_col_oids) delete version; + versioned_col_oids.clear(); + + gc_.PerformGarbageCollection(); + gc_.PerformGarbageCollection(); + } +} +} // namespace terrier diff --git a/test/storage/sql_table_test.cpp b/test/storage/sql_table_test.cpp index 7204264c70..79728fe817 100644 --- a/test/storage/sql_table_test.cpp +++ b/test/storage/sql_table_test.cpp @@ -1,6 +1,7 @@ #include "storage/sql_table.h" #include #include +#include #include #include #include @@ -13,10 +14,10 @@ namespace terrier { /** * Help class to simplify operations on a SqlTable */ -class SqlTableRW { +class SqlTableTestRW { public: - explicit SqlTableRW(catalog::table_oid_t table_oid) : table_oid_(table_oid) {} - ~SqlTableRW() { + explicit SqlTableTestRW(catalog::table_oid_t table_oid) : version_(0), table_oid_(table_oid) {} + ~SqlTableTestRW() { delete pri_; delete pr_map_; delete schema_; @@ -38,11 +39,48 @@ class SqlTableRW { cols_.emplace_back(name, type, 255, nullable, oid); } + void AddColumn(transaction::TransactionContext *txn, std::string name, type::TypeId type, bool nullable, + catalog::col_oid_t oid, byte *default_value = nullptr) { + // update columns, schema and layout + cols_.emplace_back(name, type, nullable, oid, default_value); + delete schema_; + schema_ = new catalog::Schema(cols_, next_version_++); + + table_->UpdateSchema(*schema_); + + col_oids_.clear(); + for (const auto &c : cols_) { + col_oids_.emplace_back(c.GetOid()); + } + + delete pri_; + delete pr_map_; + + // save information needed for (later) reading and writing + auto row_pair = table_->InitializerForProjectedRow(col_oids_, version_); + pri_ = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); + pr_map_ = new storage::ProjectionMap(std::get<1>(row_pair)); + } + + /** + * Set a default value for the given column oid. This doesn't need to call UpdateSchema + * @param oid oid of the column + * @param default_value the default value to be set + */ + void SetColumnDefault(catalog::col_oid_t oid, byte *default_value) { + // Get the index of this oid in the col_oids vector + auto col_pos = std::find(col_oids_.begin(), col_oids_.end(), oid); + TERRIER_ASSERT(col_pos != col_oids_.end(), "oid doesn't exist in the table"); + auto idx = col_pos - col_oids_.begin(); + + cols_.at(idx).SetDefault(default_value); + } + /** * Create the SQL table. */ void Create() { - schema_ = new catalog::Schema(cols_); + schema_ = new catalog::Schema(cols_, next_version_++); table_ = new storage::SqlTable(&block_store_, *schema_, table_oid_); for (const auto &c : cols_) { @@ -50,50 +88,84 @@ class SqlTableRW { } // save information needed for (later) reading and writing - auto row_pair = table_->InitializerForProjectedRow(col_oids_); + auto row_pair = table_->InitializerForProjectedRow(col_oids_, version_); pri_ = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); pr_map_ = new storage::ProjectionMap(std::get<1>(row_pair)); } /** - * First step in writing a row. + * First step in inserting a row. */ - void StartRow() { - insert_buffer_ = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); - insert_ = pri_->InitializeRow(insert_buffer_); - } + void StartInsertRow() { ResetProjectedRow(col_oids_); } /** * Insert the row into the table * @return slot where the row was created */ - storage::TupleSlot EndRowAndInsert() { - auto txn = txn_manager_.BeginTransaction(); - auto slot = table_->Insert(txn, *insert_); - insert_ = nullptr; - txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); - - delete[] insert_buffer_; - delete txn; + storage::TupleSlot EndInsertRow(transaction::TransactionContext *txn) { + auto slot = table_->Insert(txn, *pr_, version_); + pr_ = nullptr; + delete[] buffer_; return storage::TupleSlot(slot.GetBlock(), slot.GetOffset()); } + void StartUpdateRow(const std::vector &col_oids) { + auto pr_pair = table_->InitializerForProjectedRow(col_oids, version_); + buffer_ = common::AllocationUtil::AllocateAligned(pr_pair.first.ProjectedRowSize()); + pr_ = pr_pair.first.InitializeRow(buffer_); + + delete pri_; + delete pr_map_; + pri_ = new storage::ProjectedRowInitializer(std::get<0>(pr_pair)); + pr_map_ = new storage::ProjectionMap(std::get<1>(pr_pair)); + } + + std::pair EndUpdateRow(transaction::TransactionContext *txn, storage::TupleSlot slot) { + auto result_pair = table_->Update(txn, slot, pr_, *pr_map_, version_); + pr_ = nullptr; + delete[] buffer_; + return result_pair; + } + + /** + * Check if a tuple is visible to you + * @param txn + * @param slot + * @return + */ + bool Visible(transaction::TransactionContext *txn, storage::TupleSlot slot) { + buffer_ = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); + pr_ = pri_->InitializeRow(buffer_); + bool result = table_->Select(txn, slot, pr_, *pr_map_, version_); + delete[] buffer_; + return result; + } + + bool IsNullColInRow(transaction::TransactionContext *txn, catalog::col_oid_t col_oid, storage::TupleSlot slot) { + auto read_buffer = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); + + storage::ProjectedRow *read = pri_->InitializeRow(read_buffer); + table_->Select(txn, slot, read, *pr_map_, version_); + byte *col_p = read->AccessWithNullCheck(pr_map_->at(col_oid)); + + delete[] read_buffer; + return (col_p == nullptr); + } + /** * Read an integer from a row * @param col_num column number in the schema * @param slot - tuple to read from * @return integer value */ - uint32_t GetIntColInRow(int32_t col_num, storage::TupleSlot slot) { - auto txn = txn_manager_.BeginTransaction(); + uint32_t GetIntColInRow(transaction::TransactionContext *txn, catalog::col_oid_t col_oid, storage::TupleSlot slot) { auto read_buffer = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); + storage::ProjectedRow *read = pri_->InitializeRow(read_buffer); - table_->Select(txn, slot, read); - byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); - txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + table_->Select(txn, slot, read, *pr_map_, version_); + byte *col_p = read->AccessWithNullCheck(pr_map_->at(col_oid)); auto ret_val = *(reinterpret_cast(col_p)); - delete txn; delete[] read_buffer; return ret_val; } @@ -104,16 +176,14 @@ class SqlTableRW { * @param slot - tuple to read from * @return integer value */ - uint64_t GetInt64ColInRow(int32_t col_num, storage::TupleSlot slot) { - auto txn = txn_manager_.BeginTransaction(); + uint64_t GetInt64ColInRow(transaction::TransactionContext *txn, catalog::col_oid_t col_oid, storage::TupleSlot slot) { auto read_buffer = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); + storage::ProjectedRow *read = pri_->InitializeRow(read_buffer); - table_->Select(txn, slot, read); - byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); - txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + table_->Select(txn, slot, read, *pr_map_, version_); + byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oid)); auto ret_val = *(reinterpret_cast(col_p)); - delete txn; delete[] read_buffer; return ret_val; } @@ -124,16 +194,14 @@ class SqlTableRW { * @param slot - tuple to read from * @return integer value */ - uint16_t GetInt16ColInRow(int32_t col_num, storage::TupleSlot slot) { - auto txn = txn_manager_.BeginTransaction(); + uint16_t GetInt16ColInRow(transaction::TransactionContext *txn, catalog::col_oid_t col_oid, storage::TupleSlot slot) { auto read_buffer = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); + storage::ProjectedRow *read = pri_->InitializeRow(read_buffer); - table_->Select(txn, slot, read); - byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); - txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + table_->Select(txn, slot, read, *pr_map_, version_); + byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oid)); auto ret_val = *(reinterpret_cast(col_p)); - delete txn; delete[] read_buffer; return ret_val; } @@ -144,16 +212,14 @@ class SqlTableRW { * @param slot - tuple to read from * @return integer value */ - uint8_t GetInt8ColInRow(int32_t col_num, storage::TupleSlot slot) { - auto txn = txn_manager_.BeginTransaction(); + uint8_t GetInt8ColInRow(transaction::TransactionContext *txn, catalog::col_oid_t col_oid, storage::TupleSlot slot) { auto read_buffer = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); + storage::ProjectedRow *read = pri_->InitializeRow(read_buffer); - table_->Select(txn, slot, read); - byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); - txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + table_->Select(txn, slot, read, *pr_map_, version_); + byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oid)); auto ret_val = *(reinterpret_cast(col_p)); - delete txn; delete[] read_buffer; return ret_val; } @@ -163,8 +229,8 @@ class SqlTableRW { * @param col_num column number in the schema * @param value to save */ - void SetIntColInRow(int32_t col_num, int32_t value) { - byte *col_p = insert_->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); + void SetIntColInRow(catalog::col_oid_t col_oid, int32_t value) { + byte *col_p = pr_->AccessForceNotNull(pr_map_->at(col_oid)); (*reinterpret_cast(col_p)) = value; } @@ -173,8 +239,8 @@ class SqlTableRW { * @param col_num column number in the schema * @param value to save */ - void SetInt64ColInRow(int32_t col_num, uint64_t value) { - byte *col_p = insert_->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); + void SetInt64ColInRow(catalog::col_oid_t col_oid, uint64_t value) { + byte *col_p = pr_->AccessForceNotNull(pr_map_->at(col_oid)); (*reinterpret_cast(col_p)) = value; } @@ -183,8 +249,8 @@ class SqlTableRW { * @param col_num column number in the schema * @param value to save */ - void SetInt16ColInRow(int32_t col_num, uint16_t value) { - byte *col_p = insert_->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); + void SetInt16ColInRow(catalog::col_oid_t col_oid, uint16_t value) { + byte *col_p = pr_->AccessForceNotNull(pr_map_->at(col_oid)); (*reinterpret_cast(col_p)) = value; } @@ -193,24 +259,28 @@ class SqlTableRW { * @param col_num column number in the schema * @param value to save */ - void SetInt8ColInRow(int32_t col_num, uint8_t value) { - byte *col_p = insert_->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); + void SetInt8ColInRow(catalog::col_oid_t col_oid, uint8_t value) { + byte *col_p = pr_->AccessForceNotNull(pr_map_->at(col_oid)); (*reinterpret_cast(col_p)) = value; } + /** + * Set the attribute of col_oid to null + * @param col_oid col_oid of the column in the schema + */ + void SetNullInRow(catalog::col_oid_t col_oid) { pr_->SetNull(pr_map_->at(col_oid)); } + /** * Read a string from a row * @param col_num column number in the schema * @param slot - tuple to read from - * @return malloc'ed C string (with null terminator). Caller must - * free. + * @return malloc'ed C string (with null terminator). Caller must free. */ - char *GetVarcharColInRow(int32_t col_num, storage::TupleSlot slot) { - auto txn = txn_manager_.BeginTransaction(); + char *GetVarcharColInRow(transaction::TransactionContext *txn, catalog::col_oid_t(col_oid), storage::TupleSlot slot) { auto read_buffer = common::AllocationUtil::AllocateAligned(pri_->ProjectedRowSize()); storage::ProjectedRow *read = pri_->InitializeRow(read_buffer); - table_->Select(txn, slot, read); - byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); + table_->Select(txn, slot, read, *pr_map_, version_); + byte *col_p = read->AccessForceNotNull(pr_map_->at(col_oid)); auto *entry = reinterpret_cast(col_p); // stored string has no null terminator, add space for it @@ -220,8 +290,6 @@ class SqlTableRW { std::memcpy(ret_st, entry->Content(), size); // add the null terminator *(ret_st + size - 1) = 0; - txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); - delete txn; delete[] read_buffer; return ret_st; } @@ -231,8 +299,8 @@ class SqlTableRW { * @param col_num column number in the schema * @param st C string to save. */ - void SetVarcharColInRow(int32_t col_num, const char *st) { - byte *col_p = insert_->AccessForceNotNull(pr_map_->at(col_oids_[col_num])); + void SetVarcharColInRow(catalog::col_oid_t col_oid, const char *st) { + byte *col_p = pr_->AccessForceNotNull(pr_map_->at(col_oid)); // string size, without null terminator auto size = static_cast(strlen(st)); if (size <= storage::VarlenEntry::InlineThreshold()) { @@ -246,14 +314,18 @@ class SqlTableRW { } } - private: - storage::RecordBufferSegmentPool buffer_pool_{100, 100}; - transaction::TransactionManager txn_manager_ = {&buffer_pool_, true, LOGGING_DISABLED}; + public: + // This is a public field that transactions can set and read. + // The purpose is to record the version for each transaction. In reality this information should be retrieved from + // catalog. We just make it for the test case. + storage::layout_version_t version_; + storage::SqlTable *table_ = nullptr; + private: storage::BlockStore block_store_{100, 100}; catalog::table_oid_t table_oid_; - storage::SqlTable *table_ = nullptr; + // keep meta-data for most recent schema catalog::Schema *schema_ = nullptr; std::vector cols_; std::vector col_oids_; @@ -261,67 +333,463 @@ class SqlTableRW { storage::ProjectedRowInitializer *pri_ = nullptr; storage::ProjectionMap *pr_map_ = nullptr; - byte *insert_buffer_ = nullptr; - storage::ProjectedRow *insert_ = nullptr; + byte *buffer_ = nullptr; + storage::ProjectedRow *pr_ = nullptr; + + storage::layout_version_t next_version_ = storage::layout_version_t(0); + + void ResetProjectedRow(const std::vector &col_oids) { + auto pr_pair = table_->InitializerForProjectedRow(col_oids, version_); + buffer_ = common::AllocationUtil::AllocateAligned(pr_pair.first.ProjectedRowSize()); + pr_ = pr_pair.first.InitializeRow(buffer_); + + delete pri_; + delete pr_map_; + pri_ = new storage::ProjectedRowInitializer(std::get<0>(pr_pair)); + pr_map_ = new storage::ProjectionMap(std::get<1>(pr_pair)); + } }; struct SqlTableTests : public TerrierTest { void SetUp() override { TerrierTest::SetUp(); } void TearDown() override { TerrierTest::TearDown(); } + + storage::RecordBufferSegmentPool buffer_pool_{100, 100}; + transaction::TransactionManager txn_manager_ = {&buffer_pool_, true, LOGGING_DISABLED}; }; // NOLINTNEXTLINE -TEST_F(SqlTableTests, SelectInsertTest) { - SqlTableRW table(catalog::table_oid_t(2)); - +TEST_F(SqlTableTests, SelectTest) { + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); table.DefineColumn("datname", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); table.Create(); - table.StartRow(); - table.SetIntColInRow(0, 100); - table.SetIntColInRow(1, 15721); - storage::TupleSlot row1_slot = table.EndRowAndInsert(); + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetIntColInRow(catalog::col_oid_t(1), 10000); + storage::TupleSlot row1_slot = table.EndInsertRow(txn); - table.StartRow(); - table.SetIntColInRow(0, 200); - table.SetIntColInRow(1, 25721); - storage::TupleSlot row2_slot = table.EndRowAndInsert(); + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 200); + table.SetIntColInRow(catalog::col_oid_t(1), 10001); + storage::TupleSlot row2_slot = table.EndInsertRow(txn); - uint32_t id = table.GetIntColInRow(0, row1_slot); + uint32_t id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row1_slot); + EXPECT_EQ(100, id); + uint32_t datname = table.GetIntColInRow(txn, catalog::col_oid_t(1), row1_slot); + EXPECT_EQ(10000, datname); + + // Update the observable schema version. Outside of testing this would be + // an update in the catalog as part of the larger transaction. + table.version_ = storage::layout_version_t(1); + int default_val = 42; + // Add a new column with a default value + table.AddColumn(txn, "new_col", type::TypeId::INTEGER, true, catalog::col_oid_t(2), + reinterpret_cast(&default_val)); + + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row1_slot); EXPECT_EQ(100, id); - uint32_t datname = table.GetIntColInRow(1, row1_slot); - EXPECT_EQ(15721, datname); + datname = table.GetIntColInRow(txn, catalog::col_oid_t(1), row1_slot); + EXPECT_EQ(10000, datname); - id = table.GetIntColInRow(0, row2_slot); + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row2_slot); EXPECT_EQ(200, id); - datname = table.GetIntColInRow(1, row2_slot); - EXPECT_EQ(25721, datname); + datname = table.GetIntColInRow(txn, catalog::col_oid_t(1), row2_slot); + EXPECT_EQ(10001, datname); + + // The new_column should return the default_value for the old version slots + uint32_t new_col = table.GetIntColInRow(txn, catalog::col_oid_t(2), row1_slot); + EXPECT_EQ(default_val, new_col); + new_col = table.GetIntColInRow(txn, catalog::col_oid_t(2), row2_slot); + EXPECT_EQ(default_val, new_col); + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; +} + +// NOLINTNEXTLINE +TEST_F(SqlTableTests, InsertTest) { + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); + table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); + table.DefineColumn("datname", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); + table.Create(); + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetIntColInRow(catalog::col_oid_t(1), 10000); + storage::TupleSlot row1_slot = table.EndInsertRow(txn); + + uint32_t id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row1_slot); + EXPECT_EQ(100, id); + uint32_t datname = table.GetIntColInRow(txn, catalog::col_oid_t(1), row1_slot); + EXPECT_EQ(10000, datname); + + // manually set the version of the transaction to be 1 + table.version_ = storage::layout_version_t(1); + table.AddColumn(txn, "new_col", type::TypeId::INTEGER, true, catalog::col_oid_t(2), nullptr); + + // insert (300, 10002, null) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 300); + table.SetIntColInRow(catalog::col_oid_t(1), 10002); + storage::TupleSlot row3_slot = table.EndInsertRow(txn); + + // insert (400, 10003, 42) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 400); + table.SetIntColInRow(catalog::col_oid_t(1), 10003); + table.SetIntColInRow(catalog::col_oid_t(2), 42); + storage::TupleSlot row4_slot = table.EndInsertRow(txn); + + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row3_slot); + EXPECT_EQ(300, id); + datname = table.GetIntColInRow(txn, catalog::col_oid_t(1), row3_slot); + EXPECT_EQ(10002, datname); + bool new_col_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(2), row1_slot); + EXPECT_TRUE(new_col_is_null); + + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row4_slot); + EXPECT_EQ(400, id); + datname = table.GetIntColInRow(txn, catalog::col_oid_t(1), row4_slot); + EXPECT_EQ(10003, datname); + datname = table.GetIntColInRow(txn, catalog::col_oid_t(2), row4_slot); + EXPECT_EQ(42, datname); + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; +} + +// NOLINTNEXTLINE +TEST_F(SqlTableTests, DeleteTest) { + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); + table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); + table.DefineColumn("datname", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); + table.Create(); + + // insert (100, 10000) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetIntColInRow(catalog::col_oid_t(1), 10000); + storage::TupleSlot row1_slot = table.EndInsertRow(txn); + + // insert (200, 10001) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetIntColInRow(catalog::col_oid_t(1), 10001); + storage::TupleSlot row2_slot = table.EndInsertRow(txn); + + // manually set the version of the transaction to be 1 + table.version_ = storage::layout_version_t(1); + table.AddColumn(txn, "new_col", type::TypeId::INTEGER, true, catalog::col_oid_t(2), nullptr); + + // insert (300, 10002, null) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 300); + table.SetIntColInRow(catalog::col_oid_t(1), 10002); + storage::TupleSlot row3_slot = table.EndInsertRow(txn); + + // insert (400, 10003, 42) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 400); + table.SetIntColInRow(catalog::col_oid_t(1), 10003); + table.SetIntColInRow(catalog::col_oid_t(2), 42); + storage::TupleSlot row4_slot = table.EndInsertRow(txn); + + // delete (100, 10000, null) and (300, 10002, null) + EXPECT_TRUE(table.table_->Delete(txn, row1_slot, table.version_)); + EXPECT_TRUE(table.table_->Delete(txn, row3_slot, table.version_)); + + EXPECT_TRUE(table.Visible(txn, row2_slot)); + EXPECT_TRUE(table.Visible(txn, row4_slot)); + + EXPECT_FALSE(table.Visible(txn, row1_slot)); + EXPECT_FALSE(table.Visible(txn, row3_slot)); + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; +} + +// NOLINTNEXTLINE +TEST_F(SqlTableTests, UpdateTest) { + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); + table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); + table.DefineColumn("datname", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); + table.Create(); + + // insert (100, 10000) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetIntColInRow(catalog::col_oid_t(1), 10000); + storage::TupleSlot row1_slot = table.EndInsertRow(txn); + + // insert (200, 10001) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 200); + table.SetIntColInRow(catalog::col_oid_t(1), 10001); + storage::TupleSlot row2_slot = table.EndInsertRow(txn); + + // manually set the version of the transaction to be 1 + table.version_ = storage::layout_version_t(1); + table.AddColumn(txn, "new_col", type::TypeId::INTEGER, true, catalog::col_oid_t(2), nullptr); + + // insert (300, 10002, null) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 300); + table.SetIntColInRow(catalog::col_oid_t(1), 10002); + table.EndInsertRow(txn); + + // insert (400, 10003, 42) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 400); + table.SetIntColInRow(catalog::col_oid_t(1), 10003); + table.SetIntColInRow(catalog::col_oid_t(2), 42); + storage::TupleSlot row4_slot = table.EndInsertRow(txn); + + // update (200, 10001, null) -> (200, 11001, null) + std::vector update_oids; + update_oids.emplace_back(catalog::col_oid_t(1)); + table.StartUpdateRow(update_oids); + table.SetIntColInRow(catalog::col_oid_t(1), 11001); + auto result = table.EndUpdateRow(txn, row2_slot); + + EXPECT_TRUE(result.first); + EXPECT_EQ(result.second.GetBlock(), row2_slot.GetBlock()); + EXPECT_EQ(result.second.GetOffset(), row2_slot.GetOffset()); + uint32_t new_val = table.GetIntColInRow(txn, catalog::col_oid_t(1), row2_slot); + EXPECT_EQ(new_val, 11001); + + // update (400, 10003, 42) -> (400, 11003, 420) + update_oids.clear(); + update_oids.emplace_back(catalog::col_oid_t(1)); + update_oids.emplace_back(catalog::col_oid_t(2)); + table.StartUpdateRow(update_oids); + table.SetIntColInRow(catalog::col_oid_t(1), 11003); + table.SetIntColInRow(catalog::col_oid_t(2), 420); + result = table.EndUpdateRow(txn, row4_slot); + + EXPECT_TRUE(result.first); + EXPECT_EQ(result.second.GetBlock(), row4_slot.GetBlock()); + EXPECT_EQ(result.second.GetOffset(), row4_slot.GetOffset()); + new_val = table.GetIntColInRow(txn, catalog::col_oid_t(1), result.second); + EXPECT_EQ(new_val, 11003); + new_val = table.GetIntColInRow(txn, catalog::col_oid_t(2), result.second); + EXPECT_EQ(new_val, 420); + + // update (100, 10000, null) -> (100, 11000, 420) + update_oids.clear(); + update_oids.emplace_back(catalog::col_oid_t(1)); + update_oids.emplace_back(catalog::col_oid_t(2)); + table.StartUpdateRow(update_oids); + table.SetIntColInRow(catalog::col_oid_t(1), 11000); + table.SetIntColInRow(catalog::col_oid_t(2), 420); + result = table.EndUpdateRow(txn, row1_slot); + + EXPECT_TRUE(result.first); + EXPECT_NE(result.second.GetBlock(), row1_slot.GetBlock()); + new_val = table.GetIntColInRow(txn, catalog::col_oid_t(1), result.second); + EXPECT_EQ(new_val, 11000); + new_val = table.GetIntColInRow(txn, catalog::col_oid_t(2), result.second); + EXPECT_EQ(new_val, 420); + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; +} + +// NOLINTNEXTLINE +TEST_F(SqlTableTests, ScanTest) { + std::map datname_map; + std::map new_col_map; + uint32_t new_col_default_value = 1729; + std::map seen_map; + + datname_map[100] = 10000; + datname_map[200] = 10001; + datname_map[300] = 10002; + datname_map[400] = 10003; + + // new_col_map[100] = NULL (created before column added) + // new_col_map[200] = NULL (created before column added) + // new_col_map[300] = NULL (Not explicitly specified); + new_col_map[400] = 42; + + seen_map[100] = false; + seen_map[200] = false; + seen_map[300] = false; + seen_map[400] = false; + + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); + table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); + table.DefineColumn("datname", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); + table.Create(); + + // insert (100, 10000) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetIntColInRow(catalog::col_oid_t(1), datname_map[100]); + table.EndInsertRow(txn); + + // insert (200, 10001) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 200); + table.SetIntColInRow(catalog::col_oid_t(1), datname_map[200]); + table.EndInsertRow(txn); + + // manually set the version of the transaction to be 1 + table.version_ = storage::layout_version_t(1); + table.AddColumn(txn, "new_col", type::TypeId::INTEGER, true, catalog::col_oid_t(2), + reinterpret_cast(&new_col_default_value)); + + // insert (300, 10002, 1729) - Default value populated by the execution engine + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 300); + table.SetIntColInRow(catalog::col_oid_t(1), datname_map[300]); + table.SetIntColInRow(catalog::col_oid_t(2), new_col_default_value); + table.EndInsertRow(txn); + + // insert (400, 10003, 42) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 400); + table.SetIntColInRow(catalog::col_oid_t(1), datname_map[400]); + table.SetIntColInRow(catalog::col_oid_t(2), new_col_map[400]); + table.EndInsertRow(txn); + + // begin scan + std::vector all_col_oids; + all_col_oids.emplace_back(0); + all_col_oids.emplace_back(1); + all_col_oids.emplace_back(2); + + auto pc_pair = table.table_->InitializerForProjectedColumns(all_col_oids, 10, table.version_); + byte *buffer = common::AllocationUtil::AllocateAligned(pc_pair.first.ProjectedColumnsSize()); + storage::ProjectedColumns *pc = pc_pair.first.Initialize(buffer); + + // scan + auto start_pos = table.table_->begin(table.version_); + table.table_->Scan(txn, &start_pos, pc, pc_pair.second, table.version_); + EXPECT_TRUE(start_pos != table.table_->end()); + + // check the number of tuples we found + EXPECT_EQ(pc->NumTuples(), 2); + + auto row1 = pc->InterpretAsRow(0); + byte *value = row1.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(0))); + EXPECT_NE(value, nullptr); + uint32_t id = *reinterpret_cast(value); + EXPECT_FALSE(seen_map[id]); + seen_map[id] = true; + value = row1.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(1))); + EXPECT_NE(value, nullptr); + uint32_t datname = *reinterpret_cast(value); + EXPECT_EQ(datname, datname_map[id]); + value = row1.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(2))); + uint32_t new_col = *reinterpret_cast(value); + if (id != 400) { + EXPECT_EQ(new_col, new_col_default_value); + } else { + EXPECT_EQ(new_col, new_col_map[id]); + } + + // check the if we get (200, 10001, null) + auto row2 = pc->InterpretAsRow(1); + value = row2.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(0))); + EXPECT_NE(value, nullptr); + id = *reinterpret_cast(value); + EXPECT_FALSE(seen_map[id]); + seen_map[id] = true; + value = row2.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(1))); + EXPECT_NE(value, nullptr); + datname = *reinterpret_cast(value); + EXPECT_EQ(datname, datname_map[id]); + value = row2.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(2))); + new_col = *reinterpret_cast(value); + if (id != 400) { + EXPECT_EQ(new_col, new_col_default_value); + } else { + EXPECT_EQ(new_col, new_col_map[id]); + } + + pc = pc_pair.first.Initialize(buffer); + // Need to scan again to get the rest of the data + table.table_->Scan(txn, &start_pos, pc, pc_pair.second, table.version_); + EXPECT_EQ(pc->NumTuples(), 2); + EXPECT_TRUE(start_pos == table.table_->end()); + + auto row3 = pc->InterpretAsRow(0); + value = row3.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(0))); + EXPECT_NE(value, nullptr); + id = *reinterpret_cast(value); + EXPECT_FALSE(seen_map[id]); + seen_map[id] = true; + value = row3.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(1))); + EXPECT_NE(value, nullptr); + datname = *reinterpret_cast(value); + EXPECT_EQ(datname, datname_map[id]); + value = row3.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(2))); + new_col = *reinterpret_cast(value); + if (id != 400) { + EXPECT_EQ(new_col, new_col_default_value); + } else { + EXPECT_EQ(new_col, new_col_map[id]); + } + + // check the if we get (200, 10001, null) + auto row4 = pc->InterpretAsRow(1); + value = row4.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(0))); + EXPECT_NE(value, nullptr); + id = *reinterpret_cast(value); + EXPECT_FALSE(seen_map[id]); + seen_map[id] = true; + value = row4.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(1))); + EXPECT_NE(value, nullptr); + datname = *reinterpret_cast(value); + EXPECT_EQ(datname, datname_map[id]); + value = row4.AccessWithNullCheck(pc_pair.second.at(catalog::col_oid_t(2))); + new_col = *reinterpret_cast(value); + if (id != 400) { + EXPECT_EQ(new_col, new_col_default_value); + } else { + EXPECT_EQ(new_col, new_col_map[id]); + } + + for (auto iter : seen_map) EXPECT_TRUE(iter.second); + + delete[] buffer; + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; } // NOLINTNEXTLINE TEST_F(SqlTableTests, VarlenInsertTest) { - SqlTableRW table(catalog::table_oid_t(2)); + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); table.DefineColumn("datname", type::TypeId::VARCHAR, false, catalog::col_oid_t(1)); table.Create(); - table.StartRow(); - table.SetIntColInRow(0, 100); - table.SetVarcharColInRow(1, "name"); - storage::TupleSlot row_slot = table.EndRowAndInsert(); + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 100); + table.SetVarcharColInRow(catalog::col_oid_t(1), "name"); + storage::TupleSlot row_slot = table.EndInsertRow(txn); - uint32_t id = table.GetIntColInRow(0, row_slot); + uint32_t id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row_slot); EXPECT_EQ(100, id); - char *table_name = table.GetVarcharColInRow(1, row_slot); + char *table_name = table.GetVarcharColInRow(txn, catalog::col_oid_t(1), row_slot); EXPECT_STREQ("name", table_name); delete[] table_name; + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; } // NOLINTNEXTLINE TEST_F(SqlTableTests, MultipleColumnWidths) { - SqlTableRW table(catalog::table_oid_t(2)); + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); table.DefineColumn("bigint", type::TypeId::BIGINT, false, catalog::col_oid_t(1001)); table.DefineColumn("integer", type::TypeId::INTEGER, false, catalog::col_oid_t(1002)); @@ -329,22 +797,193 @@ TEST_F(SqlTableTests, MultipleColumnWidths) { table.DefineColumn("tinyint", type::TypeId::TINYINT, false, catalog::col_oid_t(1004)); table.Create(); - table.StartRow(); - table.SetInt64ColInRow(0, 10000000000); - table.SetIntColInRow(1, 100000); - table.SetInt16ColInRow(2, 512); - table.SetInt8ColInRow(3, 42); - storage::TupleSlot row_slot = table.EndRowAndInsert(); + table.StartInsertRow(); + table.SetInt64ColInRow(catalog::col_oid_t(1001), 10000000000); + table.SetIntColInRow(catalog::col_oid_t(1002), 100000); + table.SetInt16ColInRow(catalog::col_oid_t(1003), 512); + table.SetInt8ColInRow(catalog::col_oid_t(1004), 42); + storage::TupleSlot row_slot = table.EndInsertRow(txn); // Check data - uint64_t bigint = table.GetInt64ColInRow(0, row_slot); - uint32_t integer = table.GetIntColInRow(1, row_slot); - uint16_t smallint = table.GetInt16ColInRow(2, row_slot); - uint8_t tinyint = table.GetInt8ColInRow(3, row_slot); + uint64_t bigint = table.GetInt64ColInRow(txn, catalog::col_oid_t(1001), row_slot); + uint32_t integer = table.GetIntColInRow(txn, catalog::col_oid_t(1002), row_slot); + uint16_t smallint = table.GetInt16ColInRow(txn, catalog::col_oid_t(1003), row_slot); + uint8_t tinyint = table.GetInt8ColInRow(txn, catalog::col_oid_t(1004), row_slot); EXPECT_EQ(bigint, 10000000000); EXPECT_EQ(integer, 100000); EXPECT_EQ(smallint, 512); EXPECT_EQ(tinyint, 42); + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; +} + +// TODO(Sai): Merge the sql_table_test and sql_table_concurrent_test frameworks to avoid repetition +// NOLINTNEXTLINE +TEST_F(SqlTableTests, BasicDefaultValuesTest) { + // Test for adding new columns with default values + // Default value for a column should only be filled in for the versions that don't have that column + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); + + // Create a table of 2 columns, with no default values at the start. + table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); + table.DefineColumn("col1", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); + table.Create(); + + // Insert (1, 100) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 1); + table.SetIntColInRow(catalog::col_oid_t(1), 100); + storage::TupleSlot row1_slot = table.EndInsertRow(txn); + + // Add a new column with a default value and insert a row + // Explicitly set the layout version number + table.version_ = storage::layout_version_t(1); + int col2_default = 42; + table.AddColumn(txn, "col2", type::TypeId::INTEGER, true, catalog::col_oid_t(2), + reinterpret_cast(&col2_default)); + + // Insert (2, NULL, 890) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 2); + table.SetIntColInRow(catalog::col_oid_t(2), 890); + storage::TupleSlot row2_slot = table.EndInsertRow(txn); + + // 1st row should be (1, 100, 42) by now + uint32_t id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row1_slot); + EXPECT_EQ(1, id); + uint32_t col1 = table.GetIntColInRow(txn, catalog::col_oid_t(1), row1_slot); + EXPECT_EQ(100, col1); + uint32_t col2 = table.GetIntColInRow(txn, catalog::col_oid_t(2), row1_slot); + EXPECT_EQ(col2_default, col2); + + // Add another column with a default value and insert a row + table.version_ = storage::layout_version_t(2); + int col3_default = 1729; + table.AddColumn(txn, "col3", type::TypeId::INTEGER, true, catalog::col_oid_t(3), + reinterpret_cast(&col3_default)); + + // Insert (3, 300, NULL, NULL) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 3); + table.SetIntColInRow(catalog::col_oid_t(1), 300); + storage::TupleSlot row3_slot = table.EndInsertRow(txn); + + // SELECT and validate all the rows + // 1st row should be (1, 100, 42, 1729) + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row1_slot); + EXPECT_EQ(1, id); + col1 = table.GetIntColInRow(txn, catalog::col_oid_t(1), row1_slot); + EXPECT_EQ(100, col1); + col2 = table.GetIntColInRow(txn, catalog::col_oid_t(2), row1_slot); + EXPECT_EQ(col2_default, col2); + uint32_t col3 = table.GetIntColInRow(txn, catalog::col_oid_t(3), row1_slot); + EXPECT_EQ(col3_default, col3); + + // 2nd tuple should be (2, NULL, 890, 1729) + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row2_slot); + EXPECT_EQ(2, id); + bool col1_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(1), row2_slot); + EXPECT_TRUE(col1_is_null); + col2 = table.GetIntColInRow(txn, catalog::col_oid_t(2), row2_slot); + EXPECT_EQ(890, col2); + col3 = table.GetIntColInRow(txn, catalog::col_oid_t(3), row2_slot); + EXPECT_EQ(col3_default, col3); + + // 3rd tuple should be (3, 300, NULL, NULL) + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row3_slot); + EXPECT_EQ(3, id); + col1 = table.GetIntColInRow(txn, catalog::col_oid_t(1), row3_slot); + EXPECT_EQ(300, col1); + bool is_col2_null = table.IsNullColInRow(txn, catalog::col_oid_t(2), row3_slot); + EXPECT_TRUE(is_col2_null); + bool is_col3_null = table.IsNullColInRow(txn, catalog::col_oid_t(3), row3_slot); + EXPECT_TRUE(is_col3_null); + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; +} + +// NOLINTNEXTLINE +TEST_F(SqlTableTests, ModifyDefaultValuesTest) { + // Testing the case of adding a column without a default value + // and then setting the default value for that column. + // This should NOT retro-actively populate the default value for older tuples + SqlTableTestRW table(catalog::table_oid_t(2)); + auto txn = txn_manager_.BeginTransaction(); + + // Create a table of 2 columns, with no default values at the start. + table.DefineColumn("id", type::TypeId::INTEGER, false, catalog::col_oid_t(0)); + table.DefineColumn("col1", type::TypeId::INTEGER, false, catalog::col_oid_t(1)); + table.Create(); + + // Insert (1, 100) + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 1); + table.SetIntColInRow(catalog::col_oid_t(1), 100); + storage::TupleSlot row1_slot = table.EndInsertRow(txn); + + // Explicitly set the layout version number + table.version_ = storage::layout_version_t(1); + // Add a new column WITHOUT a default value + table.AddColumn(txn, "col2", type::TypeId::INTEGER, true, catalog::col_oid_t(2)); + + // Insert (2, 200, NULL) i.e. nothing passed in for col2 + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 2); + table.SetIntColInRow(catalog::col_oid_t(1), 200); + storage::TupleSlot row2_slot = table.EndInsertRow(txn); + + // Now set the default value of the column to something + int col2_default = 42; + auto *col2_default_bytes = reinterpret_cast(&col2_default); + table.SetColumnDefault(catalog::col_oid_t(2), col2_default_bytes); + + // Add a new column - to trigger the UpdateSchema + table.version_ = storage::layout_version_t(2); + table.AddColumn(txn, "col3", type::TypeId::INTEGER, true, catalog::col_oid_t(3)); + + // Insert (3, 300, 42, NULL) - The default value for col2 will be populated by the execution engine + table.StartInsertRow(); + table.SetIntColInRow(catalog::col_oid_t(0), 3); + table.SetIntColInRow(catalog::col_oid_t(1), 300); + table.SetIntColInRow(catalog::col_oid_t(2), col2_default); + storage::TupleSlot row3_slot = table.EndInsertRow(txn); + + // Validate the tuples + // 1st row should be (1, 100, NULL, NULL) + uint32_t id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row1_slot); + EXPECT_EQ(1, id); + uint32_t col1 = table.GetIntColInRow(txn, catalog::col_oid_t(1), row1_slot); + EXPECT_EQ(100, col1); + bool col2_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(2), row1_slot); + EXPECT_TRUE(col2_is_null); + bool col3_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(3), row1_slot); + EXPECT_TRUE(col3_is_null); + + // 2nd tuple should be (2, 200, NULL, NULL) + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row2_slot); + EXPECT_EQ(2, id); + col1 = table.GetIntColInRow(txn, catalog::col_oid_t(1), row2_slot); + EXPECT_EQ(200, col1); + col2_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(2), row2_slot); + EXPECT_TRUE(col2_is_null); + col3_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(3), row2_slot); + EXPECT_TRUE(col3_is_null); + + // 3rd tuple should be (3, 300, 42, NULL) + id = table.GetIntColInRow(txn, catalog::col_oid_t(0), row3_slot); + EXPECT_EQ(3, id); + col1 = table.GetIntColInRow(txn, catalog::col_oid_t(1), row3_slot); + EXPECT_EQ(300, col1); + uint32_t col2 = table.GetIntColInRow(txn, catalog::col_oid_t(2), row3_slot); + EXPECT_EQ(col2_default, col2); + col3_is_null = table.IsNullColInRow(txn, catalog::col_oid_t(3), row3_slot); + EXPECT_TRUE(col3_is_null); + + txn_manager_.Commit(txn, TestCallbacks::EmptyCallback, nullptr); + delete txn; } } // namespace terrier diff --git a/test/transaction/deferred_actions_test.cpp b/test/transaction/deferred_actions_test.cpp index c54e55988d..240b747708 100644 --- a/test/transaction/deferred_actions_test.cpp +++ b/test/transaction/deferred_actions_test.cpp @@ -63,7 +63,7 @@ TEST_F(DeferredActionsTest, CommitAction) { catalog::Schema schema(cols); storage::SqlTable table(&block_store, schema, catalog::table_oid_t(24)); - auto row_pair = table.InitializerForProjectedRow(col_oids); + auto row_pair = table.InitializerForProjectedRow(col_oids, storage::layout_version_t(0)); auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); @@ -82,7 +82,7 @@ TEST_F(DeferredActionsTest, CommitAction) { auto *data = reinterpret_cast(insert->AccessForceNotNull(pr_map->at(col_oid))); *data = 42; - table.Insert(txn, *insert); + table.Insert(txn, *insert, storage::layout_version_t(0)); EXPECT_FALSE(aborted); EXPECT_FALSE(committed); @@ -236,7 +236,7 @@ TEST_F(DeferredActionsTest, CommitBootstrapDefer) { catalog::Schema schema(cols); storage::SqlTable table(&block_store, schema, catalog::table_oid_t(24)); - auto row_pair = table.InitializerForProjectedRow(col_oids); + auto row_pair = table.InitializerForProjectedRow(col_oids, storage::layout_version_t(0)); auto pri = new storage::ProjectedRowInitializer(std::get<0>(row_pair)); auto pr_map = new storage::ProjectionMap(std::get<1>(row_pair)); @@ -270,7 +270,7 @@ TEST_F(DeferredActionsTest, CommitBootstrapDefer) { auto *data = reinterpret_cast(insert->AccessForceNotNull(pr_map->at(col_oid))); *data = 42; - table.Insert(txn, *insert); + table.Insert(txn, *insert, storage::layout_version_t(0)); EXPECT_FALSE(aborted); EXPECT_FALSE(committed);