[15721] TileGroup Compaction and GC Fixes #1349
Conversation
src/gc/tile_group_compactor.cpp
Outdated
| } // namespace gc | ||
| } // namespace peloton | ||
|
|
||
|
|
There was a problem hiding this comment.
You may not need so many blank lines here.
src/gc/recycle_stack.cpp
Outdated
|
|
||
| // Used by GC Manager to add to recycle stack (can be slower) | ||
| void RecycleStack::Push(const ItemPointer &location) { | ||
|
|
There was a problem hiding this comment.
I notice a queue was used before, so I wonder if the reason of using a stack instead of queue is it is easier to implement, since managing a stack only involves manipulate the head of the linked list?
src/gc/tile_group_compactor.cpp
Outdated
|
|
||
| constexpr auto kMinPauseTime = std::chrono::microseconds(1); | ||
| constexpr auto kMaxPauseTime = std::chrono::microseconds(100000); | ||
|
|
There was a problem hiding this comment.
I think using #define might be a better choice here.
src/gc/tile_group_compactor.cpp
Outdated
|
|
||
| LOG_TRACE("Moving Visible Tuple id : %u, Physical Tuple id : %u ", | ||
| visible_tuple_id, physical_tuple_id); | ||
|
|
There was a problem hiding this comment.
Where is this visible_tuple_id defined? I did not find it.
src/gc/tile_group_compactor.cpp
Outdated
| } | ||
|
|
||
| bool success = MoveTuplesOutOfTileGroup(table, tile_group); | ||
|
|
There was a problem hiding this comment.
I find there is only one usage of function MoveTuplesOutOfTileGroup() and it is here. It seems that the tuples in the given tile group is always moved to the tile group's table. Then I think there may not be necessary to have a data table as argument in MoveTuplesOutOfTileGroup(), since we know that this table is this tile group's table.
| ItemPointer location = recycle_stack->TryPop(); | ||
| if (location.IsNull()) { | ||
| return INVALID_ITEMPOINTER; | ||
| } | ||
|
|
There was a problem hiding this comment.
I think it may be better for this function GetRecycledTupleSlot() to have a while around TryPop() instead of having while loop around GetRecycledTupleSlot, since this function name seems it guarantees to get a tuple slot.
| } // namespace gc | ||
| } // namespace peloton | ||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
You may not need so many blank lines here.
src/gc/recycle_stack.cpp
Outdated
|
|
||
| // Used by GC Manager to add to recycle stack (can be slower) | ||
| void RecycleStack::Push(const ItemPointer &location) { | ||
|
|
src/gc/tile_group_compactor.cpp
Outdated
|
|
||
| constexpr auto kMinPauseTime = std::chrono::microseconds(1); | ||
| constexpr auto kMaxPauseTime = std::chrono::microseconds(100000); | ||
|
|
There was a problem hiding this comment.
How about moving these constants somewhere in the gc namespace? not in the function. And I think that the max_attempts should also be defined as constants.
src/gc/tile_group_compactor.cpp
Outdated
|
|
||
| size_t attempts = 0; | ||
| size_t max_attempts = 100; | ||
|
|
There was a problem hiding this comment.
I think 100 attempts is way too much, you can probably try 50.
src/gc/tile_group_compactor.cpp
Outdated
| // ensure that this is the latest version | ||
| bool is_latest_version = tile_group_header->GetPrevItemPointer(physical_tuple_id).IsNull(); | ||
| if (is_latest_version == false) { | ||
| LOG_TRACE("Skipping tuple, not latest version."); |
There was a problem hiding this comment.
Why skip the tuple here? Why not just abort the transaction?
src/gc/tile_group_compactor.cpp
Outdated
|
|
||
| LOG_TRACE("Moving Visible Tuple id : %u, Physical Tuple id : %u ", | ||
| visible_tuple_id, physical_tuple_id); | ||
|
|
| recycle_queue_map_.clear(); | ||
| immutable_tile_group_queue_ = std::make_shared<LockFreeQueue<oid_t>>(INITIAL_TG_QUEUE_LENGTH); | ||
| recycle_stacks_ = std::make_shared<peloton::CuckooMap< | ||
| oid_t, std::shared_ptr<RecycleStack>>>(INITIAL_MAP_SIZE); |
There was a problem hiding this comment.
I think recycle_stack_map is better for understanding.
| "unique_btree_index", 1235, TEST_TABLE_OID, CATALOG_DATABASE_OID, | ||
| IndexType::BWTREE, IndexConstraintType::UNIQUE, tuple_schema, | ||
| key_schema, key_attrs, unique); | ||
|
|
There was a problem hiding this comment.
What is the 1235 here? Some random number?
| /* | ||
| * @brief Set's Immutable Flag to True. Only used by the Garbage Collector | ||
| */ | ||
| inline bool SetImmutabilityWithoutNotifyingGC() { |
|
As my teammates said, Nice Job ! ! |
| @@ -154,6 +154,12 @@ bool SeqScanExecutor::DExecute() { | |||
| while (current_tile_group_offset_ < table_tile_group_count_) { | |||
| auto tile_group = | |||
| target_table_->GetTileGroup(current_tile_group_offset_++); | |||
|
|
|||
| if (tile_group == nullptr) { | |||
There was a problem hiding this comment.
I am confused that tile_group can be nullptr in this seq scan executor, but be asserted not be null in the previous index and hybrid scan executor ?
| //}; | ||
| // | ||
| //} // namespace storage | ||
| //} // namespace peloton |
There was a problem hiding this comment.
It seems that this file can be removed ?
src/codegen/inserter.cpp
Outdated
| @@ -34,6 +34,8 @@ void Inserter::Init(storage::DataTable *table, | |||
| char *Inserter::AllocateTupleStorage() { | |||
| location_ = table_->GetEmptyTupleSlot(nullptr); | |||
|
|
|||
| PELOTON_ASSERT(location_.IsNull() == false); | |||
|
|
|||
| // Get the tile offset assuming that it is in a tuple format | |||
| auto tile_group = table_->GetTileGroupById(location_.block); | |||
mbutrovich
force-pushed
the
gc_fixes
branch
4 times, most recently
from
a10a5ad
to
36d546d
Compare
mbutrovich
force-pushed
the
gc_fixes
branch
2 times, most recently
from
55c7590
to
86a27fb
Compare
mbutrovich
force-pushed
the
gc_fixes
branch
3 times, most recently
from
9c87209
to
0cdcdb3
Compare
correctness of primary and secondary indexes in the garbage collector.
… bring in tests.
…in tests, need to verify.
… bring in tests.
…in tests, need to verify.
… bring in tests.
…in tests, need to verify.
… bring in tests.
…in tests, need to verify.
… bring in tests.
…in tests, need to verify.
… bring in tests.
…in tests, need to verify.
Project: TileGroup Compaction, TileGroup Freeing, and Garbage Collection Fixes
Our project has 3 main aspects:
Status
Completed 75% 100% and 125% goals
Summary
1) TileGroup Freeing
This PR enhances the Garbage Collector to free empty TileGroups when all of their tuple slots have been recycled. We also scanned the codebase for all points (outside of codegen) that fetch tile groups from the catalog without checking for a nullptr. TileGroups can no longer be assumed to live forever, so these checks must be done. We need to follow up with Prashanth to do a similar check in the codegen engine. It also created a new class, RecycleStack to replace the Garbage Collector’s recycle queues. This concurrent data structure offers a non-blocking, constant-time TryPop() to retrieve a recycled tuple slot, a blocking, constant-time Push(), and a concurrent RemoveAllWithKey() function that uses hand-over-hand locking. The performance of TryPop() is very important because TryPop() lies on the critical path for DataTable::Insert(). We had to write a new data structure instead of using the existing concurrent queue because we need to be able to iterate through and remove recycled slots from TileGroups that become immutable. None of the existing concurrent structures support concurrent iteration, insertion, and removal.
2) TileGroup Compaction
This PR adds a class called TileGroupCompactor that performs compaction of tile groups. Compaction is triggered by the GarbageCollector, which submits a CompactTileGroup() task to the MonoQueuePool when the majority of a TileGroup is recycled garbage. The exact fraction of garbage required to trigger compaction is determined by a global setting, settings:SettingId::compaction_threshold. Compaction can also be enabled/disabled via another setting, global setting settings:SettingId::compaction_enabled.
3) Garbage Collection Fixes
This PR resolves almost all of the issues identified in issue #1325.
It includes the results of a thorough correctness audit we performed on Peleton's garbage collection system. It includes a whole new test suite for the Transaction-Level Garbage Collector and several important bug fixes to the Garbage Collector and Transaction Manager.
GC Fixes Summary:
*Modified DataTable's Insert to return the ItemPointer to the GCManager in the case of a failed insert.