feat: gc for zombie piece & metaTask & bucket migration

[jingjunLi]

Description

• GCZombiePiece
  GCZombiePieceTask is an abstract interface to record the information for collecting the piece store space by deleting zombie pieces data that dues to any exception, the piece data meta is not on chain but the pieces have been stored in piece store.

• GCMeta
  GCMetaTask is an abstract interface to record the information for collecting the SP meta store space by deleting the expired data.

• GC for Bucket Migration
  When bucket migration is completed or has failed, we need to delete redundant data to free up space:

  1. If the bucket migration is successful, we delete the data from the source node.
  2. If the bucket migration fails, we delete the data from the dest node

Implementation of the GCZombie

• New Configurations
  • GCConfig::EnableGCZombie: Enables or disables the GCZombie feature.
  • GCConfig::GCZombiePieceTimeInterval: Time interval for generating GCZombie tasks (default: DefaultGlobalBatchGcZombiePieceTimeInterval - 10*60 seconds).
  • ParallelConfig::GlobalGCZombieParallel: Maximum allowed parallel GCZombie tasks (default: 1).
  • GCZombieSafeObjectIDDistance: A reserve of object IDs during GCZombie deletion. If the scanned object ID range plus DefaultGlobalGcZombieSafeObjectIDDistance exceeds the current maximum system ID, the scan resumes from 0.
  • GCZombiePieceObjectIDInterval: Interval between generated object IDs for each GCZombie task. For example, task 1 might handle IDs 0-100, task 2 101-200, and so on (default: DefaultGlobalGcZombiePieceObjectIDInterval - 100).

Core Processes

• Generation of GCZombiePieceTask: Triggered by gcZombiePieceTicker resulting in the creation of GCZombiePieceTask.
• Processing of GCZombiePieceTask: ExecuteModular::HandleGCZombiePieceTask.
  • gcZombiePieceFromIntegrityMeta: Determines whether a piece is a ZombiePiece based on the IntegrityMeta table. Scans all IntegrityMeta within the current object ID range specified in GCZombiePieceTask (StartObjectId, EndObjectId).
  • gcZombiePieceFromPieceHash: Determines whether a piece is a ZombiePiece based on the PieceHash table. Scans all PieceHash within the current object ID range specified in GCZombiePieceTask (StartObjectId, EndObjectId).
• Report Handling of GCZombiePieceTask: ManageModular::HandleGCZombiePieceTask.
  • If the task is successful, it is removed from the gcZombieQueue.
  • If the task fails, it is reinserted into the queue. If the retry limit is exceeded, the task is canceled.

Implementation of the Meta GC

Mainly for the GC (Garbage Collection) of different GC meta, currently periodically deleting expired data from two tables: bucketTraffic and readRecord.

New Configurations:

• Expiration Time and Interval Configuration:
  • bucketTrafficKeepLatestDay: Configured in ExecutorConfig::BucketTrafficKeepTimeDay, defaults to DefaultExecutorBucketTrafficKeepTimeDay (retaining the latest 180 days).
  • readRecordKeepLatestDay: Configured in ExecutorConfig::ReadRecordKeepTimeDay, defaults to DefaultExecutorReadRecordKeepTimeDay (retaining the latest 30 days).
  • DefaultGlobalGCMetaTimeInterval: Configures the time interval for the generation of GC Meta Tasks, set to 10 * 60 (equivalent to 10 minutes).
  • ReadRecordDeleteLimit: Maximum number of records to be deleted from the read record table in each gc meta task. (default 100)
• Functional Configuration:
  • GCConfig::EnableGCMeta: Controls whether GCMeta is enabled or not.
  • SQLDBConfig::EnableTracePutEvent: Controls whether Trace Put Event is enabled or not.

Core Process:

1. Generation of GCMetaTask:
   Triggered by gcMetaTicker, resulting in the creation of GCMetaTask.
2. Processing of GCMetaTask:
   Handled in ExecuteModular::HandleGCMetaTask
   • gcMetaBucketTraffic: Deletes entries from BucketTraffic using SpDBImpl::DeleteAllBucketTrafficExpired for expired BucketTrafficTable.
   • gcMetaReadRecord: Deletes entries from ReadRecord using SpDBImpl::DeleteAllReadRecordExpired for expired ReadRecord table.

Implementation of the BucketMigration GC

Purpose:

1. After successful bucket migration, delete data from the source node.
2. In case of failed or canceled bucket migration, delete migrated data from the destination node.

New Configurations:

• GCBucketMigrationTask-related settings and parameters:
  • Two new parameters:
    • gcBucketMigrationTimeout (MinGCBucketMigrationTime 0.5-1 hour): Timeout duration for the task.
    • gcBucketMigrationRetry (MinGCBucketMigrationRetry 3-5 times): Retry attempts for the task.
  • Task Type: GfSpGCBucketMigrationTask.
  • Task scheduling priority: TypeTaskGCZombiePiece. Adjusted priority:
    • TypeTaskGCMeta → DefaultSmallerPriority / 4
    • TypeTaskGCBucketMigration → DefaultSmallerPriority / 4

Core Workflow:

• Generation of GCBucketMigrationTask:
  • For successful migration (source node deletes old data):
    1. Destination SP notifies source SP of migration completion: GfSpClient::NotifyPostMigrateBucket → GfSpNotifyPostMigrate → ManageModular::NotifyPostMigrateBucket.
    2. Upon successful migration, ManageModular::GenerateGCBucketMigrationTask generates a GCBucketMigrationTask.
    3. Enqueues the task in ManageModular::gcBucketMigrationQueue.
  • For failed migration (destination node deletes migrated data):
    • BucketMigrateScheduler::PostMigrateBucket generates a GCBucketMigrationTask on the destination node if migration fails.
• Execution of GCBucketMigrationTask:
  • ExecuteModular::HandleGCBucketMigrationBucket.
  • Iterates through all GVGs of the bucket and retrieves objects within each GVG using the ListObjectsByGVGAndBucketForGC interface.
  • Checks if the location information of each object is valid. If a segment should not be on this SP, it is cleaned.
• Report Handling of GCBucketMigrationTask:
  • If the task is successful, it is removed from gcBucketMigrationQueue.
  • If the task fails, it is reinserted into the queue. If the retry limit is exceeded, the task is canceled.

Changes

Notable changes:

• add each change in a bullet point here
• ...

[will-2012]

TODO.

[jingjunLi]

Add a TODO here, which will be solved in the next PR.
In the event of a failure or crash of the BucketMigrationBucket, it will lead to the inability to proceed with data GC tasks. This will be addressed in the future by reusing the state of the bucketMigrateTable to drive the process and record the GC status.

[sysvm]

LGTM

[ruojunm]

maybe we don't have to distinguish private or public objects here, as SP need gc for all zombie pieces.

[jingjunLi]

Yes, I will remove the handling logic related to includePrivate=false.

[ruojunm]

LGTM