Bug Report Support Upsert deletion for TTL: construct queryableDocIds when adding segments out of TTL #14885
Description
Title
Bugfix: Construct queryableDocIds When Adding Segments Out of TTL in Upsert Tables
Repository Version
release-1.0.0
Key Words / Search Terms
upsert, TTL, queryableDocIds, segment addition, deletion, upsertConfig, deleteRecordColumn
Description
In Apache Pinot, upsert tables support both soft-deletes via a designated boolean deleteRecordColumn and a time-to-live (TTL) mechanism to manage the lifecycle of primary keys. Previously, when adding segments that fall outside the defined TTL, the system did not construct the queryableDocIds, leading to potential inconsistencies in query results. This bugfix addresses the issue by ensuring that queryableDocIds are properly constructed when segments beyond the TTL are added, thereby maintaining accurate and consistent query responses.
Expected Behavior
- When segments that exceed the defined TTL are added to an upsert table, the system should construct the
queryableDocIdsto reflect the current state of data accurately. - Queries executed against the upsert table should return results consistent with the applied deletions and TTL configurations.
Current Behavior
- Segments added beyond the TTL do not trigger the construction of
queryableDocIds, resulting in potential discrepancies in query outcomes. - Soft-deleted records may still appear in query results due to the absence of updated
queryableDocIds.
Use Cases / Motivating Examples (for Feature Requests)
- Ensuring that upsert tables with both deletion and TTL configurations maintain data integrity and provide accurate query results.
- Preventing scenarios where outdated or soft-deleted records are inadvertently included in query responses due to missing
queryableDocIds. - Enhancing the reliability of upsert tables in handling complex data lifecycle management involving both deletions and TTL policies.