Optimization of the Unindexed Files [Staging Area]

[osopardo1]

Qbeast Spark supports reading files not indexed with Qbeast Metadata. There's different situations that can cause a table to have a hybrid state.

• Different set of writers. One writes with Qbeast, the others with Delta. All writers can commit new files to the transaction log. Those files written as Delta will not contain any Qbeast Metadata.
• Old Table in Delta or Parquet converted to Qbeast. When we execute the Convert To Qbeast command, we are just adding a single metadata commit to the table, without rewriting or analyzing any of the existing files.
• Deletes and Updates. If a Table receives an Update or Delete operation and uses the default Copy on Write strategy, it will create new files that are not indexed.

The current behavior is to ignore the non-indexed files when reading and writing, thus disabling part of the Sampling capabilities and reducing the precision when estimating the index. Also, optimization of this "staging area", does not select that subset of files for any rearrangement operation.

This issue is to record and analyze which is the best storyline to follow when Optimizing the Non-Indexed files.

[fpj]

@osopardo1 Are these unindexed files the result of a transaction that did not use Qbeast to append?

[osopardo1]

I will update the issue with a better problem formulation.
We still need to analyze which approaches we want to take, since Compaction, for example, is no longer used, and different techniques such as Index Without Rewrite were designed since this issue was open.

[fpj]

Let's use "unindexed files", the term unindexed exists in English: https://en.wiktionary.org/wiki/unindexed#English

[osopardo1]

Oh, at first it sounded weird to me. agree with Unindexed

[osopardo1]

Opened PR: #440