[core] Introduce global index scan structure #6635
Conversation
|
+1 |
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Thanks @leaves12138 for the contribution!
Left comments.
| new DataField( | ||
| 3, | ||
| "_INDEX_META", | ||
| new VarBinaryType())))))); |
| new DataField(5, "_EXTERNAL_PATH", newStringType(true)), | ||
| new DataField( | ||
| 6, | ||
| "_GLOBAL_INDEX", |
There was a problem hiding this comment.
Extract a static field:
public static RowType GLOBAL_INDEX = XXX;
There was a problem hiding this comment.
OK, moved to GlobalIndexMeta.SCHEMA
| "Whether to write the data into fixed bucket for batch writing a postpone bucket table."); | ||
|
|
||
| public static final ConfigOption<Long> GLOBAL_INDEX_ROW_COUNT_PER_SHARD = | ||
| key("global-index.row_count_per_shard") |
| private final long fileSize; | ||
| private final long rowCount; | ||
|
|
||
| @Nullable private final Long rowRangeStart; |
There was a problem hiding this comment.
Do not provide these fields. Introduce a GlobalIndexMeta
| @Override | ||
| public InternalRow convertTo(IndexManifestEntry record) { | ||
| IndexFileMeta indexFile = record.indexFile(); | ||
| InternalRow globalIndexRow = |
There was a problem hiding this comment.
This row should be nullable.
| /** Path factory to create an index path. */ | ||
| public interface IndexPathFactory { | ||
|
|
||
| Path getPath(String fileName); |
There was a problem hiding this comment.
toPath, let this be unified to toPath(file)
| return result; | ||
| } | ||
|
|
||
| public List<IndexManifestEntry> scan(Filter<IndexManifestEntry> readTFilter) { |
There was a problem hiding this comment.
Please just provide scan with Snapshot. We should generate Snapshot in outside to avoid load Snapshot many times.
leaves12138
force-pushed
the
global_index_core
branch
from
0ef1c78 to
f6284e9
Compare
| /** Schema for global index. */ | ||
| public class GlobalIndexMeta { | ||
|
|
||
| public static final RowType SCHEMA = |
There was a problem hiding this comment.
create a static method.
RowType schema(int startFieldId)
(cherry picked from commit 657205a)
3 participants
Purpose
Related to pip:
https://cwiki.apache.org/confluence/display/PAIMON/PIP-38%3A+Introduce+Global+Index+for+Paimon+Table
Tests
API and Format
Documentation