[core] Maintain primary-key full-text index archives#8651
Conversation
5fe12ce to
a82e386
Compare
leaves12138
left a comment
There was a problem hiding this comment.
I found one schema-validation gap in the new full-text index configuration.
| "%s entry '%s' must reference a CHAR/VARCHAR/STRING column.", | ||
| CoreOptions.PK_FULL_TEXT_INDEX_COLUMNS.key(), | ||
| indexColumn); | ||
| } |
There was a problem hiding this comment.
Could we resolve options.primaryKeyFullTextIndexOptions(indexColumn) during schema validation and add tests for malformed JSON and conflicting global/field-scoped options? Currently fields.<column>.pk-full-text.index.options = "{not-json" passes validateTableSchema, but later fails when PrimaryKeyIndexDefinitions.create initializes the writer. This lets an invalid table definition be committed and defers the error until write setup.
a82e386 to
2d5590f
Compare
leaves12138
left a comment
There was a problem hiding this comment.
LGTM. The schema-validation issue is fixed and covered by tests. I also reviewed the full-text maintenance, compaction, and rollback paths; the focused test suite passes locally.
Summary
Build on #8649 by adding LSM-style incremental maintenance for primary-key full-text index archives.
Changes
BucketedPrimaryKeyIndexMaintainerPrimaryKeyIndexLevelsand the Vector/Sorted compaction options to consolidate similarly sized archives and rebuild stale coverageScope
index.typeoption; full text remains the implicit/default index typeTesting
mvn -pl paimon-core -am -DskipTests packagemvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest='PkFullTextIndexFileTest,PkFullTextBucketIndexStateTest,BucketedFullTextIndexMaintainerTest,BucketedPrimaryKeyIndexMaintainerTest,PrimaryKeyIndexDefinitionsTest,PrimaryKeyIndexWriteTest,PrimaryKeyFullTextIndexValidationTest,SchemaManagerTest' testgit diff --check origin/master..HEADThe focused test set runs 86 tests.