Skip to content

[global-index-eslib]use thin packaging and fix pom bundle,fix query compatibility and vector metrics, and#8618

Merged
JingsongLi merged 1 commit into
apache:masterfrom
CrownChu:feature-globalindex-eslib-review-fixes
Jul 14, 2026
Merged

[global-index-eslib]use thin packaging and fix pom bundle,fix query compatibility and vector metrics, and#8618
JingsongLi merged 1 commit into
apache:masterfrom
CrownChu:feature-globalindex-eslib-review-fixes

Conversation

@CrownChu

@CrownChu CrownChu commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

This is a follow-up to #8000. It fixes persisted full-text query compatibility and vector metric handling, and changes paimon-eslib to a thin JAR so that ESLib and Lucene classes are no longer bundled into the Apache artifact.

Changes

Thin packaging

  • Remove the module-specific shading configuration that bundled ESLib and Lucene classes.
  • Keep ESLib and Lucene as regular Maven dependencies.
  • Resolve ESLib 1.0.6 artifacts from the public GitHub Maven repository.
  • Document the runtime dependency requirements in paimon-eslib/README.md.

This change only separates the artifact boundary. It does not relicense ESLib or change its dependency license.

Persisted full-text query compatibility

  • Accept the serialized column field as a routing hint when reading persisted SQL full-text predicates.
  • Continue using the resolved physical index field for query execution.
  • Add regression coverage for persisted predicates on complementary text fields.

Vector metric correctness

  • Implement the VectorGlobalIndexer metric contract.
  • Support canonical l2, cosine, and inner_product metric mappings.
  • Use the metric persisted in each index shard instead of relying on the current table configuration.
  • Reject searches across shards with inconsistent vector metrics.
  • Convert Lucene vector scores to Paimon's expected score semantics.
  • Close the reader when vector metric registration fails to avoid leaking resources.

Tests

mvn -pl paimon-eslib clean verify

Result:

Tests run: 70, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The generated paimon-eslib JAR was also verified not to contain ESLib or Lucene classes.

@CrownChu CrownChu force-pushed the feature-globalindex-eslib-review-fixes branch from ed993ed to 5be78ef Compare July 14, 2026 07:50
@CrownChu CrownChu changed the title [global-index-eslib] Fix query compatibility and vector metrics, and use thin packaging [global-index-eslib]use thin packaging and fix pom bundle,fix query compatibility and vector metrics, and Jul 14, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit a910c61 into apache:master Jul 14, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants