Skip to content

Support Java-style blob field directives#490

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/datafusion-blob-directives
Jul 9, 2026
Merged

Support Java-style blob field directives#490
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/datafusion-blob-directives

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Adds Java-compatible BLOB field comment directives so SQL schemas can mark binary columns as raw BLOB, descriptor BLOB, or Blob View fields. The directive handling lives in the core schema layer, while DataFusion only passes through column comments and supports the BYTES SQL alias.

Changes

  • Parse __BLOB_FIELD, __BLOB_DESCRIPTOR_FIELD, and __BLOB_VIEW_FIELD during core schema creation and ALTER TABLE ... ADD COLUMN.
  • Normalize configured binary BLOB fields to BlobType, update blob-field / blob-descriptor-field / blob-view-field options, and strip directive prefixes from stored comments.
  • Add DataFusion SQL coverage for BYTES and Java-style BLOB directives.
  • Document BLOB column directives and their equivalent table options in docs/src/sql.md.

Testing

  • cargo test -p paimon blob_comment_directive --lib
  • cargo test -p paimon blob_field_option --lib
  • cargo test -p paimon test_blob_view_options --lib
  • cargo test -p paimon-datafusion blob_comment_directive --lib
  • cargo test -p paimon-datafusion test_create_table_blob --lib
  • cargo test -p paimon-datafusion test_alter_table_add_blob --lib
  • git diff --check

@JingsongLi JingsongLi force-pushed the codex/datafusion-blob-directives branch from 573c68f to 9aa4b4b Compare July 9, 2026 04:10
Comment thread crates/paimon/src/spec/schema.rs Outdated
BLOB_FIELD_DIRECTIVE => BLOB_FIELD_OPTION,
BLOB_DESCRIPTOR_FIELD_DIRECTIVE => BLOB_DESCRIPTOR_FIELD_OPTION,
BLOB_VIEW_FIELD_DIRECTIVE => BLOB_VIEW_FIELD_OPTION,
_ => return None,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not silently fall back to a normal comment when the column comment starts with __BLOB but does not match one of the supported directives.

Java Paimon rejects unknown blob directives in this case, e.g. __BLOB_UNKNOWN, so a typo like __BLOB_FIEL; picture should fail fast instead of creating a plain binary column. Otherwise the user gets a non-BLOB schema with no blob-field option and only discovers the issue later.

Could we make this parser return an error for unknown __BLOB* directives and add a create/add-column test for that case?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add test_blob_comment_directive_rejects_unknown_create_directive.

@JingsongLi JingsongLi force-pushed the codex/datafusion-blob-directives branch 2 times, most recently from 1f65ee5 to 93a5e09 Compare July 9, 2026 06:36

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small Java-compatibility gap remains in the directive parser.

Java ColumnDirectiveUtils.matchDirective accepts only a marker by itself or a marker immediately followed by ;. For example, __BLOB_FIELD; comment is valid, but __BLOB_FIELD ; comment is rejected because the character after __BLOB_FIELD is a space, not ;.

The Rust parser currently does splitn(2, ';') and then trims the directive part, so COMMENT '__BLOB_FIELD ; payload bytes' is accepted, stripped, and converted into a BLOB column. I verified this locally with a temporary regression test: Schema::builder() returned Ok with blob-field=payload, whereas Java would throw an unsupported directive error.

Could we align the matching with Java by trimming the whole comment first and accepting only marker or marker;... (no whitespace between the marker and semicolon), and add a regression test for this malformed directive?

@JingsongLi JingsongLi force-pushed the codex/datafusion-blob-directives branch from 93a5e09 to e10cfa9 Compare July 9, 2026 07:16

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The Java-style BLOB directive parsing and validation issues from the previous review are addressed.

@JingsongLi JingsongLi merged commit 75ffe70 into apache:main Jul 9, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants