Skip to content

feat(core): add MaskExpression POJO and ReadRel projection support#782

Merged
nielspardon merged 17 commits intosubstrait-io:mainfrom
flex-team:feat/add-mask-expression
Apr 10, 2026
Merged

feat(core): add MaskExpression POJO and ReadRel projection support#782
nielspardon merged 17 commits intosubstrait-io:mainfrom
flex-team:feat/add-mask-expression

Conversation

@flex-seongmin
Copy link
Copy Markdown
Contributor

Summary

Implement the MaskExpression POJO layer for ReadRel projection support in substrait-java.

The Substrait protobuf spec defines Expression.MaskExpression as a field on ReadRel (field 4: projection), but the Java POJO layer had only a TODO comment and no actual implementation. This means any ReadRel projection information was silently dropped during proto-to-POJO conversion, and POJO-constructed plans could not express column projection at the read level.

This change closes that gap by introducing a full POJO model for MaskExpression and wiring it through the existing conversion infrastructure.

Changes

  • MaskExpression.java (new) -- @Value.Enclosing interface with 11 @Value.Immutable inner types (MaskExpr, Select, StructSelect, StructItem, ListSelect, ListSelectItem, ListElement, ListSlice, MapSelect, MapKey, MapKeyExpression) matching the proto definition 1:1
  • MaskExpressionProtoConverter.java (new) -- Dedicated bidirectional converter between io.substrait.proto.Expression.MaskExpression and the POJO types, keeping proto FQN references isolated from the relation converters
  • AbstractReadRel.java -- Replaced TODO comment with Optional<MaskExpression.MaskExpr> getProjection()
  • ProtoRelConverter.java -- Added optionalMaskExpression() helper; all four ReadRel subtypes (NamedScan, LocalFiles, VirtualTableScan, ExtensionTable) now populate projection during proto-to-POJO conversion
  • RelProtoConverter.java -- All four visit(ReadRel) methods now serialize projection back to proto via MaskExpressionProtoConverter.toProto()

Testing

Added 11 roundtrip tests in ReadRelRoundtripTest covering:

  • Simple flat column projection (StructSelect only)
  • Nested struct projection (StructItem with child Select.ofStruct)
  • List element and slice selection (ListSelect)
  • List selection with nested child (ListSelect + child StructSelect)
  • Map key selection (MapSelect with MapKey)
  • Map key expression / wildcard selection (MapKeyExpression)
  • Map selection with nested child (MapSelect + child StructSelect)
  • maintainSingularStruct flag
  • Projection combined with filter
  • Projection on VirtualTableScan and LocalFiles

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Mar 27, 2026

CLA assistant check
All committers have signed the CLA.

@flex-seongmin flex-seongmin marked this pull request as ready for review April 1, 2026 02:46
@nielspardon
Copy link
Copy Markdown
Member

Hi @flex-seongmin, thanks for your contribution. I enabled the Github Action workflow and it looks like your PR has some formatting issues. Running ./gradlew spotlessApply should automatically fix these for you.

Copy link
Copy Markdown
Member

@nielspardon nielspardon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some initial structural suggestions for the code from my side

Copy link
Copy Markdown
Contributor Author

@flex-seongmin flex-seongmin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for kind direction for sync code convention and architectural consistency.

I added 3 commits.

Copy link
Copy Markdown
Contributor Author

@flex-seongmin flex-seongmin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I committed that you reviewed! Thanks for review.

  • Added javadoc for class / public functions
  • Use MaskExpressionVisitor for Select dispatc.

@nielspardon
Copy link
Copy Markdown
Member

if you can fix the javadoc and the minor renaming I think this is looking good

@Value.Immutable
public interface MaskExpression {

/** The top-level struct selection describing which fields to include. */
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also missing a @return

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/** The top-level struct selection describing which fields to include. */
/**
* The top-level struct selection describing which fields to include.
*
* @return the top-level struct selection
*/

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I apologize that I didn't check it carefully. Adjusted It

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no worries. I seem to have given you a code suggestion that isn't formatted properly by accident and you would need to run spotlessApply to fix the indendation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nielspardon I executed spotlessApply task for lint

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

Co-authored-by: Niels Pardon <mail@niels-pardon.de>
Copy link
Copy Markdown
Member

@nielspardon nielspardon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you

@nielspardon nielspardon merged commit 6f79ac8 into substrait-io:main Apr 10, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants