Support Substrait Round-Trip of EmptyRelation Including produce_one_row Semantics
#18842
+157
−48
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Transform EmptyRelation plans to produce a null-typed row as Substrait virtual tables. This change allows Substrait consumption to recognize zero-column virtual tables with one row as placeholder EmptyRelation plans, restoring produce_one_row semantics on roundtrip. Added a test for literal queries without a FROM clause to ensure correct behavior.
Add #[allow(deprecated)] for values field in Producer initializations. Maintain usage of deprecated values field due to incomplete consumer support for new expressions field. Enhance Consumer compatibility by updating produce_one_row logic to accommodate both field patterns and handling for future-proof expressions field usage.
Add detailed function-level documentation for produce_one_row, covering its purpose, usage scenarios, default values context, and rationale for deprecated fields. Improve pattern detection logic in read_rel.rs with clear explanations. Expand test coverage in roundtrip_logical_plan.rs with four new tests ensuring correct serialization/deserialization for EmptyRelation and related scenarios. Verify improvements to sqllogictest functionality.
…tualTable literals
asolimando
reviewed
Nov 20, 2025
Comment on lines
+125
to
+126
| // A VirtualTable with exactly one row containing only empty/default fields represents | ||
| // an EmptyRelation with produce_one_row=true. This pattern is used for queries without |
Member
There was a problem hiding this comment.
In Oracle the "one row (dummy) table" is called the "dual table", by checking the "In other database systems" section it seems to be quite a well-known name for that concept.
I am not suggesting to necessarily adopt this name here, but maybe a reference to it in the comments could be good for people with a DB background, but not necessarily familiar with Datafusion
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This change resolves a long‑standing gap in DataFusion’s Substrait round‑trip implementation where
EmptyRelationwithproduce_one_row=truecould not be encoded or decoded. This limitation caused ~1800 sqllogictest cases to fail, particularly those involving queries without a FROM clause (e.g.,SELECT 1).By adding full support for recognizing and producing the Substrait VirtualTable pattern representing a “phantom row,” DataFusion can now faithfully round‑trip logical plans that use empty relations to provide scalar evaluation contexts. This unblocks broader Substrait compatibility and improves consistency across logical plan conversions.
What changes are included in this PR?
Implement detection of the Substrait VirtualTable patterns representing
produce_one_rowand map them toLogicalPlan::EmptyRelation.Add
from_empty_relationencoding logic that emits a properly structured VirtualTable, including default literal values whenproduce_one_row=true.Refactor literal row conversion into a helper (
convert_literal_rows) for clarity and reuse.Improve field‑count validation for expression‑based VirtualTables.
Add comprehensive round‑trip test coverage:
Are these changes tested?
Yes. New integration tests exercise all permutations of EmptyRelation encoding and decoding, including edge cases related to schema handling and subqueries. These tests ensure round‑trip correctness and prevent regressions.
Are there any user-facing changes?
No user‑facing API changes. This PR improves Substrait interoperability internally.
LLM-generated code disclosure
This PR includes LLM‑generated code and comments. All generated content has been manually reviewed and tested.