Skip to content

[fix](fe) Prevent cast project pushdown through union distinct #64080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
morrySnow wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[fix](fe) Prevent cast project pushdown through union distinct #64080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f9d17b5
[fix](fe) Prevent cast project pushdown through union distinct
morrySnow Jun 3, 2026
0b76718
[fix](fe) Refine safe cast project pushdown
morrySnow Jun 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushProjectThroughUnion.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
Expand Down Expand Up @@ -60,7 +61,12 @@ public static boolean canPushProject(List<NamedExpression> projects, LogicalSetO
if (e instanceof SlotReference) {
return true;
} else {
Expression expr = ExpressionUtils.getExpressionCoveredByCast(e.child(0));
Expression expr;
if (logicalSetOperation.getQualifier().equals(Qualifier.ALL)) {
expr = ExpressionUtils.getExpressionCoveredByCast(e.child(0));
} else {
expr = ExpressionUtils.getExpressionCoveredBySafetyCast(e.child(0));
}
return expr instanceof SlotReference;
}
});
Expand Down
9 changes: 9 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/ArrayType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.doris.nereids.types;

import org.apache.doris.catalog.Type;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.ComplexDataType;

import java.util.Objects;
Expand Down Expand Up @@ -57,6 +58,14 @@ public DataType getItemType() {
return itemType;
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof ArrayType) {
return itemType.canSafetyCastTo(((ArrayType) target).itemType);
}
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

canSafetyCastTo is now used to decide whether UNION DISTINCT duplicate elimination may be moved after the cast, so true here must mean the cast is injective. Complex-to-character casts are not always injective: for arrays/maps/structs containing FLOAT or DOUBLE, direct float/double-to-string is intentionally not marked safe, but complex stringification delegates to the nested serde (for arrays, DataTypeArraySerDe::to_string calls nested_serde->to_string) and BE formats float/double with only digits10 + 1 significant digits in cast_to_string.h. Two distinct arrays whose float elements differ only beyond that formatted precision can therefore stringify to the same value; after this rewrite the pushed cast would make UNION DISTINCT collapse a row that the original plan kept. The same unconditional target instanceof CharacterType pattern in MapType and StructType has the same issue. Please either leave complex-to-character casts unsafe or require the nested types themselves to be injective when cast to the character target.

return target instanceof CharacterType;
Comment thread
morrySnow marked this conversation as resolved.
}

@Override
public Type toCatalogDataType() {
// Catalog ArrayType defaults containsNull to true via single-arg constructor
Expand Down
8 changes: 8 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/BooleanType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,14 @@ public class BooleanType extends PrimitiveType {
private BooleanType() {
}

@Override
public boolean canSafetyCastTo(DataType target) {
return target instanceof BooleanType || target.isIntegralType() || target.isFloatLikeType()
|| (target instanceof DecimalV2Type && ((DecimalV2Type) target).getRange() >= 1)
|| (target instanceof DecimalV3Type && ((DecimalV3Type) target).getRange() >= 1)
|| target.isStringLikeType();
}

@Override
public Type toCatalogDataType() {
return Type.BOOLEAN;
Expand Down
4 changes: 4 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -813,6 +813,10 @@ public List<DataType> getAllPromotions() {

public abstract int width();

public boolean canSafetyCastTo(DataType target) {
return this.equals(target);
}

public static List<DataType> trivialTypes() {
return Type.getTrivialTypes()
.stream()
Expand Down
9 changes: 9 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/DateTimeType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

import org.apache.doris.catalog.Type;
import org.apache.doris.common.Config;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.DateLikeType;

import java.time.DateTimeException;
Expand All @@ -45,6 +46,14 @@ private DateTimeType(boolean shouldConversion) {
this.shouldConversion = shouldConversion;
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof DateTimeType || target instanceof DateTimeV2Type || target instanceof CharacterType) {
return true;
}
return false;
}

@Override
public DataType conversion() {
if (Config.enable_date_conversion && shouldConversion) {
Expand Down
13 changes: 13 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/DateTimeV2Type.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.apache.doris.nereids.trees.expressions.literal.DateTimeLiteral;
import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
import org.apache.doris.nereids.trees.expressions.literal.format.DateTimeChecker;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.DateLikeType;
import org.apache.doris.nereids.types.coercion.IntegralType;
import org.apache.doris.nereids.types.coercion.ScaleTimeType;
Expand Down Expand Up @@ -128,6 +129,18 @@ public String toSql() {
return super.toSql() + "(" + scale + ")";
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof DateTimeV2Type) {
DateTimeV2Type t2 = (DateTimeV2Type) target;
return this.scale <= t2.scale;
}
if (target instanceof DateTimeType) {
return this.scale == 0;
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return ScalarType.createDatetimeV2Type(scale);
Expand Down
6 changes: 6 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/DateType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

import org.apache.doris.catalog.Type;
import org.apache.doris.common.Config;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.DateLikeType;

import java.time.DateTimeException;
Expand All @@ -45,6 +46,11 @@ private DateType(boolean shouldConversion) {
this.shouldConversion = shouldConversion;
}

@Override
public boolean canSafetyCastTo(DataType target) {
return target instanceof DateType || target instanceof DateV2Type || target instanceof CharacterType;
}

@Override
public DataType conversion() {
if (Config.enable_date_conversion && shouldConversion) {
Expand Down
14 changes: 14 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV2Type.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.Config;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.FractionalType;

import com.google.common.base.Preconditions;
Expand Down Expand Up @@ -159,6 +160,19 @@ private static DecimalV2Type widerDecimalV2Type(
return DecimalV2Type.createDecimalV2Type(range + scale, scale);
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof DecimalV2Type) {
DecimalV2Type decimalV2Type = (DecimalV2Type) target;
return decimalV2Type.getRange() >= this.getRange() && decimalV2Type.getScale() >= this.getScale();
}
if (target instanceof DecimalV3Type) {
DecimalV3Type decimalV3Type = (DecimalV3Type) target;
return decimalV3Type.getRange() >= this.getRange() && decimalV3Type.getScale() >= this.getScale();
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return ScalarType.createDecimalType(PrimitiveType.DECIMALV2, precision, scale);
Expand Down
14 changes: 14 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalV3Type.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import org.apache.doris.nereids.annotation.Developing;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.exceptions.NotSupportedException;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.FractionalType;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.SessionVariable;
Expand Down Expand Up @@ -214,6 +215,19 @@ private static DataType widerDecimalV3Type(
}
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof DecimalV2Type) {
DecimalV2Type decimalV2Type = (DecimalV2Type) target;
return decimalV2Type.getRange() >= this.getRange() && decimalV2Type.getScale() >= this.getScale();
}
if (target instanceof DecimalV3Type) {
DecimalV3Type decimalV3Type = (DecimalV3Type) target;
return decimalV3Type.getRange() >= this.getRange() && decimalV3Type.getScale() >= this.getScale();
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return ScalarType.createDecimalV3Type(precision, scale);
Expand Down
6 changes: 6 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/JsonType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

import org.apache.doris.catalog.Type;
import org.apache.doris.nereids.annotation.Developing;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.PrimitiveType;

/**
Expand All @@ -34,6 +35,11 @@ public class JsonType extends PrimitiveType {
private JsonType() {
}

@Override
public boolean canSafetyCastTo(DataType target) {
return target instanceof JsonType || target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return Type.JSONB;
Expand Down
10 changes: 10 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/MapType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

import org.apache.doris.catalog.Type;
import org.apache.doris.nereids.annotation.Developing;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.ComplexDataType;

import java.util.Objects;
Expand Down Expand Up @@ -63,6 +64,15 @@ public DataType conversion() {
return MapType.of(keyType.conversion(), valueType.conversion());
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof MapType) {
MapType mapType = (MapType) target;
return keyType.canSafetyCastTo(mapType.keyType) && valueType.canSafetyCastTo(mapType.valueType);
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return new org.apache.doris.catalog.MapType(keyType.toCatalogDataType(), valueType.toCatalogDataType());
Expand Down
18 changes: 18 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import org.apache.doris.catalog.Type;
import org.apache.doris.nereids.annotation.Developing;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.ComplexDataType;

import com.google.common.collect.ImmutableList;
Expand Down Expand Up @@ -84,6 +85,23 @@ public DataType conversion() {
return new StructType(fields.stream().map(StructField::conversion).collect(Collectors.toList()));
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof StructType) {
StructType structType = (StructType) target;
if (this.fields.size() != structType.fields.size()) {
return false;
}
for (int i = 0; i < fields.size(); i++) {
if (!this.fields.get(i).getDataType().canSafetyCastTo(structType.fields.get(i).getDataType())) {
return false;
}
}
return true;
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return new org.apache.doris.catalog.StructType(fields.stream()
Expand Down
10 changes: 10 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/TimeStampTzType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import org.apache.doris.catalog.Type;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.DateLikeType;
import org.apache.doris.nereids.types.coercion.ScaleTimeType;

Expand All @@ -46,6 +47,15 @@ private TimeStampTzType(int scale) {
this.scale = scale;
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof TimeStampTzType) {
TimeStampTzType timeStampTzType = (TimeStampTzType) target;
return timeStampTzType.getScale() >= this.scale;
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return ScalarType.createTimeStampTzType(scale);
Expand Down
10 changes: 10 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/TimeV2Type.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TimeV2Literal;
import org.apache.doris.nereids.types.coercion.CharacterType;
import org.apache.doris.nereids.types.coercion.IntegralType;
import org.apache.doris.nereids.types.coercion.PrimitiveType;
import org.apache.doris.nereids.types.coercion.RangeScalable;
Expand All @@ -48,6 +49,15 @@ private TimeV2Type() {
scale = 0;
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof TimeV2Type) {
TimeV2Type timeV2Type = (TimeV2Type) target;
return timeV2Type.scale >= scale;
}
return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
return ScalarType.createTimeV2Type(scale);
Expand Down
5 changes: 5 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/VariantType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,11 @@ public VariantType(List<VariantField> fields, int variantMaxSubcolumnsCount,
this.enableNestedGroup = enableNestedGroup;
}

@Override
public boolean canSafetyCastTo(DataType target) {
return target.equals(this) || target instanceof VariantType;
}

@Override
public DataType conversion() {
return new VariantType(predefinedFields.stream().map(VariantField::conversion)
Expand Down
5 changes: 5 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ public int getLen() {
return len;
}

@Override
Copy link
Copy Markdown
Contributor

@englefly englefly Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

char[100] cast 成 char[10] 应该不能提升到 distinct 之上

Copy link
Copy Markdown
Contributor Author

@morrySnow morrySnow Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be现在不处理这个，所以如果还有这种cast，他是安全的。cast之后也是100个字符

public boolean canSafetyCastTo(DataType target) {
Copy link
Copy Markdown
Contributor

@englefly englefly Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

canSafetyCastTo 的目的是: 把 cast 看成函数,要求这个是单射,对吗
如果是这样,建议改个名字 isInjectiveCastTo

Copy link
Copy Markdown
Contributor Author

@morrySnow morrySnow Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

好的

return target instanceof CharacterType;
}

@Override
public Type toCatalogDataType() {
throw new RuntimeException("CharacterType is only used for implicit cast.");
Expand Down
15 changes: 15 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/IntegralType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

import org.apache.doris.nereids.types.BigIntType;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.DecimalV3Type;
import org.apache.doris.nereids.types.LargeIntType;

import org.apache.commons.lang3.NotImplementedException;

Expand All @@ -44,6 +46,19 @@ public String simpleString() {
return "integral";
}

@Override
public boolean canSafetyCastTo(DataType target) {
if (target instanceof IntegralType) {
return this.equals(target) || ((IntegralType) target).widerThan(this);
}
if (target instanceof DecimalV3Type && !(this instanceof LargeIntType)) {
DecimalV3Type other = (DecimalV3Type) target;
DecimalV3Type self = DecimalV3Type.forType(this);
return other.getRange() >= self.getRange();
}
return target instanceof CharacterType;
}

public boolean widerThan(IntegralType other) {
return this.width() > other.width();
}
Expand Down
14 changes: 14 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1133,6 +1133,20 @@ public static Expression getExpressionCoveredByCast(Expression expression) {
return expression;
}

/**
* Strip only casts that preserve distinctness of the child expression.
*/
public static Expression getExpressionCoveredBySafetyCast(Expression expression) {
while (expression instanceof Cast) {
if (((Cast) expression).child().getDataType().canSafetyCastTo(expression.getDataType())) {
expression = ((Cast) expression).child();
} else {
break;
}
}
return expression;
}

/**
* the expressions can be used as runtime filter targets
*/
Expand Down
Loading
Loading