Skip to content

[GLUTEN-9106][VL] Add support for staticInvoke CharVarcharCodegenUtils. #9107#9107

Merged
rui-mo merged 6 commits intoapache:mainfrom
Yifeng-Wang:add-staticinv-charvarcharcodegenutils
Oct 24, 2025
Merged

[GLUTEN-9106][VL] Add support for staticInvoke CharVarcharCodegenUtils. #9107#9107
rui-mo merged 6 commits intoapache:mainfrom
Yifeng-Wang:add-staticinv-charvarcharcodegenutils

Conversation

@Yifeng-Wang
Copy link
Contributor

What changes were proposed in this pull request?

To add support for static function callls in Spark's CharVarcharCodegenUtils.java, we add expr matching pattern in Gluten ExpressionConverter.

(Fixes: #9106)

How was this patch tested?

Added test cases in Gluten's ScalarFunctionsValidateSuite.
Built Gluten Velox with Spark 3.4 (Spark3.4.4 in pom.xml default) and test sqls.

for test_varcharWriteSideCheck:

create table srcvarchar(id string) stored as parquet;
create table tgt_vanilla(id varchar(3)) stored as parquet;
create table tgt(id varchar(3)) stored as parquet;

create table srcchar(id string) stored as parquet;
create table tgt_vanilla(id char(3)) stored as parquet;
create table tgt(id char(3)) stored as parquet;

insert into srcvarchar values ('a');
insert into srcvarchar values ('ab');
insert into srcvarchar values ('t  '); -- trail space but should retain
insert into srcvarchar values ('tra '); -- trail space but should trim
insert into srcvarchar values (''); -- trailing space but should retain
insert into srcvarchar values (''); -- trailing space but should trim to 3
insert into srcvarchar values ('中文中国'); -- will fail

insert into tgt select id from srcvarchar

for charWriteSideCheck:

insert into srcchar values ('a');
insert into srcchar values ('ab');
insert into srcchar values ('');
insert into srcchar values ('');
insert into tgt_vanilla select id from srcchar where id='abcd';

check plan to validate such operations offload to native and do not cause operator fallback:
3668dcde999255823c0bfcfc9f55970

@github-actions github-actions bot added CORE works for Gluten Core VELOX labels Mar 24, 2025
@github-actions
Copy link

#9106

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@Yifeng-Wang
Copy link
Contributor Author

@zzcclp @jinchengchenghh Hi, dear Zhichao and Jincheng, could you please review this PR? It addresses issue #9106 .
We are now creating a relative issue & PR at velox side, and we look forward to your feedbacks as well.
Thanks in advance, best regards,

child.map(replaceWithExpressionTransformer0(_, attributeSeq, expressionsMap)),
i)
}
} else if (objectName.endsWith("CharVarcharCodegenUtils")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please extract the logic for case i: StaticInvoke => to a new function, here will be much code

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jinchengchenghh Hi Jincheng, I now extracted the StaticInvoke branch to a new replaceStaticInvokeWithExpressionTransformer function.
Please take a look when you have time, and let me know if there are any further improvements needed. I'm happy to make additional changes if necessary.

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
val objectName = i.staticObject.getName.stripSuffix("$")

if (objectName.endsWith("UrlCodec")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use this format to simplify the code, and easy to implement others

val staticInvokeMap = Map("UrlCodec"-> Map("decode" -> ExpressionNames.URL_DECODE, "encode"-> ExpressionNames.URL_ENCODE ))
 val transformer = staticInvokeMap.filter(o => objectName.endsWith(o._1))
if (transformer.isEmpty) {
      throw
    }

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Mar 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jinchengchenghh . Here I notice there's a diff between how "UrlCodec" and "CharVarcharCodegenUtils" handles StaticInvoke's arguments.
Turns out that, "UrlCodec" only uses 1st arg val child = i.arguments.head which is the column identifier but had the 2nd argument "UTF-8" ignored. while CharVarcharCodegenUtils takes all the params (i.e. col identifier & length limit) by val children = i.arguments.
So I'm a bit unsure how we could better handle such discrepancy here since additional args handling logic would make it complex again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most of the cases we need all the argument, UrlCodec is a special case, maybe because the backend only supports "UTF-8" , we should throw exception if 2nd argument is not "UTF-8" but not.

So other common cases should use the staticInvokeMap, that can also help developer figure out the special case.

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jinchengchenghh using a staticInvokeMap in the latest fix:

  • a staticInvokeMap to define the supported objects and their functions and for future additions.
  • special handling for UrlCodec which only uses the 1st argument
    Could you please review the refactor and let me know if further adjustment is needed. thks

final val UDF_PLACEHOLDER = "udf_placeholder"
final val UDAF_PLACEHOLDER = "udaf_placeholder"

// Spark Catalyst util functions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark StaticInvoke Catalyst util functions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

@jinchengchenghh
Copy link
Contributor

And I think the static invokes functions may not support in CH backend, so only support in velox backend

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Mar 24, 2025

And I think the static invokes functions may not support in CH backend, so only support in velox backend

We haven't explored CH backend much, so far. But base on my knowledge if CH does not support, maybe similarly, a native validation would fail and thus fallback on such calls, but further investigation of code is needed.
We are currently only focusing on Velox backend and we've opened an issue/PR here as well. So if I'm getting you right, yes, we only support this feature in Velox backend. @jinchengchenghh

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@jinchengchenghh
Copy link
Contributor

Yes, you don't need to support it, just make sure it is fallback in CH backend.
The CH CI user and password is gluten/hN2xX3uQ4m, please resolve it.
Like the ExpressionMapping function extraExpressionMappings for each backend, we may need the extraStaticInvokeMappings, I suppose the failed CH CI is cause by this, native validation does not fallback.

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Mar 25, 2025

Yes, you don't need to support it, just make sure it is fallback in CH backend. The CH CI user and password is gluten/XXXXXXX, please resolve it. Like the ExpressionMapping function extraExpressionMappings for each backend, we may need the extraStaticInvokeMappings, I suppose the failed CH CI is cause by this, native validation does not fallback.

Sure I'll look into it right away. Please maybe mask the passwd later for safety? I have had the passwd saved myself just now. thx. @jinchengchenghh

@jinchengchenghh
Copy link
Contributor

jinchengchenghh commented Mar 25, 2025

The password is in the CH backend document, but hard to find, it is open to all

@Yifeng-Wang
Copy link
Contributor Author

The password is in the CH backend document, but hard to find, it is open to all

Got it. Thx >:)

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Mar 26, 2025

Hi @jinchengchenghh I have several findings and would like to check with you, please correct me if Im wrong.

  1. I located the functions are now causing errors with info "Unknown function parser" for clickhouse BE in the CI test.
    image

  2. I referenced previous code which added support staticInvoke calling url_encode/decode functions, where, URL_DECODE/URL_ENCODE were added to CH_BLACKLIST_SCALAR_FUNCTION(CHExpressionUtil.scala) to make CHValidatorApi::doExprValidate return false(CHValidatorApi). That said, the staticInvoke branch will early return the Transformer, in advance gluten validation:(ExpressionConverter.scala)
    (main br line 151) case i: StaticInvoke => ....
    (main br line 171)val substraitExprName: String = getAndCheckSubstraitName(expr, expressionsMap)
    Is this as expected? or should we, make it a work around, jus take out staticInvoke branch and position it after line171's gluten check so that we make use of that black list?

@jinchengchenghh
Copy link
Contributor

Just add the new functions name to CH_BLACKLIST_SCALAR_FUNCTION like URL_DECODE

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Mar 27, 2025

Just add the new functions name to CH_BLACKLIST_SCALAR_FUNCTION like URL_DECODE

Hi @jinchengchenghh some new findings,

  1. [Error reprod] env: gluten clickhouse (main branch) + Spark3.5.2. The other StaticInvokes url_decode/url_decode also fail e.g. :
    INSERT INTO url_tbl VALUES ('https://spark.apache.org');
    SELECT url_encode(a) AS encoded_url FROM url_tbl;
    org.apache.gluten.exception.GlutenException: Unknown function parser url_decode
    which means currently CH_BLACKLIST_SCALAR_FUNCTION did NOT helped in CH be validation.
  2. [Problem] As I mentioned in my previous comment, the ExpressionTransfomer for staticInvoke calls in current main br code (ExpressionConverter.scala line151) should not return in advance of function checks. (BUG)
  3. [Planned Solution] Velox and CH backends exhibit diffs in their ValidatorAPI implementations, particularly in doNativeValidateWithFailureReason and doExprValidate.
  • doExprValidate: Velox BE->always true; CH BE-> go through some blacklists, e.g.
  • doNativeValidateWithFailureReason: Velox BE-> validate at native; CH BE -> not supp.

Can we for now intercept unsupp staticInvoke functions for CH at replaceStaticInvokeWithExpressionTransformer with an extra ValidatorApi::doExprValidate (so that CH_BLACKLIST_SCALAR_FUNCTION can serve purpose)? : )

@jinchengchenghh
Copy link
Contributor

jinchengchenghh commented Mar 27, 2025

Yes, I agree. Add the substrait name check by ValidatorApi::doExprValidate in replaceStaticInvokeWithExpressionTransformer rather than directly calling the ExpressionNames.URL_DECODE.
@taiyang-li Do you have some suggestions that make CH backend fallback these unsupported functions?

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Mar 30, 2025

Hi, @jinchengchenghh, I've added an extra doExprValidate in replaceStaticInvokeWithExpressionTransformer, the ClickHouse CI tests are now passing. Could you please review the updates when you have time?

@jinchengchenghh
Copy link
Contributor

These function are not merged in velox, we cannot add a unit test to it, can we merge these function native implement first?

@taiyang-li
Copy link
Contributor

taiyang-li commented Mar 31, 2025

Yes, I agree. Add the substrait name check by ValidatorApi::doExprValidate in replaceStaticInvokeWithExpressionTransformer rather than directly calling the ExpressionNames.URL_DECODE. @taiyang-li Do you have some suggestions that make CH backend fallback these unsupported functions?

Sorry for the late reply. The current change is ok for CH.

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Mar 31, 2025

These function are not merged in velox, we cannot add a unit test to it, can we merge these function native implement first?

Thanks alot @jinchengchenghh and @taiyang-li , so does that mean we can now move to the Velox side PR now? link

@github-actions
Copy link

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label May 16, 2025
@Yifeng-Wang
Copy link
Contributor Author

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

Current PR awaits the PR merge at Velox side and is not stale.

Copy link
Contributor

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good overall.

GenericExpressionTransformer(exprName, childTransformers, i)
}

if (Seq("encode", "decode").contains(i.functionName) && i.objectName.endsWith("UrlCodec")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the comparisons case-sensitive?

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, both the codes in origin main branch and this PR remain case-sensitive because they're comparing against actual Java method and class names.
image

)
}

if (i.functionName == "isLuhnNumber") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, case-sensitive is intentional. Same as above.
image

)
}

if (Seq("encode", "decode").contains(i.functionName) && i.objectName.endsWith("Base64")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, case-sensitive is intentional. Same as above.
image

}

if (
Set("varcharTypeWriteSideCheck", "charTypeWriteSideCheck", "readSidePadding").contains(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, case-sensitive is intentional. Same as above.
image

// Add test suite for CharVarcharCodegenUtils functions.
// A ProjectExecTransformer is expected to be constructed after expr support.
// We currently test below functions with Spark v3.4
testWithMinSparkVersion("Test charTypeWriteSideCheck function", "3.4") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could be rename as "charTypeWriteSideCheck" to be consistent in style with other cases. And are these functions only supported after Spark 3.4?

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all these functions should be tested with min Spark version 3.4,

  1. readSidePadding
    was introduced at branch-3.4 in sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java.
  2. charTypeWriteSideCheck & varcharTypeWriteSideCheck
    prior to 3.4, Spark's analyzer introduces an extra redundant cast(... as string) expression. The current transformer for StaticInvoke does not handle this specific pattern with the extra cast, which causes a fallback to the vanilla Spark ProjectExec.
image

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Sep 1, 2025

Hi @rui-mo ! I saw spark-tests fail and test fails were due to Exception type/message mismatches, not because of incorrect functionality, e.g.: the test expect "did not contain "Exceeds char/varchar type length limitation: 5", but actually got: "(6 vs. 5) Exceeds allowed length limitation: 5",

image image

Since we should not touch Spark's tests and should remain the error msg from Velox side, I wonder the correct way we handle current situation is to exclude the vanilla spark tests in the VeloxTestSettings.scala, but enable and implement the test suite at Gluten's test suite.
Since only some cases have the error msg mismatch, there should be both err msgs, I wonder if could add

  private val ERROR_MESSAGE =
    "Exceeds char/varchar type length limitation: 5"
  private val VELOX_ERROR_MESSAGE = "Exceeds allowed length limitation: 5"

and then assert VELOX_ERROR_MESSAGE ?
Thanks for your help! 🤝

@rui-mo
Copy link
Contributor

rui-mo commented Sep 1, 2025

@Yifeng-Wang Yes, for the error msg mismatch we usually exclude Spark's test, and rewrite it in the Gluten suite.

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

2 similar comments
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@Yifeng-Wang Yifeng-Wang force-pushed the add-staticinv-charvarcharcodegenutils branch from de06b42 to c04e793 Compare September 26, 2025 07:26
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@Yifeng-Wang
Copy link
Contributor Author

Hi @rui-mo , the tests fails are now solved for each of the Spark's version. The test suites are updated to make correct message assertions. Would you please help me take a look at the changes when you have time? Thanks!

// Add test suite for CharVarcharCodegenUtils functions.
// A ProjectExecTransformer is expected to be constructed after expr support.
// We currently test below functions with Spark v3.4
testWithMinSparkVersion("charTypeWriteSideCheck", "3.4") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only test with Spark3,4? Does Spark 3.3 not have this function?

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for pointing out. From my obervation turns out that although Spark 3.3 does have this function, the execution plan is different, not triggering staticInvoke, thus tested with min=Spark 3.4

GenericExpressionTransformer(exprName, childTransformers, i)
}

if (Seq("encode", "decode").contains(i.functionName) && i.objectName.endsWith("UrlCodec")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use case match for the function name? This will be more clean

if (transformer.isEmpty) {
throw new GlutenNotSupportException(s"Not supported staticInvoke call object: $objName")

if (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not expand the replaceIcebergStaticInvoke here, this function is a bit long and will be longger

.exclude("length check for input string values: with implicit cast")
.exclude("char/varchar type values length check: partitioned columns of other types")
enableSuite[GlutenDSV2CharVarcharTestSuite]
.exclude("length check for input string values: top-level columns")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there only the error message difference? Is there any change to align the error message?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like even if we align the error message, we still need to copy the tests, so let us keep current solution.

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Keep current solution.

@jinchengchenghh
Copy link
Contributor

Please resolve the conflict, thanks!

@Yifeng-Wang
Copy link
Contributor Author

Please resolve the conflict, thanks!

Yes I am doing it right now 🤝

@Yifeng-Wang Yifeng-Wang force-pushed the add-staticinv-charvarcharcodegenutils branch from c083639 to a0fbecb Compare October 21, 2025 13:47
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@Yifeng-Wang
Copy link
Contributor Author

Yifeng-Wang commented Oct 22, 2025

The latest commit added excludeGlutenTest to ClickHouseTestSettings.scala across different versions. The added expression support feature in this PR is only supported by Velox backend, while for ClickHouse backend, it fallbacks and thus should not match the expected err msg at gluten test suite.

Copy link
Contributor

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

.includeCH("length check for input string values: nested in both map key and value")
.includeCH("length check for input string values: nested in array of struct")
.includeCH("length check for input string values: nested in array of array")
.excludeGlutenTest("length check for input string values: top-level columns")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you please add comments above to illustrate the reason to exclude these tests (because they are rewritten due to ...)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do now.

if (!BackendsApiManager.getValidatorApiInstance.doExprValidate(ExpressionNames.BASE64, i)) {
throw new GlutenNotSupportException(
s"Not supported to map current ${i.getClass} call on function: ${i.functionName}.")
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the refactor. This validations seem to be newly introduced, would you please clarify a bit?

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi rui-mo, thanks for mentioning this. Apologize for such a long context within this PR.
This doExprValidate call on validator instance was added deliberately to ensure backend actually supports.
Technically this was from my earlier findings (if you scroll up a bit, on Mar 26) , and currently it ensures that:

  • An explicit call on doExprValidate.
  • CH_BLACKLIST_SCALAR_FUNCTION is looked through during e.g., CHValidatorApi::doExprValidate so that the functions which are ONLY supported at Velox side, won't cause "Unknown function parser" for clickhouse BE(in CI tests).
  • Intercept at Gluten with GlutenNotSupportException, instead of sending to actual unsupported (ClickHouse) backend.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I understand.

val exprName = fn match {
case "varcharTypeWriteSideCheck" => ExpressionNames.VARCHAR_TYPE_WRITE_SIDE_CHECK
case "charTypeWriteSideCheck" => ExpressionNames.CHAR_TYPE_WRITE_SIDE_CHECK
case "readSidePadding" => ExpressionNames.READ_SIDE_PADDING
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mappings from Spark's name to Velox's name are more commonly to be put at https://github.com/apache/incubator-gluten/blob/main/cpp/velox/substrait/SubstraitParser.cc#L387 to allow compatibility among different backends.

Copy link
Contributor Author

@Yifeng-Wang Yifeng-Wang Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rui-mo Hi rui-mo, could you help give me some guidance on how I should update the codes?
Currently the gluten-velox run of my expression convert does not seems like relying on this map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To add some context,
at Gluten side in ExpressionNames.scala. I had e.g.:
final val VARCHAR_TYPE_WRITE_SIDE_CHECK = "varchar_type_write_side_check" .
And at velox side, there is

registerFunction<
VarcharTypeWriteSideCheckFunction,
Varchar,
Varchar,
int32_t>
({prefix + "varchar_type_write_side_check"});

so I guess substrait function name (from ExpressionNames.VARCHAR_TYPE_WRITE_SIDE_CHECK = "varchar_type_write_side_check") already matches the Velox registered function name (prefix + "varchar_type_write_side_check").

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically, in Gluten’s ExpressionNames.scala, the Spark function names are defined, while in the C++ map, the corresponding function names for a specific backend are provided. For instance, the Spark function add might be represented as plus in certain backends. To maintain consistency, Gluten’s Scala side keeps Spark’s original naming (e.g., add), and it is the responsibility of each backend to map it to the appropriate equivalent name. This is just a minor suggestion—please feel free to decide whether you’d like to make this change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi rui-mo! ​Thanks for clarifying the reasoning behind this design. Maybe I'll keep the current state for this PR, but I'll be sure to follow this mapping mechanism in subsequent PR. Really appreciate the guidance!

@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@rui-mo rui-mo merged commit 41bd1ce into apache:main Oct 24, 2025
100 of 101 checks passed
@Yifeng-Wang
Copy link
Contributor Author

Hi @rui-mo , @jinchengchenghh !

Thank you so much for your thorough review and patience on this long-running PR. Your feedback was invaluable.
Greatly appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLICKHOUSE CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement][GLUTEN][VL] Add support for staticInvoke CharVarcharCodegenUtils.

5 participants