Standardize the separator in name #10363

jayzhan211 · 2024-05-03T09:22:07Z

Which issue does this PR close?

Closes #10364 .

Rationale for this change

I got the name checking error that due to the mismatch of separator in creating name. I found that most of the name built with single comma and space, so I think we should switch others to this format.

Error is like

External error: query failed: DataFusion error: Internal error: Input field name SUM(get_field(CASE WHEN get_field(t2.struct(t1.time,t1.load1,t1.load2,t1.host), Utf8("c3")) IS NOT NULL THEN t2.struct(t1.time,t1.load1,t1.load2,t1.host) END,Utf8("c2"))) does not match with the projection expression SUM(get_field(CASE WHEN get_field(t2.struct(t1.time,t1.load1,t1.load2,t1.host),Utf8("c3")) IS NOT NULL THEN t2.struct(t1.time,t1.load1,t1.load2,t1.host) END,Utf8("c2"))).
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
[SQL] select t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] as host, sum((case when t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] is not null then t2."struct(t1.time,t1.load1,t1.load2,t1.host)" end)['c2']) from (select struct(time,load1,load2,host) from t1) t2 where t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] IS NOT NULL group by t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] order by host;

I got one with t2.struct(t1.time,t1.load1,t1.load2,t1.host), Utf8("c3") and another t2.struct(t1.time,t1.load1,t1.load2,t1.host),Utf8("c3"). That is why I think we should fix the separator.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

You can find the error here with cargo test --test sqllogictests -- expr. Although, it is not on the main branch, but I think most of the changes are unrelated to the name checking issue.

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 · 2024-05-12T08:41:27Z

datafusion/expr/src/expr.rs

@@ -1859,7 +1859,7 @@ pub(crate) fn create_name(e: &Expr) -> Result<String> {
                AggregateFunctionDefinition::UDF(..) => {
                    let names: Vec<String> =
                        args.iter().map(create_name).collect::<Result<_>>()?;
-                    names.join(",")
+                    names.join(", ")


one of the main change

jayzhan211 · 2024-05-12T08:41:30Z

datafusion/expr/src/udf.rs

@@ -291,7 +291,7 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
    ///
    fn display_name(&self, args: &[Expr]) -> Result<String> {
        let names: Vec<String> = args.iter().map(create_name).collect::<Result<_>>()?;
-        Ok(format!("{}({})", self.name(), names.join(",")))
+        Ok(format!("{}({})", self.name(), names.join(", ")))


one of the main change

Weijun-H · 2024-05-12T12:01:16Z

datafusion/common/src/test_util.rs

+#[macro_export]
+macro_rules! assert_snapshot {
+    ($CHUNKS: expr) => {
+        let formatted = $crate::arrow::util::pretty::pretty_format_batches_with_options(
+            $CHUNKS,
+            &$crate::format::DEFAULT_FORMAT_OPTIONS,
+        )
+        .unwrap()
+        .to_string();
+
+        let actual_lines: Vec<&str> = formatted.trim().lines().collect();
+
+        insta::assert_yaml_snapshot!(actual_lines);
+    };
+}
+


Why do we use this for tests?

test with insta. Because changing them manually is painful. I did this in datafusion-example, since they are not test.

alamb

Thank you @jayzhan211 this looks like a really nice contributon

Given the insta tool is a new dependency and I think has larger implications than just this PR I think we should create a new ticket / PR to propose adding it to DataFusion, rather than including it as part of this unrelated PR (though I suspect one reason to use insta is to make updating the tests easier)

alamb · 2024-05-13T18:38:40Z

datafusion/core/tests/dataframe/dataframe_functions.rs

-    ];
-
-    assert_fn_batches!(expr, expected, 1);
+    assert_fn_batches!(expr, 1);


I think it was easier to see what was going on with these tests when the results are inlined -- i think insta supports that too "Inlined snapshots" -- https://insta.rs/docs/snapshot-types/

alamb · 2024-05-13T18:39:10Z

datafusion/core/src/physical_planner.rs

@@ -121,7 +121,7 @@ fn create_function_physical_name(
        false => "",
    };

-    let phys_name = format!("{}({}{})", fun, distinct_str, names.join(","));
+    let phys_name = format!("{}({}{})", fun, distinct_str, names.join(", "));


Are the changes in this file the only in the PR?

Several one, but are all like this

github-actions · 2024-07-13T01:49:59Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions bot added logical-expr Logical plan and expressions core Core datafusion crate sqllogictest labels May 3, 2024

jayzhan211 added 5 commits May 12, 2024 09:37

rename

98491ab

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rename

8fd8817

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

apply snapshot

0a0c402

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

add more change

c9844cc

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

add comment

0356cf5

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 force-pushed the standardize-name-format branch from e95b933 to 0356cf5 Compare May 12, 2024 01:39

jayzhan211 added 6 commits May 12, 2024 12:19

fix ci

660b2c2

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix ci

c84f54e

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

upd more

f566b0b

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fmt

26d2ee3

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

file

3d0ca6f

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix tochar

7e42d8c

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 changed the title ~~Minor: Standardize the separator in name~~ Standardize the separator in name May 12, 2024

jayzhan211 commented May 12, 2024

View reviewed changes

jayzhan211 marked this pull request as ready for review May 12, 2024 08:41

Weijun-H reviewed May 12, 2024

View reviewed changes

alamb reviewed May 13, 2024

View reviewed changes

jayzhan211 marked this pull request as draft May 13, 2024 23:37

github-actions bot added the Stale PR has not had any activity for some time label Jul 13, 2024

jayzhan211 closed this Jul 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardize the separator in name #10363

Standardize the separator in name #10363

jayzhan211 commented May 3, 2024 •

edited

Loading

jayzhan211 May 12, 2024

jayzhan211 May 12, 2024

Weijun-H May 12, 2024

jayzhan211 May 13, 2024

alamb left a comment

alamb May 13, 2024

alamb May 13, 2024

jayzhan211 May 13, 2024

github-actions bot commented Jul 13, 2024

Standardize the separator in name #10363

Standardize the separator in name #10363

Conversation

jayzhan211 commented May 3, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

jayzhan211 May 12, 2024

Choose a reason for hiding this comment

jayzhan211 May 12, 2024

Choose a reason for hiding this comment

Weijun-H May 12, 2024

Choose a reason for hiding this comment

jayzhan211 May 13, 2024

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb May 13, 2024

Choose a reason for hiding this comment

alamb May 13, 2024

Choose a reason for hiding this comment

jayzhan211 May 13, 2024

Choose a reason for hiding this comment

github-actions bot commented Jul 13, 2024

jayzhan211 commented May 3, 2024 •

edited

Loading