Indexed field access for List #1006

Igosuki · 2021-09-15T20:05:31Z

Which issue does this PR close?

Rationale for this change

Supporting accessing nested values is a great improvement as it makes everything more flexible since input data doesn't need to be flat, and so it doesn't need to be transformed by ETL prior to being ingested by datafusion.

What changes are included in this PR?

Nested value access for Lists and Dictionary

Are there any user-facing changes?

users will now be able to do : list[0][0] and dict['foo']['bar']

No breaking changes.

I need to test the dictionary access I've only used the list one.

Igosuki · 2021-09-15T20:06:23Z

I had to patch sqlparser to make this work, in order for sqlparser to support numbers as field selector.

houqp · 2021-09-16T01:34:00Z

@Igosuki could you send your sqlparser-rs patch to upstream? We can help review release a new version to crates.io.

Igosuki · 2021-09-16T07:35:02Z

apache/datafusion-sqlparser-rs#356

alamb · 2021-09-17T10:50:42Z

I will try and get sql parser change merged in and a new release created something this weekend

alamb · 2021-09-17T10:52:36Z

This PR (that we never completed) from @jorgecarleitao has related work I think -- namely access to struct fields: #628

Igosuki · 2021-09-17T16:23:11Z

I had initially used it, but it only allowed to use fields of a depth of one, and was simply patching the compound field access. I could look into allowing `table.col[i].field[j]` for instance in addition to `table.col[i]['field'][j]' Le ven. 17 sept. 2021 à 12:52, Andrew Lamb ***@***.***> a écrit :

…

This PR (that we never completed) from @jorgecarleitao <https://github.com/jorgecarleitao> has related work I think -- namely access to struct fields: #628 <#628> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1006 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADDFBQRRM23KXP72PZCFODUCMMX7ANCNFSM5EDGUKOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Igosuki · 2021-09-20T08:41:04Z

Will get back to this soon once I sort out the related issue here apache/datafusion-sqlparser-rs#356

Igosuki · 2021-09-24T12:41:56Z

I updated sqlparser for arbitrary nested access. But I'm having trouble understanding how select on dict should behave for instance with this e2c00f1#diff-6a157f1b320a619e7a3896027ce7d47a8e3ffeeae8ac3d21fe9157c92ebc2a6bL4955

alamb · 2021-09-24T18:36:29Z

I am preparing a sqlparser release FWIW: apache/datafusion-sqlparser-rs#360

houqp · 2021-10-09T20:24:38Z

@Igosuki do you need any help finishing up this PR?

Igosuki · 2021-10-10T10:08:18Z

@houqp Indeed, I mean I could just remove the dict lookup and do it in another PR so we could just get array access here.
The problem is here https://github.com/apache/arrow-datafusion/pull/1006/files#diff-08a2204dc38facfa0757eed25f99d9047fcd3b44bd03df02c89f6f71afacefefR143 and here https://github.com/apache/arrow-datafusion/pull/1006/files#diff-6a157f1b320a619e7a3896027ce7d47a8e3ffeeae8ac3d21fe9157c92ebc2a6bR5218
If there I have a column, each row may contain a dict, and so if I do a reverse lookup, then it should return a value for each non-null row right, but instead the dict is treated as a single instance.

houqp · 2021-10-10T22:25:36Z

I think there might be some confusions here. In my mind, col["a"]["b"] should access col as a dictionary in the sense of hashmap, not Arrow dictionary encoded array. In other words, when index key is a string, I think we should be matching for Struct datatype instead of dictionary type.

BTW, I am perfectly fine with us splitting the PRs into two and handle the dictionary access as a follow up if you prefer to do so 👍

Igosuki · 2021-10-12T08:31:05Z

@houqp Ok, it's what I had in mind as well.
Although now I am reminded that dict lookup is pretty useful to join lookup data with queries at runtime and limit the amount of data stored, so this code will still be useful, just not in this way.

Igosuki · 2021-10-12T08:42:35Z

Removed dictionary lookup so it should be gtg now

Igosuki · 2021-10-12T13:43:50Z

needs to rerun at on sha 48cad8f

datafusion/src/field_util.rs

jimexist · 2021-10-12T15:22:20Z

datafusion/src/logical_plan/expr.rs

@@ -245,6 +246,13 @@ pub enum Expr {
    IsNull(Box<Expr>),
    /// arithmetic negation of an expression, the operand must be of a signed numeric data type
    Negative(Box<Expr>),
+    /// Returns the field of a [`ListArray`] or ['DictionaryArray'] by name


by string name or integer indices?

@Igosuki I think made a good point here, would be better to mention both index and name. On top of this, I think DictionaryArray should be removed here?

Suggested change

/// Returns the field of a [`ListArray`] or ['DictionaryArray'] by name

/// Returns the field of a [`ListArray`] by name

jimexist · 2021-10-12T15:23:38Z

datafusion/src/physical_plan/expressions/get_indexed_field.rs

+                    let iter = concat(vec.as_slice()).unwrap();
+                    Ok(ColumnarValue::Array(iter))
+                }
+                _ => Err(DataFusionError::NotImplemented(


using other and provide type name is more helpful

@Igosuki what do you think about the comment here?

Oh guess I overlooked this, will fix

Igosuki · 2021-10-26T06:33:27Z

Agree with @jimexist 's comment, the rest looks good to me. @Igosuki for the follow up struct field access support PR, you can follow the physical plan evaluation implementation that @jorgecarleitao started awhile back at: https://github.com/apache/arrow-datafusion/pull/628/files#diff-ae63212535ac34ea66a26dc4726715b594b05794376b1b1fb511ff32211ed576R75. I think your SQL syntax combined with that physical plan implementation should be solid +1

I'll get to it as soon as this is merged

Igosuki · 2021-10-26T06:34:00Z

I addressed the concerns in the comments, things should be gtg

alamb · 2021-10-27T18:20:14Z

FYI I have this PR on my list to review, I just haven't had a chance to to do so yet -- will try and get it done tomorrow

Igosuki · 2021-10-28T14:05:56Z

No worries, I've been slow as well and obviously there's a lot going on on the project. When it gets validated I'll do the follow up for struct array access.

alamb

I think this is looking good personally -- thank you @Igosuki !

Let us know if you would like some help finishing up the comments

alamb · 2021-10-28T19:38:48Z

datafusion/src/logical_plan/expr.rs

@@ -245,6 +246,13 @@ pub enum Expr {
    IsNull(Box<Expr>),
    /// arithmetic negation of an expression, the operand must be of a signed numeric data type
    Negative(Box<Expr>),
+    /// Returns the field of a [`ListArray`] or ['DictionaryArray'] by name


Suggested change

/// Returns the field of a [`ListArray`] or ['DictionaryArray'] by name

/// Returns the field of a [`ListArray`] by name

alamb · 2021-10-28T19:39:31Z

datafusion/src/lib.rs

@@ -231,6 +231,7 @@ pub mod variable;
 pub use arrow;
 pub use parquet;

+pub mod field_util;


Suggested change

pub mod field_util;

pub (crate) mod field_util;

Unless there is some reason to expose this module, I think it might be nice to leave it internal so that we can move it around as needed

alamb · 2021-10-28T19:40:14Z

datafusion/src/physical_plan/expressions/get_indexed_field.rs

+// specific language governing permissions and limitations
+// under the License.
+
+//! get field of a struct array


Suggested change

//! get field of a struct array

//! get field of a `ListArray`

alamb · 2021-10-28T19:41:02Z

datafusion/src/field_util.rs

+            ))
+        }
+        _ => Err(DataFusionError::Plan(
+            "The expression to get an indexed field is only valid for `List` or 'Dictionary'"


Suggested change

"The expression to get an indexed field is only valid for `List` or 'Dictionary'"

"The expression to get an indexed field is only valid for `List` types"

alamb · 2021-10-28T19:42:46Z

datafusion/src/physical_plan/expressions/get_indexed_field.rs

+                    Ok(ColumnarValue::Array(iter))
+                }
+                _ => Err(DataFusionError::NotImplemented(
+                    "get indexed field is only possible on lists".to_string(),


Untested:

Suggested change

"get indexed field is only possible on lists".to_string(),

format!("get indexed field is only possible on lists with int64 indexes. Tried {} with {} index", array.data_type(), self.key())

alamb · 2021-10-28T19:42:59Z

datafusion/src/physical_plan/expressions/get_indexed_field.rs

+                )),
+            },
+            ColumnarValue::Scalar(_) => Err(DataFusionError::NotImplemented(
+                "field is not yet implemented for scalar values".to_string(),


Suggested change

"field is not yet implemented for scalar values".to_string(),

"field access is not yet implemented for scalar values".to_string(),

alamb · 2021-10-28T19:46:12Z

datafusion/src/physical_plan/expressions/get_indexed_field.rs

+        let arg = self.arg.evaluate(batch)?;
+        match arg {
+            ColumnarValue::Array(array) => match (array.data_type(), &self.key) {
+                (DataType::List(_), ScalarValue::Int64(Some(i))) => {


if self.key is null, I think we could also return null

So perhaps we could add a new case like this (untested):

(DataType::List(_), _) if self.key.is_null=> { let scalar_null: ScalarValue = array.data_type().try_into()?; Ok(ColumnarValue::Scalar(scalar_null)) }

There's also the issue of an empty list which doesn't get through concat

Sounds like a good opportunity for follow up work / PRs

alamb · 2021-10-28T19:46:35Z

datafusion/src/physical_plan/planner.rs

@@ -141,6 +143,10 @@ fn create_physical_name(e: &Expr, is_first_expr: bool) -> Result<String> {
            let expr = create_physical_name(expr, false)?;
            Ok(format!("{} IS NOT NULL", expr))
        }
+        Expr::GetIndexedField { expr, key } => {


Igosuki · 2021-10-29T11:42:34Z

@alamb Added some test coverage in get_indexed_field.rs let me know if you agree with the changes

alamb · 2021-10-29T18:00:14Z

Thanks for sticking with it @Igosuki !

github-actions bot added ballista datafusion Changes in the datafusion crate sql SQL Planner labels Sep 15, 2021

houqp added the enhancement New feature or request label Sep 16, 2021

Igosuki force-pushed the map_access_i branch from ae11339 to 0e0ede2 Compare September 20, 2021 12:10

Igosuki force-pushed the map_access_i branch from e2c00f1 to ec777c0 Compare September 24, 2021 13:05

This was referenced Sep 25, 2021

Update sqlparser-rs to 0.11 #1051

Closed

Update sqlparser-rs to 0.11 #1052

Merged

Igosuki force-pushed the map_access_i branch from ec777c0 to 2f414fb Compare September 30, 2021 15:14

Igosuki force-pushed the map_access_i branch from 2f414fb to 0c38a83 Compare October 10, 2021 10:04

Igosuki force-pushed the map_access_i branch 2 times, most recently from 4b7eabc to 48cad8f Compare October 12, 2021 08:42

jimexist reviewed Oct 12, 2021

View reviewed changes

datafusion/src/field_util.rs Show resolved Hide resolved

jimexist reviewed Oct 12, 2021

View reviewed changes

houqp mentioned this pull request Oct 18, 2021

Add SQL support for referencing fields in structs #119

Closed

Igosuki force-pushed the map_access_i branch 2 times, most recently from 232c974 to bfb82de Compare October 26, 2021 06:33

houqp requested a review from jimexist October 28, 2021 15:04

alamb approved these changes Oct 28, 2021

View reviewed changes

Igosuki added 14 commits October 29, 2021 17:36

enable GetIndexedField for Array and Dictionary

b2a7a82

fix GetIndexedField which should index slices not values

98cb0a8

Compat with latest sqlparser

7b5aec6

Add two tests for indexed_field access, level one and level two nesting

c0b0920

fix compilation issues

60ba858

try fixing dictionary lookup

c96cc6b

address clippy warnings

16d190d

fix test

f04507f

Revert dictionary lookup for indexed fields

fb018c0

Reject negative ints when accessing list values in get indexed field

a630bcd

Fix doc in get_indexed_field

f0dfe94

use GetIndexedFieldExpr directly

53d361b

return the data type in unavailable field indexation error message

b2012b6

Add unit tests for the physical plan of get_indexed_field

24bac8b

Igosuki force-pushed the map_access_i branch from 7a58b12 to 24bac8b Compare October 29, 2021 15:36

Fix missing clause for const evaluator

5d61ce5

alamb approved these changes Oct 29, 2021

View reviewed changes

alamb merged commit 22a7d74 into apache:master Oct 29, 2021

nl5887 mentioned this pull request Dec 28, 2021

Python api doesn't have indexed field access (see #1005) #1501

Closed

	/// Returns the field of a [`ListArray`] or ['DictionaryArray'] by name
	/// Returns the field of a [`ListArray`] by name

	//! get field of a struct array
	//! get field of a `ListArray`

	"The expression to get an indexed field is only valid for `List` or 'Dictionary'"
	"The expression to get an indexed field is only valid for `List` types"

	"get indexed field is only possible on lists".to_string(),
	format!("get indexed field is only possible on lists with int64 indexes. Tried {} with {} index", array.data_type(), self.key())

	"field is not yet implemented for scalar values".to_string(),
	"field access is not yet implemented for scalar values".to_string(),

Indexed field access for List #1006

Indexed field access for List #1006

Conversation

Igosuki commented Sep 15, 2021

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Igosuki commented Sep 15, 2021

houqp commented Sep 16, 2021

Igosuki commented Sep 16, 2021

alamb commented Sep 17, 2021

alamb commented Sep 17, 2021

Igosuki commented Sep 17, 2021 via email

Igosuki commented Sep 20, 2021

Igosuki commented Sep 24, 2021

alamb commented Sep 24, 2021

houqp commented Oct 9, 2021

Igosuki commented Oct 10, 2021

houqp commented Oct 10, 2021 • edited Loading

Igosuki commented Oct 12, 2021

Igosuki commented Oct 12, 2021

Igosuki commented Oct 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Igosuki commented Oct 26, 2021

Igosuki commented Oct 26, 2021

alamb commented Oct 27, 2021

Igosuki commented Oct 28, 2021

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Igosuki Oct 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Igosuki commented Oct 29, 2021

alamb commented Oct 29, 2021

houqp commented Oct 10, 2021 •

edited

Loading

Igosuki Oct 29, 2021 •

edited

Loading