Add initial support for struct dtype #756

costaraphael · 2023-12-03T16:08:50Z

This PR is heavily inspired by the work done here.

What works at this point:

defining basic K/V structures
defining nested K/V structures
casting
inspection
importing/exporting from parquet/IPC/NDJSON (CSV is not supported)

The goal is to add initial support to the type now and add extra functionality (like Series.field/2 and DataFrame.unnest/2) in the future.

I'm well aware that I'm opening a PR with some big changes without discussing it first, so if this is not a feature the Explorer team wants to add, or if you want to take a completely different approach than what I did, please feel free to scrape this. 🙂

costaraphael

Left a couple of comments in parts of the code I made some judgment calls and wanted to add a bit more context/ask for feedback.

Also please let me know if I'm missing anything! 😄

costaraphael · 2023-12-03T16:14:34Z

lib/explorer/series.ex


  @type time_unit :: :nanosecond | :microsecond | :millisecond
  @type datetime_dtype :: {:datetime, time_unit}
  @type duration_dtype :: {:duration, time_unit}
  @type list_dtype :: {:list, dtype()}
+  @type struct_dtype :: {:struct, %{String.t() => dtype()}}


I went with this typespec for a couple of reasons:

:struct - to use the same name used on Polars

a map as a second element - to keep track of the inner dtypes, so we can pluck them later on

using a String.t() instead of atom() for the field name - since we can import large JSON datasets, with potentially random keys, it felt prudent to use a string here to avoid leaking atoms

The trade-offs for these choices are:

:struct already has a meaning in Elixir. We could name it :map instead but we would lose the 1:1 naming parity with Polars.

The dtype gets very verbose (as you can see on the tests). I think this one is pretty minor though, as I think people won't be dealing with this dtype directly that often.

Please let me know if you have any other ideas regarding the dtype name and how to represent it.

Hi! I really like this PR and think this missing functionality would be very helpful.

That said, I'm hesitant about the term "struct" since as you say it's overloaded. What do you think about DF.from_rows? Then it would pair with DF.to_rows just like Series.from_list/Series.to_list.

So this is not importing data from Elixir structs, but literally having a struct column inside Explorer. You could even have lists of structs. :)

Whoops! Thank you, I was skimming 😅

I still worry that "struct" is overloaded ("map" may be better?), but now less so. It's sorta like how we have {:f, 32} even though there's no real equivalent in Elixir.

Yeah, I was a bit iffy about naming it struct as well because of the pre-existing meaning within Elixir, but also naming it the more Elixir-appropriate map would mean people used to Polars would have to keep this "translation" in mind when using Explorer.

Feels like a "pick your poison" type of choice 😅

I think we could solve the naming issue by documenting it (like I mentioned in my other comment). Since we are following Arrow data structs, we could just document that. WDYT?

@philss Yeah if struct aligns with polars and arrow then it's the right call 👍 And documenting the name in case of potential confusion is def the way to go.

I've added the extra bits to the docs here: e565494

WDYT?

costaraphael · 2023-12-03T16:16:27Z

native/explorer/src/encoding.rs

@@ -573,6 +574,12 @@ pub fn term_from_value<'b>(v: AnyValue, env: Env<'b>) -> Result<Term<'b>, Explor
        AnyValue::Duration(v, time_unit) => encode_duration(v, time_unit, env),
        AnyValue::Categorical(idx, mapping, _) => Ok(mapping.get(idx).encode(env)),
        AnyValue::List(series) => list_from_series(ExSeries::new(series), env),
+        AnyValue::Struct(_, _, fields) => v
+            ._iter_struct_av()


This function looks very undocumented, but it is part of the public API from Polars: https://docs.rs/polars/latest/polars/datatypes/enum.AnyValue.html#method._iter_struct_av

The Python implementation deals with struct values through it as well.

I see. Maybe it's a unstable API, since normally it would not have the _ at the beginning. We can ask them later if this should be used, or if there is an alternative.

Yeah. I tried dealing with the enum entry directly as well, but it quickly required me to do a bunch of unsafe operations 😅

That function is still performing those unsafe calls, but I trust their unsafe Rust more than mine 😂

costaraphael · 2023-12-03T16:19:02Z

test/explorer/series/struct_test.exs

+               """
+               #Explorer.Series<
+                 Polars[2]
+                 struct[1] [%{"a" => 1}, %{"a" => 2}]


I followed the Polars approach here and only displayed the number of keys in the struct series. This made sense to me, as it would get out of hand pretty quick with large structs and nested values.

Let me know if you disagree and believe this should be more descriptive.

costaraphael · 2023-12-03T16:22:17Z

lib/explorer/shared.ex

@@ -2,7 +2,7 @@ defmodule Explorer.Shared do
  # A collection of **private** helpers shared in Explorer.
  @moduledoc false

-  @non_list_types [
+  @scalar_types [


With struct type being added the name didn't seem right anymore. Hope I got the intention right here. 😅

costaraphael · 2023-12-03T16:23:54Z

lib/explorer/shared.ex

@@ -27,7 +27,7 @@ defmodule Explorer.Shared do
  within lists inside.
  """
  def dtypes do
-    @non_list_types ++ [{:list, :any}]
+    @scalar_types ++ [{:list, :any}, {:struct, :any}]


I followed a similar approach from the list type implementation and used :any to denote "a struct of any shape". I'm not 100% happy with it though, please let me know if you have any suggestions.

costaraphael · 2023-12-03T16:28:01Z

lib/explorer/shared.ex

+    Enum.map(list, fn item ->
+      Map.new(item, fn {field, inner_value} ->
+        inner_dtype = Map.fetch!(dtypes, to_string(field))
+        [casted_value] = cast_numerics_deep([inner_value], inner_dtype)


This feels kinda inefficient (it's going to wrap/run through Enum.map/unwrap each value of the list of maps), but I was not too worried about it since I don't see Series.from_list as being the main way people will interact with this dtype. I feel like building a struct series inside a dataframe, or reading a struct series from an external file would be far more common.

Let me know if you disagree though.

lib/explorer/shared.ex

josevalim

Looks great to me, only some minor nits!

philss

This is a really great feature and awesome PR! 😺

I added some minor comments, but overall it looks great!

lib/explorer/series.ex

philss · 2023-12-04T15:33:07Z

lib/explorer/series.ex


  @type time_unit :: :nanosecond | :microsecond | :millisecond
  @type datetime_dtype :: {:datetime, time_unit}
  @type duration_dtype :: {:duration, time_unit}
  @type list_dtype :: {:list, dtype()}
+  @type struct_dtype :: {:struct, %{String.t() => dtype()}}


I think we could solve the naming issue by documenting it (like I mentioned in my other comment). Since we are following Arrow data structs, we could just document that. WDYT?

lib/explorer/shared.ex

philss · 2023-12-04T15:37:24Z

native/explorer/src/encoding.rs

@@ -573,6 +574,12 @@ pub fn term_from_value<'b>(v: AnyValue, env: Env<'b>) -> Result<Term<'b>, Explor
        AnyValue::Duration(v, time_unit) => encode_duration(v, time_unit, env),
        AnyValue::Categorical(idx, mapping, _) => Ok(mapping.get(idx).encode(env)),
        AnyValue::List(series) => list_from_series(ExSeries::new(series), env),
+        AnyValue::Struct(_, _, fields) => v
+            ._iter_struct_av()


I see. Maybe it's a unstable API, since normally it would not have the _ at the beginning. We can ask them later if this should be used, or if there is an alternative.

test/explorer/series/struct_test.exs

Co-authored-by: Philip Sampaio <philip.sampaio@gmail.com>

josevalim · 2023-12-04T22:05:30Z

Thank you! 💚 💙 💜 💛 ❤️

I assume the next step is an API for adding/updating/removing struct fields? :D

costaraphael · 2023-12-05T08:12:03Z

I assume the next step is an API for adding/updating/removing struct fields? :D

Something like that!

My next target would be to allow breaking a struct series into its component series (DataFrame.unnest/2) and building a struct from existing series (Series.build_struct/1). These two operations would unblock any workflow regarding parsing/exporting data with nested values.

Operations within struct series are kinda limited in Polars after that, but the interesting features I can see being added:

Plucking a single field as a series (Series.field/2)
Renaming struct fields (Series.rename_fields/2)
JSON encoding/decoding (Series.json_(encode|decode)/1)

Should I open an issue to keep track of this?

josevalim · 2023-12-05T08:17:28Z

No need for an issue, i think for these it is fine to contribute as the need arises :)

Add initial support for struct type

76f5c87

costaraphael commented Dec 3, 2023

View reviewed changes

josevalim reviewed Dec 3, 2023

View reviewed changes

lib/explorer/shared.ex Outdated Show resolved Hide resolved

josevalim reviewed Dec 3, 2023

View reviewed changes

lib/explorer/shared.ex Outdated Show resolved Hide resolved

josevalim reviewed Dec 3, 2023

View reviewed changes

lib/explorer/shared.ex Show resolved Hide resolved

josevalim approved these changes Dec 3, 2023

View reviewed changes

Address PR suggestions

53e3e87

philss reviewed Dec 4, 2023

View reviewed changes

costaraphael and others added 3 commits December 4, 2023 15:56

Update lib/explorer/shared.ex

0847fe4

Co-authored-by: Philip Sampaio <philip.sampaio@gmail.com>

Add tests mixing lists and structs

a5eb947

Add extra docs around the type name

e565494

josevalim merged commit 6c12034 into elixir-explorer:main Dec 4, 2023
4 checks passed

costaraphael mentioned this pull request Dec 5, 2023

Add DataFrame.unnest/2 #758

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support for struct dtype #756

Add initial support for struct dtype #756

costaraphael commented Dec 3, 2023

costaraphael left a comment

costaraphael Dec 3, 2023

billylanchantin Dec 4, 2023

josevalim Dec 4, 2023

billylanchantin Dec 4, 2023

costaraphael Dec 4, 2023

philss Dec 4, 2023

billylanchantin Dec 4, 2023

costaraphael Dec 4, 2023

costaraphael Dec 3, 2023

philss Dec 4, 2023

costaraphael Dec 4, 2023

costaraphael Dec 3, 2023

costaraphael Dec 3, 2023

costaraphael Dec 3, 2023

costaraphael Dec 3, 2023

josevalim left a comment

philss left a comment

philss Dec 4, 2023

philss Dec 4, 2023

josevalim commented Dec 4, 2023

costaraphael commented Dec 5, 2023

josevalim commented Dec 5, 2023

Add initial support for struct dtype #756

Add initial support for struct dtype #756

Conversation

costaraphael commented Dec 3, 2023

costaraphael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josevalim left a comment

Choose a reason for hiding this comment

philss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josevalim commented Dec 4, 2023

costaraphael commented Dec 5, 2023

josevalim commented Dec 5, 2023