Pass down backend to lazy series and enable re_named_captures/2 usage
#896
Conversation
The idea is to simplify and validate the dtypes for lazy series.
The idea is to enable using it in case of any requirement.
lib/explorer/backend/lazy_frame.ex
Outdated
| @@ -73,10 +74,21 @@ defmodule Explorer.Backend.LazyFrame do | |||
| @impl true | |||
| def pull(df, column) do | |||
| dtype_for_column = df.dtypes[column] | |||
| series_backend = get_series_backend(df.data.original_data) | |||
|
|
|||
| data = LazySeries.new(:column, [column], dtype_for_column) | |||
There was a problem hiding this comment.
And we should probably pass the backend as a required argument here. We need to make sure we always pass it forward, otherwise something like re_named_captures(s.foo <> s.bar, ~S"...") won't work if we forget to pass the backend inside? Would this be too large of a change?
There was a problem hiding this comment.
I'm passing down based on the args - see the new/3 function at LazySeries -, which will mostly contain a lazy series. The only exceptions are from_list/2 and from_binary/2. I honestly don't know how to solve this, because there is no point of reference to get the backend. I was thinking about getting the default backend for them. WDYT?
There was a problem hiding this comment.
Using the default in c708f46
There is no way to test that scenario today.
lib/explorer/backend/lazy_series.ex
Outdated
|
|
||
| Logger.warning( | ||
| "cannot get backend from LazySeries. Using the default one: #{inspect(backend)}" | ||
| ) |
There was a problem hiding this comment.
For this function I couldn't find a way to reproduce the exception. But if a series operation that accepts scalar values for all its arguments is called, it will call "from_list", and it will raise here. But this is unlikely to happen IMHO.
There was a problem hiding this comment.
What if we raise instead then?
There was a problem hiding this comment.
Yeah, it makes sense! I will update it later today :)
Co-authored-by: José Valim <jose.valim@dashbit.co>
This is a way to enable the usage of backend functions for series inside the context of
Query(in the LazySeries implementation). With that, we can usere_named_captures/2in the functions that are using lazy series.This is related to the following discussion: #895 (comment)