-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suggestion to update the UDF documentation #12912
Comments
could you show an example of a udf you'd like to use? |
@MarcoGorelli Unfortunately, I don't have any now. I do, however, have a simpler UDF that revolves third-party packages. Lets take this fictive example:
|
thanks - have you looked at https://pola-rs.github.io/polars/user-guide/migration/pandas/#pandas-transform ? does this answer your question? (btw, note that if you're running a third party package with a lambda, then pandas transform won't be vectorised either) |
One thing to note is how powerful using numba's guvectorize is for making UDFs since they are vectorized ufuncs. |
@MarcoGorelli You are right. Yes, i looked at it and understand that polars want us to use native functions instead. Although not always possible. Tried this |
thanks - what do you mean by 'no luck'? could you make a reproducible example please |
You often need to wrap the custom function in
|
Strongly agree with this. That being said, for heavy usage the syntax gets kind of clunky. For users that know what they're doing however, I suspect the Polar's team would rather steer them into writing native Rust kernels with the py03-polars extension. But all in all, I find it hard to beat the rapid prototyping & performance, all from a single python environment that Numba guvectorize + Polars(including the Lazyframe eval) can get you. There are some rough edges with this though. Starting with the next Numba release, we should get this: numba/numba#9058 Also, looking forward to the future, it would be interesting to see if Polars will play nice with Mojo kernels/code, as (assuming Mojo takes off) I wager that would be a decent overlap among the user bases. |
@deanm0000 I tried, and unfortunately, it didn't work. @MarcoGorelli, here is a simple reproducible example:
Keep in mind that you need to do pip install pandas-ta first. I already know this can be done in polars by using the rolling function, but thought of leaving an example here |
It is mentioned in the Notes section of the docs, but perhaps it could be made clearer.
(Looks like there is also some formatting issue with the rst file?) |
looks to me like In [42]: ta.sma(dfpl['Values'], length=2)
In [43]: ta.sma(dfpl['Values'].to_numpy(), length=2)
In [44]: ta.sma(pd.Series(dfpl['Values']), length=2)
Out[44]:
0 NaN
1 0.795927
2 0.643506
3 0.690988
4 0.525113
5 0.267241
6 0.262692
7 0.494509
8 0.680310
9 0.742702
10 0.762478
11 0.646630
12 0.389337
13 0.055688
14 0.333522
15 0.333357
16 0.090452
17 0.220633
18 0.362725
19 0.410134
20 0.665995
21 0.678027
22 0.531861
23 0.698057
24 0.826965
25 0.660905
26 0.422452
27 0.240534
28 0.102339
29 0.347099
30 0.375864
31 0.235960
32 0.545532
33 0.841617
34 0.879748
35 0.766130
36 0.498510
37 0.551201
38 0.502415
39 0.153784
40 0.384721
41 0.589381
42 0.424279
43 0.496428
44 0.431987
45 0.451654
46 0.786145
47 0.776003
48 0.384494
49 0.283826
Name: SMA_2, dtype: float64 so, barring converting to pandas (or requesting that they support polars) I'm not sure there's much that can be done here |
For reference, here is a good example of But going back to your question - have you tried just using Polars codedf = pl.DataFrame({
'Group': np.random.choice(['A', 'B', 'C'], size=50),
'Values': np.random.rand(50)
})
df1 = df.group_by('Group').agg(pl.col('Values'))
# This is what we have so far:
# shape: (3, 2)
# ┌───────┬──────────────────────────────────┐
# │ Group ┆ Values │
# │ --- ┆ --- │
# │ str ┆ list[f64] │
# ╞═══════╪══════════════════════════════════╡
# │ C ┆ [0.035229, 0.925109, … 0.297392] │
# │ A ┆ [0.789259, 0.004903, … 0.56967] │
# │ B ┆ [0.917179, 0.396942, … 0.962115] │
# └───────┴──────────────────────────────────┘
df2 = df1.with_columns(**{
'Moving Average': pl.col('Values').map_elements(lambda x: ....) # Use your UDF that takes in list of values here
})
# Then an explode to have it in the original format. |
that won't work I'm afraid In [14]: df2 = df1.with_columns(**{
...: 'Moving Average': pl.col('Values').map_elements(lambda x: ta.sma(x, lenght=2)) # Us
...: e your UDF that takes in list of values here
...: })
In [15]: df2
Out[15]:
shape: (3, 3)
┌───────┬──────────────────────────────────┬────────────────┐
│ Group ┆ Values ┆ Moving Average │
│ --- ┆ --- ┆ --- │
│ str ┆ list[f64] ┆ list[null] │
╞═══════╪══════════════════════════════════╪════════════════╡
│ B ┆ [0.015636, 0.165267, … 0.520834] ┆ null │
│ C ┆ [0.230894, 0.241025, … 0.385417] ┆ null │
│ A ┆ [0.844534, 0.74732, … 0.651077] ┆ null │
└───────┴──────────────────────────────────┴────────────────┘ The issue, as far as I can tell, is that I'd suggest opening a feature request to them to accept numpy arrays (or, even better, Polars) |
Can't that be fixed with code like: def pandas_ta_wrapper(x:list)->pd.Series:
return pd.Series(x)
df2 = df1.with_columns(**{
'Moving Average': pl.col('Values').map_elements(lambda x: ta.sma(pandas_ta_wrapper(x), lenght=2))
}) |
You'd need to then convert back to Polars This works: (
dfpl.with_columns(
MA2=pl.col("Values")
.map_batches(lambda x: pl.Series(ta.sma(x.to_pandas(), length=2)))
.fill_null(strategy="backward")
.over("Group")
)
) EDITThis comment is not correct, it requires |
Thanks for the input. Ye, I didn't think the package explicitly needed a pandas series @MarcoGorelli to work. However, i tried you solution and it worked, and surprisingly with a ~30% better performance on larger datasets. |
The last example uses (It's not generating per-group values.) |
it's using hmm this doesn't look like I would have expected it to - have opened #12941 about this In [11]: df = pl.DataFrame({'a': [1,1, 2], 'b': [4,5,6]})
In [12]: df.with_columns(pl.col('b').map_batches(lambda x: x.shift()).over('a'))
Out[12]:
shape: (3, 2)
┌─────┬──────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪══════╡
│ 1 ┆ null │
│ 1 ┆ 4 │
│ 2 ┆ 5 │
└─────┴──────┘ |
Yeah, I mentioned it earlier: #12912 (comment)
(The distinction was perhaps clearer with the previous names of import pandas_ta as ta
import polars as pl
df = pl.DataFrame({
"Group": ["A", "A", "B", "C", "C", "C"],
"Values": [1, 2, 3, 4, 5, 6]
}) df.with_columns(
MA2=pl.col("Values")
.map_batches(lambda x: [print(x), pl.Series(ta.sma(x.to_pandas(), length=2))][1])
.over("Group")
)
# shape: (6,)
# Series: 'Values' [i64]
# [
# 1
# 2
# 3
# 4
# 5
# 6
# ]
# shape: (6, 3)
# ┌───────┬────────┬──────┐
# │ Group ┆ Values ┆ MA2 │
# │ --- ┆ --- ┆ --- │
# │ str ┆ i64 ┆ f64 │
# ╞═══════╪════════╪══════╡
# │ A ┆ 1 ┆ null │
# │ A ┆ 2 ┆ 1.5 │
# │ B ┆ 3 ┆ 2.5 │
# │ C ┆ 4 ┆ 3.5 │
# │ C ┆ 5 ┆ 4.5 │
# │ C ┆ 6 ┆ 5.5 │
# └───────┴────────┴──────┘ df.with_columns(
MA2=pl.col("Values")
.map_elements(lambda x: [print(x), pl.Series(ta.sma(x.to_pandas(), length=2))][1])
.over("Group")
)
# shape: (2,)
# Series: '' [i64]
# [
# 1
# 2
# ]
# shape: (1,)
# Series: '' [i64]
# [
# 3
# ]
# shape: (3,)
# Series: '' [i64]
# [
# 4
# 5
# 6
# ]
# shape: (6, 3)
# ┌───────┬────────┬──────┐
# │ Group ┆ Values ┆ MA2 │
# │ --- ┆ --- ┆ --- │
# │ str ┆ i64 ┆ f64 │
# ╞═══════╪════════╪══════╡
# │ A ┆ 1 ┆ null │
# │ A ┆ 2 ┆ 1.5 │
# │ B ┆ 3 ┆ null │
# │ C ┆ 4 ┆ null │
# │ C ┆ 5 ┆ 4.5 │
# │ C ┆ 6 ┆ 5.5 │
# └───────┴────────┴──────┘ |
I petitioned for this one to be reopened. It really seems like map_batches ought to "turn in to" map_elements when it's an over/agg. |
Thanks all for your comments! Right, so going back to the original example, the solution is indeed to use import pandas as pd
import numpy as np
import pandas_ta as ta
import polars as pl
from polars.testing import assert_series_equal
np.random.seed(42)
df = pd.DataFrame({
'Group': np.random.choice(['A', 'B', 'C'], size=50),
'Values': np.random.rand(50)
})
df = df.sort_values(by='Group')
df["Moving_Average"] = (df.groupby('Group')["Values"].transform(lambda x: ta.sma(close=x, length=2))).bfill()
# Alternative:
def My_UDF_Function(x):
return ta.sma(close=x, length=2)
dfpl = pl.from_pandas(df)
dfpl = (
dfpl.with_columns(
MA2=pl.col("Values")
.map_elements(lambda x: pl.Series(ta.sma(x.to_pandas(), length=2)))
.fill_null(strategy="backward")
.over("Group")
)
)
df["MA2"] = df.groupby('Group')["Values"].transform(My_UDF_Function).bfill()
dfpl = dfpl.with_columns(
pandas_MA2=pl.from_pandas(df['MA2'])
)
assert_series_equal(dfpl['MA2'], dfpl['pandas_MA2'], check_names=False) I'll try to update the docs |
The user guide's UDF page is still to be updated. Current version talks about map and apply these two outdated functions. |
Description
So by reading the documentation I'm still struggling on how I would implement longer UDF functions from python to polars. Would be nice if one or several examples could be dropped. Also in pandas, I used
transform
a lot compared to apply and map as you could do vectorized UDF operations similar to this:df.groupby("Group")["Relevant_Column"].transform(My_UDF_function)
. How would this work in polars if the UDF functions become slightly advanced? Would be nice to update the documentation a bit and see more examples of thisLink
https://pola-rs.github.io/polars/user-guide/expressions/user-defined-functions/
No response
The text was updated successfully, but these errors were encountered: