Skip to content

[SPARK-43660][CONNECT][PS][FOLLOWUP] Remove JVM dependency for resample #42410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
itholic wants to merge 4 commits into from
Closed

[SPARK-43660][CONNECT][PS][FOLLOWUP] Remove JVM dependency for resample #42410

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
88986e3
[SPARK-43660][CONNECT][PS][FOLLOWUP] Remove JVM dependency for resample
itholic Aug 9, 2023
a72679c
Merge branch 'master' of https://github.com/apache/spark into resampl…
itholic Aug 10, 2023
6837d0b
Fix linter
itholic Aug 10, 2023
aab4b6d
Fix linter
itholic Aug 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 9 additions & 18 deletions python/pyspark/pandas/resample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@

_builtin_table = SelectionMixin._builtin_table # type: ignore[attr-defined]

from pyspark import SparkContext
from pyspark.sql import Column, functions as F
from pyspark.sql.types import (
NumericType,
Expand All @@ -67,7 +66,6 @@
scol_for,
verify_temp_column_name,
)
from pyspark.sql.utils import is_remote
from pyspark.pandas.spark.functions import timestampdiff


Expand Down Expand Up @@ -145,22 +143,15 @@ def _agg_columns_scols(self) -> List[Column]:
def get_make_interval( # type: ignore[return]
self, unit: str, col: Union[Column, int, float]
) -> Column:
if is_remote():
from pyspark.sql.connect.functions import lit, make_interval

col = col if not isinstance(col, (int, float)) else lit(col) # type: ignore[assignment]
if unit == "MONTH":
return make_interval(months=col) # type: ignore
if unit == "HOUR":
return make_interval(hours=col) # type: ignore
if unit == "MINUTE":
return make_interval(mins=col) # type: ignore
if unit == "SECOND":
return make_interval(secs=col) # type: ignore
else:
sql_utils = SparkContext._active_spark_context._jvm.PythonSQLUtils
col = col._jc if isinstance(col, Column) else F.lit(col)._jc
return sql_utils.makeInterval(unit, col)
col = col if not isinstance(col, (int, float)) else F.lit(col)
if unit == "MONTH":
return F.make_interval(months=col)
if unit == "HOUR":
return F.make_interval(hours=col)
if unit == "MINUTE":
return F.make_interval(mins=col)
if unit == "SECOND":
return F.make_interval(secs=col)

def _bin_timestamp(self, origin: pd.Timestamp, ts_scol: Column) -> Column:
key_type = self._resamplekey_type
Expand Down