Skip to content

[SPARK-40264][ML] add batch_infer_udf function to pyspark.ml.functions #37734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leewyang wants to merge 32 commits into from
Closed

[SPARK-40264][ML] add batch_infer_udf function to pyspark.ml.functions #37734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
74cfe40
add batch_infer_udf function to pyspark.ml.functions
leewyang Aug 30, 2022
660e908
handle more error cases
leewyang Aug 31, 2022
f1022f7
pep8 formatting
leewyang Aug 31, 2022
a152eef
fix lint-python errors
leewyang Aug 31, 2022
99c2835
better caching; minor cleanup
leewyang Sep 1, 2022
67fa0b3
convert dict values to np.arrays; refactor tests
leewyang Sep 7, 2022
50a99f1
address comments
leewyang Sep 13, 2022
618d267
more typing annotations and docs
leewyang Sep 14, 2022
23be792
support col and str; add test for multiple outputs
leewyang Sep 19, 2022
0286345
remove kwargs; reformat
leewyang Sep 20, 2022
7250dbd
remove input_names arg; remove auto-conversion of multi-columms to te…
leewyang Sep 21, 2022
0a59234
remove batch dim; support sparse input_tensor_shapes
leewyang Sep 21, 2022
947e15c
add examples to pydoc; squeeze single-col scalar
leewyang Oct 4, 2022
24b4fdb
address some comments
leewyang Nov 1, 2022
ca995dd
add validation of predictions against return_type
leewyang Nov 1, 2022
960c1c7
add more validations
leewyang Nov 2, 2022
c7c411b
refactored validations; add TODOs
leewyang Nov 4, 2022
83fad6b
address more comments
leewyang Nov 7, 2022
1415175
Merge branch 'apache:master' into SPARK-40264
leewyang Nov 7, 2022
bff8e37
Merge branch 'apache:master' into SPARK-40264
leewyang Nov 8, 2022
2af57e8
address more comments
leewyang Nov 9, 2022
fd3252a
update tests
leewyang Nov 9, 2022
67ba974
convert ModelCache to LRU cache
leewyang Nov 11, 2022
019f068
fix typo
leewyang Nov 11, 2022
c206b93
address more comments
leewyang Nov 12, 2022
1aa327e
add one more check
leewyang Nov 12, 2022
d2ce77b
add PredictFunction for type-checking; rename predict_batch_fn to mak…
leewyang Jan 4, 2023
2bfb030
Merge branch 'apache:master' into SPARK-40264
leewyang Jan 5, 2023
b53757c
rename PredictFunction to PredictBatchFunction; doc updates
leewyang Jan 12, 2023
4f1a19a
remove references to struct function
leewyang Jan 12, 2023
bb1610d
revert dev code
leewyang Jan 12, 2023
97ee900
Merge branch 'apache:master' into SPARK-40264
leewyang Jan 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions dev/sparktestsupport/modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -621,8 +621,10 @@ def __hash__(self):
"pyspark.ml.tests.test_base",
"pyspark.ml.tests.test_evaluation",
"pyspark.ml.tests.test_feature",
"pyspark.ml.tests.test_functions",
"pyspark.ml.tests.test_image",
"pyspark.ml.tests.test_linalg",
"pyspark.ml.tests.test_model_cache",
"pyspark.ml.tests.test_param",
"pyspark.ml.tests.test_persistence",
"pyspark.ml.tests.test_pipeline",
Expand Down
Loading