[SPARK-25345][ML] Deprecate public APIs from ImageSchema #22349
Conversation
|
Test build #95749 has finished for PR 22349 at commit
|
|
Test build #95751 has finished for PR 22349 at commit
|
python/pyspark/ml/image.py
Outdated
| @@ -20,6 +20,9 @@ | |||
|
|
|||
| An attribute of this module that contains the instance of :class:`_ImageSchema`. | |||
|
|
|||
| .. note:: Deprecated in 2.4.0. Use `spark.read.format("image").load(path)` instead and | |||
There was a problem hiding this comment.
FYI, that datasource wouldn't be able to replace toNDArray and toImage which require numpy. Might be better leave a note that it needs manual conversion with NumPy API for both.
| @@ -35,6 +35,8 @@ import org.apache.spark.sql.types._ | |||
| */ | |||
| @Experimental | |||
| @Since("2.3.0") | |||
| @deprecated("use `spark.read.format(\"image\").load(path)` and this `ImageSchema` will be " + | |||
There was a problem hiding this comment.
There are other methods defined under ImageSchema that are not covered by the image data source. So we shall only deprecate readImages and leave other public methods as experimental. Same for Python.
|
Test build #95796 has finished for PR 22349 at commit
|
| @@ -207,6 +207,9 @@ def readImages(self, path, recursive=False, numPartitions=-1, | |||
| .. note:: If sample ratio is less than 1, sampling uses a PathFilter that is efficient but | |||
| potentially non-deterministic. | |||
|
|
|||
| .. note:: Deprecated in 2.4.0. Use `spark.read.format("image").load(path)` instead and | |||
There was a problem hiding this comment.
Would it make sense to issue a warning when people call this code path (e.g. warnings.warn)? We do it in some of the other Python deprecated APIs but not all.
|
Test build #95817 has finished for PR 22349 at commit
|
python/pyspark/ml/image.py
Outdated
| @@ -30,6 +30,7 @@ | |||
| from pyspark import SparkContext | |||
| from pyspark.sql.types import Row, _create_row, _parse_datatype_json_string | |||
| from pyspark.sql import DataFrame, SparkSession | |||
| import warnings | |||
There was a problem hiding this comment.
Technically builtin package should be ordered above per PEP 8. I wonder why this is not caught.
import sys
import warnings
import numpy as np
from pyspark import SparkContext
from pyspark.sql.types import Row, _create_row, _parse_datatype_json_string
from pyspark.sql import DataFrame, SparkSession
python/pyspark/ml/image.py
Outdated
| @@ -222,7 +226,8 @@ def readImages(self, path, recursive=False, numPartitions=-1, | |||
|
|
|||
| .. versionadded:: 2.3.0 | |||
| """ | |||
|
|
|||
| warnings.warn("`ImageSchema.readImage` is deprecated. " + | |||
| "Use `spark.read.format(\"image\").load(path)` instead.") | |||
There was a problem hiding this comment.
warnings.warn(..., DeprecationWarning)
|
@WeichenXu123 Could you address the comments? |
|
Test build #95831 has finished for PR 22349 at commit
|
## What changes were proposed in this pull request? Deprecate public APIs from ImageSchema. ## How was this patch tested? N/A Closes #22349 from WeichenXu123/image_api_deprecate. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: Xiangrui Meng <meng@databricks.com> (cherry picked from commit 08c02e6) Signed-off-by: Xiangrui Meng <meng@databricks.com>
|
LGTM. Merged into master and branch-2.4. Thanks! |
What changes were proposed in this pull request?
Deprecate public APIs from ImageSchema.
How was this patch tested?
N/A