[SPARK-54446][ML] FPGrowth supports local filesystem #53150
Conversation
| Using.resource( | ||
| new ObjectInputStream(new BufferedInputStream(new FileInputStream(path))) | ||
| ) { ois => | ||
| val schema = ois.readObject().asInstanceOf[StructType] |
There was a problem hiding this comment.
This uses Java deserializer which seems unsafe (risk of Remote Code Execution)
Related commit: #50922
There was a problem hiding this comment.
resort to arrow format suggested by @cloud-fan
WeichenXu123
left a comment
WeichenXu123
left a comment
There was a problem hiding this comment.
We need to address the RCE issue :)
zhengruifeng
force-pushed
the
local_fs_fpg
branch
from
b0d6e92 to
7a3db44
Compare
nit
zhengruifeng
force-pushed
the
local_fs_fpg
branch
from
cd9a177 to
8fe0d6e
Compare
| val spark = df.sparkSession | ||
| val schema = df.schema | ||
| val maxRecordsPerBatch = spark.sessionState.conf.arrowMaxRecordsPerBatch | ||
| df.queryExecution.executedPlan.execute().mapPartitionsInternal { iter => |
There was a problem hiding this comment.
Dataset already has def toArrowBatchRdd, shall we reuse it?
There was a problem hiding this comment.
we can reuse it, with some change
| val schema: StructType = df.schema | ||
| dos.writeUTF(schema.json) | ||
|
|
||
| val iter = DatasetUtils.toArrowBatchRDD(df, "UTC").toLocalIterator |
There was a problem hiding this comment.
does the arrow library provide APIs to write to local file?
holdenk
left a comment
holdenk
left a comment
There was a problem hiding this comment.
Arrow isn't intended for long term storage it's intended as a wire protocol -- I don't love using it for persisting models. I'm -0.9 on this change for now. Parquet seems like a better choice most likely.
@holdenk @cloud-fan Arrow supports Random Access Files, and it provides APIs to write to local file. But our arrow utils mainly works with serialized |
SparkML model "saveToLocal" is an internal API and it is only used in SparkConnect server side to cache model within one session, it is not used for long term storage. So it should be fine to use it here. |
|
The PR to apply the arrow file format #53232 |
5 participants
What changes were proposed in this pull request?
FPGrowth supports local filesystem
Why are the changes needed?
to make FPGrowth work with local filesystem
Does this PR introduce any user-facing change?
yes, FPGrowth will work when local saving mode is one
How was this patch tested?
updated tests
Was this patch authored or co-authored using generative AI tooling?
no