[SPARK-41589][PYTHON][ML][BUILD][FOLLOW-UP] Add pyspark.ml.torch to setup.py

[HyukjinKwon]

What changes were proposed in this pull request?

This PR is a followup of #39146 that adds pyspark.ml.torch to setup.py.

Why are the changes needed?

In order for PyPI users to be able to use pyspark.ml.torch package.

Does this PR introduce any user-facing change?

No, the main change has not been released yet.
It adds the package into PyPI-packaged PySpark.

How was this patch tested?

CI in pip packaging test should check the change.

[HyukjinKwon]

cc @WeichenXu123 @zhengruifeng @rithwik-db FYI

[rithwik-db]

Thank you!

[HyukjinKwon]

@Yikun, I updated Dockerfile at c4a33ec but I think after that it doesn't cache the Docker image properly in my fork (and the forks from other people too), see also https://github.com/HyukjinKwon/spark/actions/runs/3888736834/jobs/6636367051

Do you happen to know why?

[HyukjinKwon]

Okay, actually I think the caching image is sort of broken (in the master branch too https://github.com/apache/spark/actions/runs/3888703266/jobs/6636299262)

[Yikun]

@HyukjinKwon

I just saw there are some failures on cache image and write images (this should be the github ghcr flaky):

https://github.com/apache/spark/actions/runs/3879879532/jobs/6617462093#step:6:127

#34 [auth] hyukjinkwon/apache-spark-ci-image:pull,push token for ghcr.io
#34 DONE 0.0s

#33 exporting to image
#33 pushing layers 1.3s done
#33 ERROR: failed to push ghcr.io/hyukjinkwon/apache-spark-ci-image:master-3888736834: failed to copy: io: read/write on closed pipe
------
 > exporting to image:
------
ERROR: failed to solve: failed to push ghcr.io/hyukjinkwon/apache-spark-ci-image:master-3888736834: failed to copy: io: read/write on closed pipe

If github can't recover soon, we might want to use https://github.com/apache/spark/pkgs/container/spark%2Fapache-spark-github-action-image-cache/62600543?tag=master-static directly. And change

https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L59 to static image ghcr.io/apache/spark/apache-spark-github-action-image-cache:master-static (c4a33ec) first.

[Yikun]

I also retriggered the cache image job to see it had been recovered or not:

https://github.com/apache/spark/actions/runs/3879879532

[HyukjinKwon]

Related tests passed (pip packaging and linter).

Merged to master.