[FEATURE REQUEST]: Support spark 3.2.2

[SparkSnail]

Is your feature request related to a problem? Please describe.
Spark 3.2.2 is released in Jul 17, 2022( https://spark.apache.org/releases/spark-release-3-2-2.html), but latest spark.net does not support that spark version yet.

Describe the solution you'd like
Do you have a plan to support Spark 3.2.2? When will you support it?

[fwaris]

spark jar is not recognizing its own version - 3.2.2. This jar is shipped with microsoft.spark v. 2.1.1

If I submit the following command in a console window:

spark-submit --driver-memory 10G --executor-memory 15G --executor-cores 8 --class org.apache.spark.deploy.dotnet.DotnetRunner --master local[*] C:\Users\fwaris1\.nuget\packages\microsoft.spark\2.1.1\jars\microsoft-spark-3-2_2.12-2.1.1.jar debug

I see this output:

...
Exception in thread "main" java.lang.IllegalArgumentException: Unsupported spark version used: '3.2.2'. Normalized spark version used: '3.2.2'. Supported versions: '3.2.0, 3.2.1'.
        at org.apache.spark.util.dotnet.Utils$.validateSparkVersions(Utils.scala:228)
...

[Niharikadutta]

@SparkSnail we are working on adding support for Spark 3.2.2 and will update this issue when this is ready

[fwaris]

@Niharikadutta support for Spark 3.2.2. is a bit urgent as Spark 3.2.1 does not run on windows due to a 'path' issue (on the Spark side). This has been fixed in 3.2.2.

As-is, its hard to do local testing and development.

Also, at this stage of Spark.net, we may consider relaxing version matching to allow Spark.net to continue but with a warning, instead of termination

[matt-zappitello]

@fwaris

Spark 3.2.1 does not run on windows due to a 'path' issue

I was able to fix this by changing my JAVA_HOME user variable that's set by Java away from the "C:\Users..." directory to the directory I installed Java in. In my case: "C:\Program Files\Java\jdk-11.0.17". This got me up and running. The only issue is that they have an extra "\bin" in there, which is not what Java defaults to putting in the C:\Users... directory.