Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
[SPARK-28736][SPARK-28735][PYTHON][ML][TESTS] Fix PySpark ML tests to pass in JDK 11 #25475
What changes were proposed in this pull request?
This PR proposes to fix both tests below:
to pass in JDK 11.
The root cause seems to be different float values being understood via Py4J. This issue also was found in #25132 before.
When floats are transferred from Python to JVM, the values are sent as are. Python floats are not "precise" due to its own limitation - https://docs.python.org/3/tutorial/floatingpoint.html.
This seems why only some tests in PySpark with floats are being failed.
So, this PR fixes it by increasing tolerance in identified test cases in PySpark.
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
Manually tested as described in JIRAs: