Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
[SPARK-21642][CORE] Use FQDN for DRIVER_HOST_ADDRESS instead of ip address #18846
What changes were proposed in this pull request?
The patch lets spark web ui use FQDN as its hostname instead of ip address.
In current implementation, ip address of a driver host is set to DRIVER_HOST_ADDRESS. This becomes a problem when we enable SSL using "spark.ssl.enabled", "spark.ssl.trustStore" and "spark.ssl.keyStore" properties. When we configure these properties, spark web ui is launched with SSL enabled and the HTTPS server is configured with the custom SSL certificate you configured in these properties.
To avoid the exception, we should use FQDN of the driver host for DRIVER_HOST_ADDRESS.
Error message that client gets when the client accesses spark web ui:
How was this patch tested?
In my understanding, this does not affect driver/executor's heartbeat behavior since this PR only changes DRIVER_HOST_ADDRESS.
The reason I want to apply this change is the client of Spark context web UI is not Spark components. (The client will be web browser and Hadoop ResourceManager's webproxy) Since it is difficult to change these clients' behavior, I thought we should change Spark side behavior.
@zsxwing Thank you for the pointer. I tested manually, as far as I tested, Spark works as expected even if we apply this patch. I was able to confirm that driver/executor shut down when its connection lost.
On the other hand, all the tests were failed. All the reasons of test failure were different, it looks the test failed randomly. Still looking why the test failed.