[SPARK-23155][DEPLOY] log.server.url links in SHS

[gerashegalov]

What changes were proposed in this pull request?

Ensure driver/executor log availability via Spark History Server UI even if the original NodeManagers are gone. The patch is a minimum viable prototype that assumes access to NM's yarn conf with a fixed RPC port on a SHS machine because it does not add anything to the event log regarding YARN nodeId.

This is behavior is in line with MR JobHistory server, and is extremely important for job diagnostics with AWS EMR auto-scaling where NodeManagers going away is the norm.

How was this patch tested?

Manually tested with a pseudo-distirbuted YARN cluster.

[gerashegalov]

@vanzin do you mind considering this issue?

[vanzin]

I need to take a closer look at what is happening here and whether there's a cleaner way... putting YARN-specific code in not one, but two different places in core sounds a little sketchy.

While I do that, it'd be nice to have tests, and there are a few style issues (mainly closures should follow the foo { bar => ... } syntax).

[gerashegalov]

thanks for feedback, Marcelo. I was also thinking back and forth where to put this logic. Ideally YARN should provide a permalink similar to AM proxy for logs as well. However, this was faster to put in production and in line with MapReduce.

[AmplabJenkins]

Can one of the admins verify this patch?

[vanzin]

I'm closing this one since the approach in the PR linked above is going in a better direction. Hopefully it can be made more generic (not just YARN).

[gerashegalov]

👍 happy to kill the fork