[SPARK-23386][DEPLOY] enable direct application links in SHS before replay

[gerashegalov]

What changes were proposed in this pull request?

Enable direct job links already in the scan thread before full replay. Otherwise, direct job links might not be available for hours.

How was this patch tested?

Test with a deploy on multiple 10k apps. This is currently a prototype for YARN, but should be generalizable.

[vanzin]

If doing this, it would be cleaner to do it as part of parsing the logs. e.g., if you make AppListingListener write the app info to the store when interesting events happen, that would be much better and less race-prone.

[gerashegalov]

@vanzin what do you mean by "as part of parsing the logs"? This PR is about avoiding the long wait for eventLogs to be read from a remote filesystem, and being parsed.

[vanzin]

True. Still, to be able to do that, you're hardcoding YARN-isms into the code, e.g., how application IDs look, so that you can create a "fake" application entry that will, hopefully, eventually match the actual contents of the log file.

What you're trying here is a stop-gap fix for SPARK-6951. I was hoping we could have an actual solution to that problem. I thought about skipping data (instead of the current code that still reads the data, just doesn't process events it doesn't care about), but couldn't figure out how to make that work with compression on.

There have been suggestions thrown around, like having Spark write a summary file side-by-side with the event log, for the SHS to consume. But that doesn't help existing event logs.

If you'd like to go down this path I'd suggest forgetting about the whole app id parsing thing, and creating actual, fake entries for these logs that clearly indicate they're fake and temporary, and cleaning them up once the log file is parsed. You could do that by creating the fake entry (if the app's entry doesn't exist yet) and providing it to the parsing task, so that once it's done it cleans up the temp entry before writing the real one.

[gerashegalov]

Thanks for suggestions, I will look into them. I agree that a solution into the right direction will definitely involve changing the write call path. I did not go down this path because I have no control over my customer's Spark versions (at least for now).

There is another long standing issue with FsHistoryProvider. It uses rename which is inefficient on S3.

[AmplabJenkins]

Can one of the admins verify this patch?

[vanzin]

There's a fix for SPARK-6951 that should cover a lot of ground here. If there's extra stuff you want to do, please update the PR (that will re-open it).