[SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and Javadocs

[mateiz]

I used the sbt-unidoc plugin (https://github.com/sbt/sbt-unidoc) to create a unified Scaladoc of our public packages, and generate Javadocs as well. One limitation is that I haven't found an easy way to exclude packages in the Javadoc; there is a SBT task that identifies Java sources to run javadoc on, but it's been very difficult to modify it from outside to change what is set in the unidoc package. Some SBT-savvy people should help with this. The Javadoc site also lacks package-level descriptions and things like that, so we may want to look into that. We may decide not to post these right now if it's too limited compared to the Scala one.

Example of the built doc site: http://people.csail.mit.edu/matei/spark-unified-docs/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14270/

[markhamstra]

One other thing that you might consider is turning on the option to generate Type Hierarchy diagrams in the published scaladocs. It's not hard to do, and works well if you've got the necessary external dependencies installed and configured, but I wouldn't recommend turning the option on by default since anyone trying to build the docs on their own will see a bunch of errors if Graphviz isn't already installed.

You can see the results of the -diagrams option by looking at the scala-lang Scala API docs and expanding the Type Hierarchy section in the description of any class.

http://lamp.epfl.ch/files/content/sites/lamp/files/teaching/student-project-reports/semester/DamienObrist-SemesterProjectReport.pdf
http://stackoverflow.com/questions/13435474/how-to-generate-type-hierarchy-diagrams-with-scaladoc

[pwendell]

It might make sense to have the javadoc link go to the org.apache.spark package.

Also, it might be nice to have the Scaladoc for the org.apache.spark package now link to the other package docs (MLLib/SQL/etc) since as it stands there is no index into those packages pages.

Also - it's unrelated to this pull request, but it would be nice to group the things in the org.apache.spark package. Right now the doc is a bit overwhelming.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[mateiz]

@pwendell I've modified the Javadoc landing page to list things by group, which I think is better: http://people.csail.mit.edu/matei/spark-unified-docs/api/java/index.html. Unfortunately you can't link to the Spark package by default without losing the frames on the left, and as far as I can tell you can't modify the HTML on the landing page in any way other than creating groups.

I also saw that you need to create a separate package-info.java to have package-level docs in Java. I've added one as an example but it will be quite a bit of work to create them for every package. Later on it may be possible to modify Genjavadoc to create them from package.scala.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14278/

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14303/

[AmplabJenkins]

Build triggered.

[mateiz]

I think this is ready to go now -- see http://people.csail.mit.edu/matei/spark-unified-docs/ for updated docs. I've included some code from Michael to clean up the classes shown in Javadoc, and fixed links in our main docs to go to Javadoc. One thing still missing is injecting some JavaScript to deal with the ::Experimental:: and ::DeveloperAPI:: tags in Javadoc, but we can add that in a separate PR. Same thing with adding package-info.java for all the Java classes.

[AmplabJenkins]

Build started.

[mateiz]

@markhamstra I tried -diagrams for a bit but it complained that Graphviz failed on too many of our files, and then it gave up on generating them. It might be possible to debug it but I'd push it for later -- it sounds like this feature isn't 100% robust (my guess is that it generated .dot files that Graphviz wasn't able to parse).

[markhamstra]

Ah, ok -- it's been awhile since I tried it on the Spark tree, so it may not be as usable on the current state of the code. The diagrams would be nice to have, but certainly not necessary.

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished. All automated tests passed.

[AmplabJenkins]

Build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14307/

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14308/

[pwendell]

Okay I tested this locally and I'm going to merge it. One thing I noticed is that in catalyst/hive a ton of things are exposed that we might want to either make package-private or just exclude when we build docs (e.g. type classes, expression classes, etc). Probably @marmbrus can take a look.