Mirror of Apache Spark Website
HTML Other
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.github add Pull Request Template Nov 21, 2016
_layouts Add docs for Spark 2.3.1. Jun 11, 2018
css CSS tweaks Jun 10, 2016
graphx Port wiki page Committers to committers.html, Contributing to Spark a… Nov 21, 2016
images Stash pride logo for next year Aug 14, 2018
js Point to archive site for non-current releases; remove old releases f… Jul 15, 2018
mllib Update text/wording to more "modern" Spark and more consistent. Apr 12, 2018
news spark summit eu 2018 Jul 25, 2018
releases/_posts Spark 2.2.2 docs Jul 6, 2018
screencasts Use site.baseurl, not site.url, to work with Jekyll 3.3. Require Jeky… Nov 15, 2016
site Mention correctness issues as blockers in developer docs; mention 18 … Aug 16, 2018
sql Update text/wording to more "modern" Spark and more consistent. Apr 12, 2018
streaming Use "Apache Foo" consistently; remove defunct Tachyon and EC2 refs Apr 6, 2018
talks Added PySpark talk Sep 11, 2013
.gitignore Add docs for Spark 2.3.1. Jun 11, 2018
LICENSE Add notice that site is licensed as AL2 Jan 19, 2018
README.md Updates to the release guide. Jun 12, 2018
_config.yml add merge script Apr 10, 2018
committers.md Update my affiliation Aug 13, 2018
community.md Squashed commit of the following: Mar 16, 2018
content Add possibly temporary content -> site symlink to see if it makes the… Jul 13, 2016
contributing.md Mention correctness issues as blockers in developer docs; mention 18 … Aug 16, 2018
developer-tools.md Update Yourkit usage Sep 13, 2017
doap.rdf update doap file Jun 11, 2018
documentation.md Spark 2.2.2 docs Jul 6, 2018
downloads.md [SPARK-24679] Download page should not link to unreleased code Jun 28, 2018
examples.md Patch references to docs/programming-guide.html to docs/rdd-programmi… Jul 12, 2017
faq.md Use "Apache Foo" consistently; remove defunct Tachyon and EC2 refs Apr 6, 2018
favicon.ico Initial port of Spark website from spark-project.org wordpress to Jek… Aug 23, 2013
history.md Squashed commit of the following: Mar 16, 2018
improvement-proposals.md identify SPIP jira tickets by summary as well as label Aug 29, 2017
index.md Add Alluxio back to replace Tachyon as one possible data source Aug 13, 2018
mailing-lists.md Update site look and add pages for Streaming and MLlib Jan 22, 2014
merge_pr.py Check if 'original_head' variable is defined in clean_up at merge script Jun 11, 2018
powered-by.md Small updates to Powered By page Jun 16, 2018
release-process.md Update release process to use Python 3 for Python API documentation Jul 11, 2018
research.md replace with valid url to rdd paper Sep 16, 2016
robots.txt Replace most http links with https as a best practice, where possible May 10, 2017
security.md Add CVE-2018-11770 Aug 13, 2018
sitemap.xml Patch references to docs/programming-guide.html to docs/rdd-programmi… Jul 12, 2017
third-party-projects.md For projects using names that are likely proscribed, either update to… Aug 16, 2018
trademarks.md Replace most http links with https as a best practice, where possible May 10, 2017
versioning-policy.md Mention correctness issues as blockers in developer docs; mention 18 … Aug 16, 2018

README.md

Generating the website HTML

In this directory you will find text files formatted using Markdown, with an .md suffix.

Building the site requires Jekyll 3.3.0 or newer. The easiest way to install jekyll is via a Ruby Gem. This will create a directory called site containing index.html as well as the rest of the compiled directories and files.

To install Jekyll and its required dependencies, execute sudo gem install jekyll pygments.rb and sudo pip install Pygments. See also https://github.com/apache/spark/blob/master/docs/README.md

You can generate the html website by running jekyll build in this directory. Use the --watch flag to have jekyll recompile your files as you save changes.

In addition to generating the site as HTML from the markdown files, jekyll can serve the site via a web server. To build the site and run a web server use the command jekyll serve which runs the web server on port 4000, then visit the site at http://localhost:4000.

Please make sure you always run jekyll build after testing your changes with jekyll server, otherwise you end up with broken links in a few places.

Docs sub-dir

The docs are not generated as part of the website. They are built separately for each release of Spark from the Spark source repository and then copied to the website under the docs directory. See the instructions for building those in the readme in the Spark project's /docs directory.

Pygments

We also use pygments for syntax highlighting in documentation markdown pages.

To mark a block of code in your markdown to be syntax highlighted by jekyll during the compile phase, use the following syntax:

{% highlight scala %}
// Your scala code goes here, you can replace scala with many other
// supported languages too.
{% endhighlight %}

You probably don't need to install that unless you want to regenerate the pygments CSS file. It requires Python, and can be installed by running sudo easy_install Pygments.

Merge PR

To merge pull request, use the merge_pr.py script which also squashes the commits.