[SYSTEMDS-501] Documentation webpage framework

This commit moves the documentation back to the master branch.
It also clean up the previous documentation (by deleting it).
Such that we have a clean start.

Furthermore this commit, merges back the documentation on master,
into the webpage documentation.

Related PRs: apache#949 apache#922

Discussion Mails:
https://tinyurl.com/yal7fd3r
https://preview.tinyurl.com/yal7fd3r

Author: Baunsgaard

.gitignore

@@ -61,7 +61,8 @@ conf/systemds-env.sh
 
 # Documentation artifacts
 docs/_site
-docs/api
+# TODO Make the API auto generate and relocate into this api folder for webpage
+# docs/api
 
 # Test Artifacts
 src/test/scripts/**/*.dmlt

bin/README.md

@@ -19,78 +19,5 @@ limitations under the License.
 
 # Scripts to run SystemDS
 
-This directory contains scripts to launch systemds.
-
-## Setting SYSTEMDS_ROOT environment variable
-
-In order to run SystemDS from your development directory and leave the
-SystemDS files untouched, the following setup could be used (example for bash):
-The settings are the same if you download a release of SystemDS.
-
-The following example works if you open an terminal at the root of the downloaded release,
-or a cloned repository. You can also change the `$(pwd)` with the full path to the folder.
-
-```bash
-export SYSTEMDS_ROOT=$(pwd)
-export PATH=$SYSTEMDS_ROOT/bin:$PATH
-```
-
-It can be beneficial to enter these into your `~/.profile` for linux,
-(but remember to change `$(pwd` to the full folder path)
-or your environment variables in windows to enable reuse between terminals and restarts.
-
-## Hello World example
-
-To quickly verify that the system is setup correctly.
-You can run a simple hello world, using the launch script.
-
-first open an terminal and go to an empty folder, then execute the following.
-
-```bash
-# Create a hello World script
-echo 'print("HelloWorld!")' > hello.dml
-# Execute hello world Script
-systemds hello.dml
-# Remove the hello.dml
-rm hello.dml
-```
-
-## Running a first example
-
-To see SystemDS in action a simple example using the `Univar-stats.dml`
-script can be executed. This example is taken from the
-[SystemML documentation](http://apache.github.io/systemml/standalone-guide).
-The relevant commands to run this example with SystemDS will be listed here.
-See their documentation for further details.  
-
-### Example preparations
-
-```bash
-# download test data
-wget -P data/ http://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data
-
-# generate a metadata file for the dataset
-echo '{"rows": 306, "cols": 4, "format": "csv"}' > data/haberman.data.mtd
-
-# generate type description for the data
-echo '1,1,1,2' > data/types.csv
-echo '{"rows": 1, "cols": 4, "format": "csv"}' > data/types.csv.mtd
-```
-
-### Executing the DML script
-
-```shell script
-bin/systemds Univar-Stats.dml -nvargs X=data/haberman.data TYPES=data/types.csv STATS=data/univarOut.mtx CONSOLE_OUTPUT=TRUE
-```
-
-## Using Intel MKL native instructions
-
-To use the MKL acceleration download and install the latest MKL library from [1],
-set the environment variables with the MKL-provided script `$ compilervars.sh intel64` and set
-the option `sysds.native.blas` in `SystemDS-config.xml`.
-
-[1]: https://software.intel.com/mkl "Intel Math Kernel Library"
-
-## Further reading
-
-More documentation is available in the [docs directory of our github repository](/dev/docs/README.md)
+This directory contains scripts to launch SystemDS.
+For details look at: [RunSystemDS](/docs/site/run.md).

dev/docs/CodeStyle_eclipse.xml dev/CodeStyle_eclipse.xmldev/docs/CodeStyle_eclipse.xml renamed to dev/CodeStyle_eclipse.xml

@@ -1,4 +1,22 @@
 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+-->
 <profiles version="16">
 <profile kind="CodeFormatterProfile" name="SystemDS" version="16">
 <setting id="org.eclipse.jdt.core.formatter.insert_space_after_ellipsis" value="insert"/>

dev/docs/Tasks.txt dev/Tasks.txtdev/docs/Tasks.txt renamed to dev/Tasks.txt

@@ -1,3 +1,22 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% end comment %}
+-->
+
 ---
 GENERAL NOTES:
  * Assign IDs for epics by default with a key range of 10
@@ -321,5 +340,8 @@ SYSTEMDS-410 Lineage Tracing, Reuse and Integration II
  * 412 Robust lineage tracing (non-recursive, parfor)                 OK
  * 413 Cache and reuse MultiReturnBuiltin instructions                OK
 
+SYSTEMDS-500 Documentation Webpage Reintroduction
+ * 501 Make Documentation webpage framework                           OK
+
 Others:
  * Break append instruction to cbind and rbind 

dev/docs/README.md

@@ -0,0 +1,23 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+source "https://rubygems.org"
+gem "jekyll"
+gem "rouge"

docs/Gemfile

@@ -0,0 +1,91 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# Apache SystemDS Documentation
+
+The primary SystemDS documentation is written in markdown (*.md) and can be converted to HTML using
+[Jekyll](http://jekyllrb.com).
+
+The Documentation is separated into different parts by sub folders.
+
+``` py
+.
+├── _layouts
+│   ├── global.html # The Default content layout and html file.
+├── _site # If run locally a _site folder is generated containing the compiled site.
+├── _config.yml # The configuration for jekyll to construct the site.
+├── Gemfile.yml # The dependency list for the documentation
+├── index.md #  The main entry File
+├── _layouts
+│   ├── global.html
+├── css # The style files folder
+├── img # The Images folder
+├── js  # The JavaScript folder
+├── site # The different pages in the documentation
+├── api # Contains autogenerated files. Don't touch manually
+│   ├── java # The generated Java docs from source code.
+│   ├── python # The generated Python docs from source code.
+```
+
+## Ubuntu Install
+
+1. Install Ruby development environment [Jekyll on Ubuntu](https://jekyllrb.com/docs/installation/ubuntu/).
+2. Install Jekyll inside the Ruby environment.
+
+   - `gem install jekyll jekyll-redirect-from bundler pygments.rb`
+   - To do this you might need to change permissions on folders `/var/lib/gems` and `/var/lib/gems/2.5.0`.
+   - The Pygments package is optional, and does require sudo permissions.
+
+3. Install python dependencies (Optionally).
+
+   - `pip install Pygments`
+
+4. Launch the Documentation locally
+
+   - `jekyll serve -w`
+   - This is done from the root of the gh-pages branch of the system.
+   - The serving will per default be on localhost port 4000 [Link](http://localhost:4000)
+
+### Update API
+
+To update the API documentation run the `updateAPI.sh` file.
+Note that you have to install both the Java and Python -doc dependencies.
+
+## Mac Install
+
+Jekyll (and optionally Pygments) can be installed on the Mac OS in the following manner.
+
+```bash
+brew install ruby
+gem install jekyll
+gem install jekyll-redirect-from
+gem install bundler
+brew install python
+pip install Pygments
+gem install pygments.rb
+```
+
+To generate SystemDS documentation in HTML, navigate to the ```docs``` folder, the root directory of the
+documentation. From there, you can have Jekyll convert the markdown files to HTML. If you run in server mode,
+Jekyll will serve up the generated documentation by default at <http://127.0.0.1:4000>. Modifications
+to *.md files will be converted to HTML and can be viewed in a web browser.
+
+```bash
+jekyll serve -w
+```

docs/README.md

@@ -0,0 +1,42 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+highlighter: rouge
+markdown: kramdown
+
+kramdown:
+  entity_output: numeric
+
+include:
+  - img
+  - api/python/
+
+exclude:
+  - README.md
+  - updateAPI.sh
+
+# These allow the documentation to be updated with newer releases
+SYSTEMDS_VERSION: 2.0.0-SNAPSHOT
+
+# if 'analytics_on' is true, analytics section will be rendered on the HTML pages
+analytics_on: true
+analytics_provider: google_universal
+analytics_google_universal_tracking_id : UA-71553733-1