Skip to content
Permalink
Browse files
[BAHIR-50] Publish extensions documentation to website
Provide a script to gather extension documentation based
on its README.md and publish it as part of the current
documentation on the website.

When a release is performed, the "current" folder should
be copied to the release name (e.g. 2.0.0) as a documentation
snapshot.
  • Loading branch information
lresende committed Aug 10, 2016
1 parent d09e63b commit f8d356989c150427f4d85c1ff12e2f06864f4d63
Showing 10 changed files with 264 additions and 2 deletions.
@@ -37,6 +37,22 @@ running from within the directory:
```


## Update extensions documentation

To update the current documentation contents, use the update-doc script.
This script will update the documentation contents based on the README.md
files from the source of each extension.

```
1. update-doc.sh
2. git commit -a -m"My updates"
3. git push
4. ./publish.sh
5. git push origin asf-site
```



## Publishing to live site

Bahir is using [gitpubsub](http://www.apache.org/dev/gitpubsub.html) for publishing the website,
@@ -48,8 +64,9 @@ shell script.
```
1. Make your changes
2. git commit -a -m"My updates"
3. ./publish.sh
4. git push origin asf-site
3. git push
4. ./publish.sh
5. git push origin asf-site
```

Within a few minutes, gitpubsub should kick in and you'll be able to
@@ -45,6 +45,11 @@ topnav:
- title: Project Committers
url: /community-members

- title: Documentation
subcategories:
- title: Current
url: /docs/spark/current/documentation

- title: GitHub
url: https://github.com/apache/bahir

@@ -0,0 +1 @@
spark*.md
@@ -0,0 +1,46 @@
---
layout: page
title: Extensions for Apache Spark
description: Extensions for Apache Spark
group: nav-right
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

{% include JB/setup %}

### Apache Bahir Extensions for Apache Spark

<br/>

#### Structured Streaming Data Sources

[MQTT data source](/docs/spark/current/spark-sql-streaming-mqtt)

<br/>

#### Discretized Streams (DStreams) Connectors

[Akka connector](/docs/spark/current/spark-streaming-akka)

[MQTT connector](/docs/spark/current/spark-streaming-mqtt)

[Twitter connector](/docs/spark/current/spark-streaming-twitter)

[ZeroMQ connector](/docs/spark/current/spark-streaming-zeromq)
@@ -0,0 +1,27 @@
---
layout: page
title: Spark Structured Streaming MQTT
description: Spark Structured Streaming MQTT
group: nav-right
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

{% include JB/setup %}

@@ -0,0 +1,26 @@
---
layout: page
title: Spark Streaming Akka
description: Spark Streaming Akka
group: nav-right
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

{% include JB/setup %}
@@ -0,0 +1,27 @@
---
layout: page
title: Spark Structured Streaming MQTT
description: Spark Structured Streaming MQTT
group: nav-right
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

{% include JB/setup %}

@@ -0,0 +1,26 @@
---
layout: page
title: Spark Streaming Twitter
description: Spark Streaming Twitter
group: nav-right
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

{% include JB/setup %}
@@ -0,0 +1,26 @@
---
layout: page
title: Spark Streaming ZeroMQ
description: Spark Streaming ZeroMQ
group: nav-right
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->

{% include JB/setup %}
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to you under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

set -e

BASE_DIR=$(pwd)
WEBSITE_DOC_DIR=$BASE_DIR/site/docs/spark/current
BAHIR_SOURCE_DIR=$BASE_DIR/target/bahir

function checkout_code {
# Checkout code
rm -rf target
mkdir target
cd target
rm -rf bahir
git clone https://git-wip-us.apache.org/repos/asf/bahir.git
cd bahir
git checkout $GIT_REF
git_hash=`git rev-parse --short HEAD`
echo "Checked out Bahir git hash $git_hash"

git clean -d -f -x

cd "$BASE_DIR" #return to base dir
}

checkout_code

rm $WEBSITE_DOC_DIR/spark*.md

cp $WEBSITE_DOC_DIR/spark-sql-streaming-mqtt.template $WEBSITE_DOC_DIR/spark-sql-streaming-mqtt.md
cat $BAHIR_SOURCE_DIR/sql-streaming-mqtt/README.md >> $WEBSITE_DOC_DIR/spark-sql-streaming-mqtt.md

cp $WEBSITE_DOC_DIR/spark-streaming-akka.template $WEBSITE_DOC_DIR/spark-streaming-akka.md
cat $BAHIR_SOURCE_DIR/streaming-akka/README.md >> $WEBSITE_DOC_DIR/spark-streaming-akka.md

cp $WEBSITE_DOC_DIR/spark-streaming-mqtt.template $WEBSITE_DOC_DIR/spark-streaming-mqtt.md
cat $BAHIR_SOURCE_DIR/streaming-mqtt/README.md >> $WEBSITE_DOC_DIR/spark-streaming-mqtt.md

cp $WEBSITE_DOC_DIR/spark-streaming-twitter.template $WEBSITE_DOC_DIR/spark-streaming-twitter.md
cat $BAHIR_SOURCE_DIR/streaming-twitter/README.md >> $WEBSITE_DOC_DIR/spark-streaming-twitter.md

cp $WEBSITE_DOC_DIR/spark-streaming-zeromq.template $WEBSITE_DOC_DIR/spark-streaming-zeromq.md
cat $BAHIR_SOURCE_DIR/streaming-zeromq/README.md >> $WEBSITE_DOC_DIR/spark-streaming-zeromq.md

set +e

0 comments on commit f8d3569

Please sign in to comment.