This is a place to put s2i images and utilities for spark application builders for openshift
Shell Scala Other
Clone or download
elmiko and tmckayus add tests for py36 images
This change add the specific tests for the python 36 build and build dc.
It also enables the travis testing for python36 and renames the original
pyspark template test directory to pyspark-py27.
Latest commit cff35fd Jun 26, 2018
Permalink
Failed to load latest commit information.
hack Make default timeout five minutes for tests Sep 26, 2017
hooks Update make files and hooks/pre-commit Nov 8, 2017
java-build Update the oshinko CLI version to v0.5.3 Jun 6, 2018
modules Set group permission on /etc/passwd to support spark entrypoint Jun 6, 2018
pyspark-build change the description for the pyspark image Jul 2, 2018
pyspark-py36-build add initial python36 implementation Jun 19, 2018
scala-build Update the oshinko CLI version to v0.5.3 Jun 6, 2018
templates change the description for the pyspark image Jul 2, 2018
test add tests for py36 images Jul 2, 2018
.gitignore Making images buildable by concreate as well as docker autobuilds. Feb 9, 2018
.travis.yml add tests for py36 images Jul 2, 2018
Jenkinsfile Jenkinsfile - restrict Jenkins label on which node tests may run Mar 29, 2018
LICENSE Add the license file May 9, 2017
Makefile add tests for py36 images Jul 2, 2018
Makefile.java Refactor release.sh into two general utility scripts Feb 26, 2018
Makefile.pyspark Refactor release.sh into two general utility scripts Feb 26, 2018
Makefile.pyspark-py36 add initial python36 implementation Jun 19, 2018
Makefile.scala Refactor release.sh into two general utility scripts Feb 26, 2018
README.md typo May 29, 2018
change-yaml.sh Remove obsolete options from change_yaml.sh Jun 6, 2018
image.java.yaml Update the oshinko CLI version to v0.5.3 Jun 6, 2018
image.pyspark-py36.yaml add initial python36 implementation Jun 19, 2018
image.pyspark.yaml change the description for the pyspark image Jul 2, 2018
image.scala.yaml Update the oshinko CLI version to v0.5.3 Jun 6, 2018
make-build-dirs.sh add initial python36 implementation Jun 19, 2018
release-templates.sh changes to md5 and set -ex May 21, 2018

README.md

Build Status Docker python build Docker java build

oshinko-s2i

This is a place to put s2i images and utilities for Apache Spark application builders for OpenShift.

Building the s2i images

The easiest way to build the s2i images is to use the makefiles provided:

# To build all images
$ make

# To build images individually
$ make -f Makefile.pyspark
$ make -f Makefile.java
$ make -f Makefile.scala

The default repository for the image can be overridden with the LOCAL_IMAGE var:

$ LOCAL_IMAGE=myimage make -f Makefile.pyspark

Modifying dependencies in the image yaml files

The concreate tool generates the image context directories based on the content of the image.*.yaml files.

A script has been provided to make altering the image.*.yaml files simpler. It handles modifying the specified versions of oshinko, spark, scala, and sbt. Run this for more details

$ change-yaml.sh -h

Remaking image context directories when things change

The image context directories are generated with the concreate tool and contain the artifacts needed to build the images. They are:

* pyspark-build
* java-build
* scala-build

If the yaml files used by concreate change (ie image.*.yaml) or the content included in an image changes (essentially anything under modules/), the image context directories need to be rebuilt.

Rebuilding context directories for an upstream pull request

If the changes being made are part of a PR to github.com/radanalyticsio/oshinko-s2i then all of the build directories should be generated from scratch. The best way to do this is with the make-build-dirs.sh script

$ make-build-dirs.sh

This will recreate the context directories starting from a clean environment, make sure any tarballs are truncated for github, and add all of the changes to the commit.

Rebuilding a particular context directory for testing/development

To regenerate a particular context directory, like pyspark-build, do this

$ make -f Makefile.pyspark clean-context context

To regenerate the context directory and also build the image, do this

$ make -f Makefile.pyspark clean build

Git pre-commit hook

The hooks/pre-commit hook can be installed in a local repo to prevent commits with non-zero length tarballs in the image build directories or to warn when changes have been made to yaml files or scripts but the image build directories have not changed. To install the hook locally do something like this:

$ cd .git/hooks
$ ln -s ../../hooks/pre-commit pre-commit

This is recommended, since the CI tests will reject a pull request with non-zero length tarballs anyway. Save some time, install the hook.

Using release-templates.sh

The templates included in this repository always reference the latest s2i images. Those images may change during the normal course of development.

The release-templates.sh script can be used to create local versions of the templates that reference s2i images from a particular oshinko release. You may want to use this script to guarantee that you are using a stable image. For example:

$ ./release-templates.sh v0.2.5

Successfully wrote templates to release_templates/ with version tag v0.2.5

grep radanalyticsio/radanalytics-.*spark:v0.2.5 *

release_templates/javabuilddc.json:            "name": "radanalyticsio/radanalytics-java-spark:v0.2.5"
release_templates/javabuild.json:              "name": "radanalyticsio/radanalytics-java-spark:v0.2.5"
release_templates/pysparkbuilddc.json:         "name": "radanalyticsio/radanalytics-pyspark:v0.2.5"
release_templates/pysparkbuild.json:           "name": "radanalyticsio/radanalytics-pyspark:v0.2.5"
release_templates/scalabuilddc.json:           "name": "radanalyticsio/radanalytics-scala-spark:v0.2.5"
release_templates/scalabuild.json:             "name": "radanalyticsio/radanalytics-scala-spark:v0.2.5"

$ oc create -f release_templates/pysparkbuilddc.json

MacOS Tips

For MacOS you will also need to download these tools: gsed and truncate. You can install these using homebrew and these commands:

brew install truncate
brew install gnu-sed