DockerHub public images - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr / SolrCloud, Presto, Apache Drill, Nifi, Spark, Superset, H2O, Mesos, Serf, Consul, Riak, Alluxio, Jython, Advanced Nagios Plugins Collection / PyTools / Tools repos on CentOS / Ubuntu / Debian / Alpine
Switch branches/tags
Nothing to show
Clone or download
Latest commit 9e7953e Oct 25, 2018
Permalink
Failed to load latest commit information.
alluxio updated Dockerfile Oct 14, 2018
alpine-dev updated all references to DevOps Python & Perl Tools Aug 15, 2018
alpine-github updated all references to DevOps Python & Perl Tools Aug 15, 2018
apache-drill updated get_versions Oct 7, 2018
bash-tools @ 205f6fa updated bash-tools Jul 25, 2018
cassandra-dev updated Dockerfile Oct 14, 2018
centos-dev updated all references to DevOps Python & Perl Tools Aug 15, 2018
centos-github updated all references to DevOps Python & Perl Tools Aug 15, 2018
centos-java stripped out Makefiles to use Makefile.in Oct 17, 2017
centos-scala stripped out Makefiles to use Makefile.in Oct 17, 2017
collectd replaced tabs with spaces May 7, 2018
consul-dev Revert "updated Dockerfile" Oct 24, 2018
consul updated get_versions Oct 24, 2018
debian-dev updated all references to DevOps Python & Perl Tools Aug 15, 2018
debian-github updated all references to DevOps Python & Perl Tools Aug 15, 2018
debian-java stripped out Makefiles to use Makefile.in Oct 17, 2017
devops-perl-tools updated Dockerfile Aug 25, 2018
devops-python-tools updated Dockerfile Aug 25, 2018
h2o updated post build hooks Jul 26, 2018
hadoop-dev added hadoop/get_versions, hadoop-dev/get_versions Oct 14, 2018
hadoop added hadoop/get_versions, hadoop-dev/get_versions Oct 14, 2018
hbase-dev updated hbase-dev/Dockerfile Oct 7, 2018
hbase updated docker-compose.yml Oct 9, 2018
jython updated post build hooks Jul 26, 2018
kafka updated get_versions Oct 14, 2018
mesos updated post build hooks Jul 26, 2018
nagios-plugin-kafka moved all README.md so DockerHub will get the top level repo README i… Nov 30, 2017
nagios-plugins-alpine moved all README.md so DockerHub will get the top level repo README i… Nov 30, 2017
nagios-plugins-centos moved all README.md so DockerHub will get the top level repo README i… Nov 30, 2017
nagios-plugins-debian moved all README.md so DockerHub will get the top level repo README i… Nov 30, 2017
nagios-plugins-ubuntu moved all README.md so DockerHub will get the top level repo README i… Nov 30, 2017
nifi added get_versions Oct 7, 2018
old added .aws and .env to .dockerignore Sep 12, 2017
presto-admin updated post build hooks Jul 26, 2018
presto-cli-dev updated post build hooks Jul 26, 2018
presto-cli updated post build hooks Jul 26, 2018
presto-dev added presto-dev/get_versions Oct 7, 2018
presto updated post build hooks Jul 26, 2018
rabbitmq-cluster updated post build hooks Jul 26, 2018
ranger updated post build hooks Jul 26, 2018
riak-dev updated post build hooks Jul 26, 2018
riak updated post build hooks Jul 26, 2018
serf updated Dockerfile Oct 14, 2018
solr updated Dockerfile Oct 14, 2018
solrcloud-dev updated Dockerfile Oct 14, 2018
solrcloud updated Dockerfile Oct 14, 2018
spark-apps stripped out Makefiles to use Makefile.in Oct 17, 2017
spark added get_versions Oct 7, 2018
spotify-tools stripped out Makefiles to use Makefile.in Oct 17, 2017
superset moved all README.md so DockerHub will get the top level repo README i… Nov 30, 2017
tachyon updated post build hooks Jul 26, 2018
tcollector updated post build hooks Jul 26, 2018
teamcity updated post build hooks Jul 26, 2018
tests merged check_for_new_versions.sh into one script Oct 7, 2018
ubuntu-dev updated Dockerfile Aug 20, 2018
ubuntu-github updated all references to DevOps Python & Perl Tools Aug 15, 2018
ubuntu-java stripped out Makefiles to use Makefile.in Oct 17, 2017
zookeeper-dev updated Dockerfile Oct 14, 2018
zookeeper updated Dockerfile Oct 14, 2018
.gitignore added .gitignore Sep 15, 2017
.gitmodules added bash-tools submodule Jul 25, 2016
.travis.yml added cache Oct 17, 2017
LICENSE updated all references to DevOps Python & Perl Tools Aug 15, 2018
Makefile added commit-hooks make target Jul 26, 2018
Makefile.in updated Makefile.in Oct 14, 2018
README.md updated README.md Aug 15, 2018

README.md

Hadoop, Big Data, NoSQL & DevOps Docker Images

Build Status Codacy Badge GitHub stars GitHub forks Platform DockerHub

Contains 45 DockerHub repos with 340+ tags, many different versions of standard official open source software, see Full Inventory futher down.

These docker images are tested by hundreds of tools and also used in the full functional test suites of various other GitHub repos.

Overview - this repo contains:

  • Hadoop & Big Data ecosystem technologies (Spark, Kafka, Presto, Drill, Nifi, ZooKeeper)
  • NoSQL datastores (HBase, Cassandra, Riak, SolrCloud)
  • OS & development images (Alpine, CentOS, Debian, Ubuntu)
  • DevOps tech, anything cool and open source (RabbitMQ Cluster, Mesos, Consul)
  • My GitHub repos containing hundreds of tools related to these technologies with all dependencies pre-built

These images are all available pre-built on My DockerHub - https://hub.docker.com/u/harisekhon/.

Quality and Testing - this repo has entire test suites run against it from various GitHub repositories to validate the docker images' functionality, branches vs tagged versions align, latest contains correct version from master branch, syntax checks covering all common build and file formats (Make/JSON/CSV/INI/XML/YAML configurations) etc.

These are reusable tests that can anybody can implement and can be found in my DevOps Python Tools and Bash-Tools repos as well as the Advanced Nagios Plugins Collection which contains hundreds of technology specific API-level test programs to ensure the docker images are functioning as intended.

Continuous Integration in run on this and adjacent repos that form a bi-directional validation between these docker images and several other repositories full of hundreds of programs. All of this is intended to keep the quality of this repo as high as possible.

Hari Sekhon

Big Data Contractor, United Kingdom

(ex-Cloudera, Hortonworks Consultant)

https://www.linkedin.com/in/harisekhon

(you're welcome to connect with me on LinkedIn)

Ready to run Docker images

docker search harisekhon
docker run harisekhon/nagios-plugins

To see more than the 25 DockerHub repos limited by docker search (docker issue 23055) I wrote dockerhub_search.py using the DockerHub API, available in my DevOps Python Tools github repo and as a pre-built docker image:

docker run harisekhon/pytools dockerhub_search.py harisekhon

There are lots of tagged versions of official software in my repos to allow development testing across multiple versions, usually more versions than available from the official repos (and new version updates available on request, just raise a GitHub issue).

DockerHub tags are not shown by docker search (docker issue 17238) so I wrote dockerhub_show_tags.py available in my DevOps Python Tools github repo and as a pre-built docker image - eg. to see an organized list of all CentOS tags:

docker run harisekhon/pytools dockerhub_show_tags.py centos

For service technologies like Hadoop, HBase, ZooKeeper etc for which you'll also want port mappings, each directory in the GitHub project contains both a standard docker-compose configuration as well as a make run shortcut (which doesn't require docker-compose to be installed) - either way you don't have to remember all the command line switches and port number specifics:

cd zookeeper
docker-compose up

or for technologies with interactive shells like Spark, ZooKeeper, HBase, Drill, Cassandra where you want to be dropped in to an interactive shell, use the make run shortcut instead:

cd zookeeper
make run

which is much easier to type and remember than the equivalent bigger commands like:

docker run -ti -p 2181:2181 harisekhon/zookeeper

and avoid this for more complex services like Hadoop / HBase:

docker run -ti -p 2181:2181 -p 8080:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 -p 16000:16000 -p 16010:16010 -p 16201:16201 -p 16301:16301 harisekhon/hbase
docker run -ti -p 8020:8020 -p 8032:8032 -p 8088:8088 -p 9000:9000 -p 10020:10020 -p 19888:19888 -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075 -p 50090:50090 harisekhon/hadoop

Full Inventory:

Official Standard Open Source Technologies:

More specific information can be found in the readme page under each respective directory in the Dockerfiles git repo.

Repos suffixed with -dev are the official technologies + development & debugging tools + my github repos with all dependencies pre-built.

My GitHub Repos (with all libs + deps pre-built):
  • Advanced Nagios Plugins Collection - 350+ nagios plugins for every Hadoop distribution and every major NoSQL technology - Hadoop, Redis, Elasticsearch, Solr, HBase, Cassandra & DataStax OpsCenter, MongoDB, MySQL, Kafka, Riak, Memcached, Couchbase, CouchDB, Mesos, Spark, Neo4j, Datameer, H2O, WanDisco, Yarn, HDFS, Impala, Apache Drill, Presto, ZooKeeper, Cloudera, Hortonworks, MapR, IBM BigInsights, Infrastructure - Linux, DNS, Whois, SSL Certs etc

    • Tags:
      • nagios-plugins:latest (centos)
      • nagios-plugins:alpine
      • nagios-plugins:centos
      • nagios-plugins:debian
      • nagios-plugins:ubuntu
  • DevOps Python Tools - 50+ tools for Hadoop, Spark, Pig, Ambari Blueprints, AWS CloudFormation, Linux, Data Converters & Validators (Avro/Parquet/JSON/CSV/XML/YAML), Elasticsearch, Solr, IPython - CLI tools

  • DevOps Perl Tools - 25+ tools for Hadoop, Hive, Solr, Linux, SQL, Ambari, Datameer, Web and various Linux CLI Tools

  • Spotify Tools - Backup & Play Automation: Spotify Lookup - converts Spotify URIs to 'Artist - Track' form by querying the Spotify Metadata API. Spotify Cmd - command line control of Spotify on Mac via AppleScript for automation, auto timed track flick through etc.

  • CentOS + all Github repos pre-built

  • Debian + all Github repos pre-built

  • Ubuntu + all Github repos pre-built

  • Alpine + all Github repos pre-built

Base Images:

Dev images:

  • CentOS latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect, EPEL etc.
  • Debian latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
  • Ubuntu latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
  • Alpine latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
Base Images of Java / Scala:

All builds use OpenJDK with jre and jdk numbered tags. See this article below for why it might be illegal to bundle Oracle Java (and why no Linux distributions do this either):

https://www.javacodegeeks.com/2016/03/running-java-docker-youre-breaking-law.html

  • CentOS latest combinations of Java 7/8 and Scala 2.10/2.11
  • Debian latest with Java 7
  • Ubuntu 14.04 with Java 7
  • Ubuntu latest with Java 8, 9

Build from Source

All images come pre-built on DockerHub but if you want to compile from source for any reason such as developing improvements, I've made this easy to do:

git clone https://github.com/harisekhon/Dockerfiles

cd Dockerfiles

To build all Docker images, just run the make command at the top level:

make

To build a specific Docker image, enter its directory and run make:

cd nagios-plugins

make

You can also build a specific version by checking out the git branch for the version and running the build:

cd consul
git checkout consul-0.9
make

or build all versions of a given software project like so:

cd hadoop
make build-versions

See the top level Makefile as well as the Makefile.in which is sourced per project with any project specific overrides in the <project_directory>/Makefile.

Support

Please raise tickets for issues and improvements at https://github.com/harisekhon/dockerfiles/issues