Skip to content
Permalink
Browse files
Added support for Kubernetes
  • Loading branch information
michellebeard committed Apr 18, 2018
1 parent 9f80900 commit a575d479351a159a034c2b927963a78cc1f3802b
Showing 41 changed files with 1,747 additions and 94 deletions.
@@ -1,3 +1,4 @@
.idea
release/incubator-*
release/apache-*
.sass-cache
@@ -1,5 +1,5 @@
SensSoft Docker Containers and Website
--------------------------------------

Specific build instructions located in docker and site folders.
Specific build instructions located in docker, kubernetes, release and site folders.

@@ -0,0 +1,68 @@
How to Build SensSoft Docker Containers
---------------------------------------

1. Install [``Docker``](http://docker.com) on your machine. Require Docker 1.7 and above.

1. Install ``docker-compose`` in an virtual environment.
Full instructions can be found [``here``](https://docs.docker.com/compose/install/).

```bash
$ python3 -m venv env
$ source env/bin/activate
$ pip install -r requirements.txt
```

1. Before launching the Docker containers, ensure your ``vm_max_map_count``
kernel setting is set to at least 262144.
Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions.

```bash
# Example for Linux systems
$ sysctl -w vm.max_map_count=262144
```

1. Start Elasticsearch cluster:

```bash
$ docker-compose -f docker-compose.cluster.yaml up -d --scale elasticsearch=3 elasticsearch
$ docker-compose -f docker-compose.cluster.yaml up -d loadbalancer
```

The loadbalancer node exposes port 9200 on localhost and is the only node
that has HTTP enabled. Services such as Kibana and Logstash connect to the
loadbalancer node directly. Loadbalancer accepts requests from Kibana and Logstash
and balances them across the elasticsearch worker nodes. The elasticsearch
worker nodes communicate to each other and the loadbalancer via TCP on port 9300.

<aside class="warning">
Starting an elasticsearch cluster is not recommended on a single server. This
is just for demonstration purposes only. Please refer to our [Kubernetes] guide to
deploy an Elasticsearch cluster.
</aside>

1. Confirm cluster state:
```bash
$ curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "SensSoft",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
```
Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3.

1. Follow remaining instructions in README.md, starting at #6.

© Copyright 2016 The Charles Stark Draper Laboratory, Inc. All rights reserved.
@@ -1,7 +1,7 @@
How to Build SensSoft Docker Containers
---------------------------------------

1. Install [``Docker``](http://docker.com) on your machine.
1. Install [``Docker``](http://docker.com) on your machine. Requires Docker 1.7 and above.

1. Install ``docker-compose`` in an virtual environment.
Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
@@ -21,28 +21,22 @@ How to Build SensSoft Docker Containers
$ sysctl -w vm.max_map_count=262144
```

1. Start Elasticsearch cluster:

```bash
$ docker-compose up -d --scale elasticsearch=3 elasticsearch loadbalancer
```

The loadbalancer node exposes port 9200 on localhost and is the only node
that has HTTP enabled. Services such as Kibana and Logstash connect to the
loadbalancer node directly. Loadbalancer accepts requests from Kibana and Logstash
and balances them across the elasticsearch worker nodes. The elasticsearch
worker nodes communicate to each other and the loadbalancer via TCP on port 9300.


1. Confirm cluster state:
1. Start Elasticsearch. Give Elasticsearch about 2 minutes to start before confirming
its state.

```bash
$ docker-compose up -d loadbalancer
```

1. Confirm state:
```bash
$ curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "SensSoft",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 3,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
@@ -55,7 +49,6 @@ How to Build SensSoft Docker Containers
"active_shards_percent_as_number" : 100.0
}
```
Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3.

1. Launch logging server. Give Logstash about 2 minutes to start before confirming
its state.
@@ -105,8 +98,13 @@ How to Build SensSoft Docker Containers

![alt text][dashboard]

1. To Launch Tap and Distill
```bash
$ docker-compose up -d distill tap
```

1. To stop all containers.
```sh
```bash
$ docker-compose stop
```

@@ -123,15 +121,9 @@ Having Issues?
$ docker-compose logs > err.dump
```

Todo
----
- [ ] TAP docker deployment instructions.
- [ ] Distill docker deployment instructions.
- [ ] Apache SensSoft Docker + Kubernetes instructions.

[configure_index]: ./images/configure_index.png "Configure Kibana index"
[confirmation]: ./images/confirmation.png "Confirm index pattern conflicts"
[dashboard]: ./images/dashboard.png "Apache Senssoft Dashboard"
[management]: ./images/management.png "Kibana management console"
[configure_index]: ./docs/images/configure_index.png "Configure Kibana index"
[confirmation]: ./docs/images/confirmation.png "Confirm index pattern conflicts"
[dashboard]: ./docs/images/dashboard.png "Apache Senssoft Dashboard"
[management]: ./docs/images/management.png "Kibana management console"

© Copyright 2016 The Charles Stark Draper Laboratory, Inc. All rights reserved.
@@ -0,0 +1,133 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Apache SensSoft Single-Node Build

version: "3"
services:

# Startup
startup:
build: ./startup
image: senssoft/startup:latest
container_name: senssoft-start
depends_on:
- "loadbalancer"
command: bash /opt/entrypoint.sh

# # Site
site:
build: ./site
image: senssoft/website:latest
container_name: senssoft-app
ports:
- 8080:8080
depends_on:
- "logstash"
command: python -m SimpleHTTPServer 8080

# Elasticsearch Loadbalancer/Entry
loadbalancer:
build: ./elasticsearch/5.6.3
image: senssoft/elasticsearch:5.6.3
container_name: senssoft-loadbalancer
environment:
- NODE_MASTER=true
- NUMBER_OF_MASTERS=1
- NODE_COORDINATE=true
- "DISCOVERY_SERVICE=elasticsearch"
- NODE_DATA=true
- NODE_INGEST=true
- MAX_LOCAL_STORAGE_NODES=1
- NETWORK_HOST=0.0.0.0
- HTTP_ENABLE=true
- HTTP_CORS_ENABLE=true
- HTTP_CORS_ALLOW_ORIGIN=*
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ports:
- "9200:9200"
- "9300"
networks:
- esnet

# Kibana
kibana:
build: ./kibana/5.6.3
image: senssoft/kibana:5.6.3
container_name: senssoft-kibana
environment:
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- "ELASTICSEARCH_URL=http://loadbalancer:9200"
ports:
- "5601:5601"
depends_on:
- "loadbalancer"
networks:
- esnet

# Logstash
logstash:
build: ./logstash/5.6.3
image: senssoft/logstash:5.6.3
container_name: senssoft-logstash
environment:
- "ELASTICSEARCH_URL=loadbalancer:9200"
ports:
- 8100:8100
volumes:
- ./logstash/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
- ./logstash/templates/userale.json:/usr/share/logstash/templates/userale.json
depends_on:
- "loadbalancer"
networks:
- esnet

# Distill
distill:
build: ./distill
image: senssoft/distill:0.1.0
container_name: senssoft-distill
ports:
- 8090:8090
depends_on:
- "loadbalancer"
networks:
- esnet

# TAP's Database
db:
build: ./db
image: senssoft/postgres:latest
container_name: senssoft-postgres
ports:
- "5432:5432"

# TAP
tap:
build: ./tap
image: senssoft/tap:0.1.0
container_name: senssoft-tap
# command: ["bash", "./wait-for-postgres.sh"]
command: bash -c "/usr/src/app/wait-for-postgres.sh"
ports:
- "8010:8010"
depends_on:
- db
links:
- db:db

networks:
esnet:
driver: bridge

1 comment on commit a575d47

@poorejc
Copy link
Contributor

@poorejc poorejc commented on a575d47 Jan 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mooshu1x2 Did you move over all your work from Kubernetes branch to master in this commit? I see some diffs between Kubernetes branch and master, but the timeline doesn't make sense--you added Kubs support on master in April '18, right when you stopped work on Kubernetes branch? Can you advise if I need to merge these branches or just remove Kubernetes? Would be so helpful! Thnx!

Please sign in to comment.