raintank-docker screen based dev environment
This stack runs a development stack, using every component used at raintank (minus Grafana.net), that is:
- Grafana (of course) at http://localhost:3000
- the backend: metrictank, eventtank, Kafka, rabbitmq, graphite-api, Cassandra, ElasticSearch, Mysql, taskAgent, taskServer, tsdbgw, carbon-relay-ng
- monitoring and utiltities: collectd, graphite, statsdaemon, graphitewatcher, benchmark/load test tools, kafkamanager and kafkaoffsetmonitor.
This is a whole lot of stuff because this stack covers every possible flavour of deployment plus extra tools which may be useful, but often not.
In practice you'll probably only use a fraction and that's OK. In the future we may create different scripts to launch different, more minimal versions of the stack. But for now, this approach works well enough.
Note: raintank-apps like taskServer and taskAgent are currently disabled. See this issue
To provision a full dev stack using docker
- install docker
- install docker-compose - We require >= 1.7
- clone this git repo:
git clone https://github.com/raintank/raintank-docker.git
- cd in the dir and run
./build_all.sh. This will build all of the needed raintank images.
Some of the images such as metrictank are built automatically via with github commit hooks and will be pulled in automatically via dockerhub.
- you can run
./build_all.sh rebuildto force a rebuild of the images (but keeping base images such as ubuntu etc) if needed.
- you can use custom versions of required code by running
./setup_dev.shonce and let it download everything (but you can abort once it starts building stuff), going into
code, removing a dir like grafana and making a symlink like
ln -s </path/to/your-code-dir> .In this case, make sure to add a
.notouchfile so that
setup_dev.shdoesn't try to manage the code there.
./setup_dev.shto download and build all of the components. The script will launch a docker container and clone the git repositories of all of the needed components. Once the script has completed, the docker host server will have a
codedirectory that will have all of the required code and dependencies installed.
./launch_dev.shto start up all of the docker containers. This script will start the containers and corresponding screen sessions. Instead of using the code baked into the image, the container will instead execute what is installed in
codeon the docker host. As all containers are sharing the same /opt/raintank directory, any changes made to the /opt/raintank/* code while inside a container will be visible in all other containers.
for your convenience, this code may be all you need (tested on ubuntu 14.04) to get the stack up.
curl -sSL https://get.docker.com/ubuntu/ | sudo sh curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose git clone email@example.com:raintank/raintank-docker.git cd raintank-docker ./build_all.sh ./setup_dev.sh ./launch_dev.sh
- You can log into Grafana at http://localhost:3000/. The default user is 'admin' and the default password is 'admin'
- enable the worldping app through the grafana menu, under 'Plugins' and then 'Apps' tab, using key
changeme. This imports the dashboards, sets up the metrictank datasource, and activates a worldping plugin which you can play with. But often all we use it for is the side-effect of having the
raintankdatasource be created, which queries metrictank (through graphite-api). So if you want to store data in metrictank and visualize it, just enable WorldPing to have the datasource.
- logs will be in the
- You can attach to the screen session with 'screen -r raintank'. To navigate between all of the screen windows press 'CTRL-a then "' (double quote). This will provide the list of windows running, use the arrow keys to select the desired window then press enter.
- to shut the stack down again, run
When stuff goes wrong
- If you get errors like
Unsupported config option for services service: 'graphiteMetrictank'then your docker-compose is too old. Make sure you have >= 1.7
- If you get vague i/o errors make sure you have enough diskspace, check the log of the docker daemon, and try upgrading docker.
- You can see running containers with
docker ps. It's important that no containers are running when you run launch_dev.sh, using stop_dev.sh should achieve this.
./launch_dev.shkeeps on emitting the following messages without end:
waiting for all xx containers to run... waiting for all xx containers to run... (...)
docker ps -a to identify which containers failed. You will see something like:
(...) 960f23654203 raintank/eventtank "/start.sh -config /e" About a minute ago Up About a minute 0.0.0.0:6062->6060/tcp raintank_eventtank_1 feb682f3a24a raintank/metrictank "/usr/bin/metrictank_" About a minute ago Exited (1) 54 seconds ago raintank_metrictank_1 386a2ee9fbe2 raintank/task-server "/start.sh --config /" About a minute ago Up About a minute 0.0.0.0:8082->80/tcp raintank_taskServer_1 (...)
Look for all the containers which exited, in the first column you can see their container id's. (note, this listing may also include old containers which have exited hours or days ago, which are not relevant. Be sure to start at the top and work your way down and look at the status column to see what's relevant.
Then, for the containers that failed, run
docker logs <container-id from first column> this should help explain what failed. If it doesn't, please post the output of this command as well as
docker ps -a in a github issue.
in a new raintank-docker branch, you can just run
/build_all.sh, which updates the "latest" images.
If your latest images have been overridden by being in a different branch,
./restore_branch_images.sh will sync up the latest tags to the images you last generated on this branch.