-
Notifications
You must be signed in to change notification settings - Fork 13
Docker Details and Tips
One of the most useful debugging techniques is to monitor the docker container logs:
docker logs -f chords_app
docker logs -f chords_mysql
Also useful is to open a shell in the running container for diagnostic purposes:
docker exec -it chords_app /bin/bash
root@1c6154c909d7:/app# cd log
root@1c6154c909d7:/app/log# ls -l
total 3636
-rw-r--r-- 1 root root 3719618 Sep 24 16:29 development.log
root@1c6154c909d7:/app/log#
- The alpine image provides a very lightweight image suitable for many basic Linus tasks.
- The original Docker technique used a Data Only Container. Docker v1.6 introduced Named Data Volumes, which apparently are now the preferred technique. They let you create a named persistent volume, that can be easily manipulated. Typically you just run an alpine container, to execute commands to access what ever is in the volume. This is demonstrated below.
-
docker-compose.yml has a new format, identified by
version: '2', which breaks the configuration into three sections: services, volumes, and networks. Named volumes are designated in the volumes section. - The CHORDS docker image is hosted on Docker Hub.
- docker-compose is used in two modes. The default method is used to build, start and stop the the CHORDS containers. The second mode, which utilizes an extra configuration file, allows development testing to occur without rebuilding the CHORDS image.
Here is how you create a new image, and then deploy it.
Make sure that your local changes have been committed and pushed to github. Now create the image and push it to docker hub:
cd <working directory>
docker-compose build
docker push ncareol/chords
To deploy a new server, or update a running one, first ssh into your server. Then:
cd <directory containing docker-compose.yml>
curl -kL https://github.com/NCAR/chords_portal/raw/docker/docker-compose.yml > docker-compose.yml
docker-compose -p chords down
docker-compose pull
CHORDS_DB_PW=<db pw> docker-compose -p chords up -d
The existing database will be retained.
Of course for testing, you can iterate over the build, down and up steps without pushing/pulling the image.
A special docker-composer configuration exists which simply mounts the working directory on the host. You can edit the Rails code and the changes will be seen in the chords_app container. This approache uses the docker-compose facility for combining multiple configurations:
cd <working directory>
CHORDS_DB_PW=<db pw> docker-compose -p chords -f docker-compose.yml -f docker-compose-dev.yml up -d
These environment variables can be set and exported, or specified on the docker-compose command. E.g.:
SECRET_KEY_BASE=aaaaaaaaaaa RAILS_ENV=production CHORDS_DB_PW=chords_ec_demo docker-compose -p chords up -d
| Variable | Meaning |
|---|---|
| SECRET_KEY_BASE | The Rails secret key. It defaults to aaaaaaaaaaa, but should be set to a long random number for real deployments. |
| RAILS_ENV | The Rails mode. Defaults to production. Allowed values are production or development |
| CHORDS_DB_PW | The database password (of your choosing). It is required on the docker-compose up command. |
The Dockerfile for chords_app runs the script create_chords_env_script.sh. This script in turn creates
a new script chords_env.sh, which gets baked into the image. This second script sets environment variables such as CHORDS_GIT_SHA and CHORDS_GIT_BRANCH.
When the container is run, the chords_startup.sh script will source chords_env.sh to set these environment variables, which are then available to the app.
We are using docker-compose to orchestrate the interaction of the CHORDS containers and volumes.
Where appropriate, the docker-compose -p chords parameter is specified. This causes Docker to use
chords as the project name, which is used in naming instances and volumes. Otherwise, the working directory
name would be used, and that is clearly lame.
# Except for CHORDS_DB_PW, the environment variables default to the shown values,
# so they can be left off. CHORDS_DB_PW must be provided.
$ SECRET_KEY_BASE=aaaaaaaaaaa RAILS_ENV=production CHORDS_DB_PW=chords_ec_demo docker-compose -p chords up -d
Creating network "chords_default" with the default driver
Creating volume "chords_mysql-data" with default driver
Creating chords_mysql
Creating chords_app
Sometimes, a container (service) will not start properly. You can start a service, running e.g. bash, in order to diagnose startup problems. In this case, you could also mount the local CHORDS app directory by adding the -dev configuration.
# app is the name of the rails application in docker-compose.yml
$ SECRET_KEY_BASE=aaaaaaaaaaa RAILS_ENV=production CHORDS_DB_PW=chords_ec_demo docker-compose \
-f docker-compose.yml -f docker-compose-dev.yml -p chords run app /bin/bash
root@dca18687f0fd:/app# ./chords_start.sh
[ ok ] Restarting nginx: nginx.
**** /var/lib/mysql/CHORDS_SEEDED_production was found. Database will not be created.
Migrating rails database.
**** /var/lib/mysql/CHORDS_SEEDED_production was found. Database will not be seeded.
**** Starting web server.
I, [2016-11-07T18:40:10.399741 #63] INFO -- : Refreshing Gem list
I, [2016-11-07T18:40:13.870411 #63] INFO -- : unlinking existing socket=/app/tmp/unicorn.chords.sock
I, [2016-11-07T18:40:13.873650 #63] INFO -- : listening on addr=/app/tmp/unicorn.chords.sock fd=9
I, [2016-11-07T18:40:13.874113 #63] INFO -- : listening on addr=0.0.0.0:8080 fd=11
I, [2016-11-07T18:40:13.891696 #66] INFO -- : worker=0 ready
I, [2016-11-07T18:40:13.895016 #69] INFO -- : worker=1 ready
I, [2016-11-07T18:40:13.898092 #72] INFO -- : worker=2 ready
I, [2016-11-07T18:40:13.899797 #63] INFO -- : master process ready
I, [2016-11-07T18:40:13.900779 #75] INFO -- : worker=3 ready
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
049dc05c5a06 mistermartin/chords_app "bundle exec bash -c " 57 seconds ago Up 51 seconds 0.0.0.0:80->80/tcp chords_app
ed97fcc98926 mysql "docker-entrypoint.sh" 57 seconds ago Up 56 seconds 0.0.0.0:32772->3306/tcp chords_mysql
There may be a bunch of anonymous volumes, which seem to be artifacts from the build process.
$ docker volume ls
DRIVER VOLUME NAME
local 037c190667ed622b5aa519613a13fa6bbb9877b6001e645e85b50ce0af292b4e
...
local c9446cc3ed8a5c19f54c4b93e66bef52d0cc37143c799d57d6b8020a4655d824
local chords_mysql-data
local d04a1634c7106761468fc26f2b26456686f099db1748d00c3e53dc13c4068b43
...
local ffd5e7f36d312b661645ee175ecb6825a927d1b91c6f9661b8b7297a1cccee57
docker run --rm ncareol/chords_app ls -l
total 88
-rw-r--r-- 1 root root 1323 Sep 5 03:14 Dockerfile
-rw-r--r-- 1 root root 1856 Sep 3 16:57 Gemfile
-rw-r--r-- 1 root root 5181 Sep 3 16:57 Gemfile.lock
-rw-r--r-- 1 root root 1650 Sep 3 16:57 README.md
-rw-r--r-- 1 root root 478 Dec 31 2015 README.rdoc
-rw-r--r-- 1 root root 249 Dec 31 2015 Rakefile
drwxr-xr-x 9 root root 4096 Nov 25 2015 app
drwxr-xr-x 8 root root 4096 Sep 3 16:57 bin
-rwxr-xr-x 1 root root 289 Sep 5 03:15 chords_env.sh
-rwxr-xr-x 1 root root 1372 Sep 5 02:51 chords_start.sh
drwxr-xr-x 5 root root 4096 Sep 3 16:57 config
-rw-r--r-- 1 root root 153 Dec 31 2015 config.ru
-rwxr-xr-x 1 root root 1049 Sep 5 02:51 create_chords_env_script.sh
drwxr-xr-x 3 root root 4096 Sep 3 16:57 db
-rw-r--r-- 1 root root 1209 Sep 3 16:57 docker-compose.yml
-rw-r--r-- 1 root root 6452 Sep 3 16:57 docker_small_h.png
drwxr-xr-x 4 root root 4096 Nov 24 2015 lib
drwxr-xr-x 3 root root 4096 Jun 2 16:36 public
drwxr-xr-x 8 root root 4096 Dec 31 2015 test
drwxr-xr-x 5 root root 4096 Sep 3 16:57 vendor
Use the alpine base image to execute the du command. Note that the --rm switch on the docker run command means that the container will be removed after it has run.
$ docker run --rm --volume chords_mysql-data:/var/lib/mysql alpine du -sh /var/lib/mysql/
199.0M /var/lib/mysql/
$ docker run --rm --volume chords_mysql-data:/var/lib/mysql alpine ls -an /var/lib/mysql/
total 176176
drwxr-xr-x 7 999 999 4096 Sep 1 05:55 .
drwxr-xr-x 6 0 0 4096 Sep 1 05:57 ..
-rw-r--r-- 1 0 0 0 Sep 1 05:54 CHORDS_SEEDED
-rw-r----- 1 999 999 56 Sep 1 05:53 auto.cnf
drwxr-x--- 2 999 999 4096 Sep 1 05:54 chords_demo_development
drwxr-x--- 2 999 999 4096 Sep 1 05:54 chords_demo_test
-rw-r----- 1 999 999 938 Sep 1 05:55 ib_buffer_pool
-rw-r----- 1 999 999 50331648 Sep 1 05:55 ib_logfile0
-rw-r----- 1 999 999 50331648 Sep 1 05:53 ib_logfile1
-rw-r----- 1 999 999 79691776 Sep 1 05:55 ibdata1
drwxr-x--- 2 999 999 4096 Sep 1 05:53 mysql
drwxr-x--- 2 999 999 4096 Sep 1 05:53 performance_schema
drwxr-x--- 2 999 999 12288 Sep 1 05:54 sys
$ docker exec chords_mysql sh -c 'exec mysqldump --all-databases' > chords-database-backup.sql
If this is not a fresh chords instance, you may need to get rid of the current database by stopping CHORDS, removing the database, and restarting:
docker-compose -p chords down
docker volume rm chords_mysql-data
CHORDS_DB_PW=<db pw> docker-compose -p chords up -d
Then you can restore the backup:
$ docker cp chords-database-backup.sql chords_mysql:/tmp/chords-database-backup.sql
$ docker exec chords_mysql sh -c 'exec mysql < /tmp/chords-database-backup.sql'
$ docker-compose -p chords down
Stopping chords_app ... done
Stopping chords_mysql ... done
Removing chords_app ... done
Removing chords_mysql ... done
Removing network chords_default
Note that the database named volume (chords_mysql-data) is still available.
docker volume rm chords_mysql-data
The docker history command shows the derivation of an image. The size of
each layer is given, making it easy to track down where the bloat comes from.
In the following, everything prior to "MAINTAINER" comes from the ruby:2.2
base image.
docker history ncareol/chords_app
IMAGE CREATED CREATED BY SIZE COMMENT
af464699726c 17 minutes ago /bin/sh -c #(nop) CMD ["passenger" "start" " 0 B
4429504098e6 17 minutes ago /bin/sh -c #(nop) ENTRYPOINT ["bundle" "exec 0 B
198d7d84c462 17 minutes ago /bin/sh -c #(nop) EXPOSE 80/tcp 0 B
e29e70cde8a2 17 minutes ago /bin/sh -c #(nop) COPY dir:6c3374253042935e48 78.69 MB
c6efd6da8012 47 hours ago /bin/sh -c gem install bundler && bundle inst 162.5 MB
31764e8a6868 47 hours ago /bin/sh -c #(nop) COPY multi:cb3cbfd9251dcfad 7.037 kB
39f8ae044a6f 47 hours ago /bin/sh -c #(nop) WORKDIR /app 0 B
f54243ba4c9c 47 hours ago /bin/sh -c mkdir -p /app 0 B
75614e706b6e 47 hours ago /bin/sh -c apt-get update && apt-get install 99.98 MB
2c5e9174a8c7 47 hours ago /bin/sh -c #(nop) MAINTAINER martinc@ucar.ed 0 B
98d746ecf184 4 months ago /bin/sh -c #(nop) CMD ["irb"] 0 B
<missing> 4 months ago /bin/sh -c mkdir -p "$GEM_HOME" "$BUNDLE_BIN" 0 B
<missing> 4 months ago /bin/sh -c #(nop) ENV PATH=/usr/local/bundle/ 0 B
<missing> 4 months ago /bin/sh -c #(nop) ENV BUNDLE_PATH=/usr/local/ 0 B
<missing> 4 months ago /bin/sh -c #(nop) ENV GEM_HOME=/usr/local/bun 0 B
<missing> 4 months ago /bin/sh -c gem install bundler --version "$BU 1.182 MB
<missing> 4 months ago /bin/sh -c #(nop) ENV BUNDLER_VERSION=1.11.2 0 B
<missing> 4 months ago /bin/sh -c set -ex && buildDeps=' bison 111.9 MB
<missing> 4 months ago /bin/sh -c #(nop) ENV RUBYGEMS_VERSION=2.6.4 0 B
<missing> 4 months ago /bin/sh -c #(nop) ENV RUBY_DOWNLOAD_SHA256=30 0 B
<missing> 4 months ago /bin/sh -c #(nop) ENV RUBY_VERSION=2.2.5 0 B
<missing> 4 months ago /bin/sh -c #(nop) ENV RUBY_MAJOR=2.2 0 B
<missing> 4 months ago /bin/sh -c mkdir -p /usr/local/etc && { ec 45 B
<missing> 4 months ago /bin/sh -c apt-get update && apt-get install 317.8 MB
<missing> 5 months ago /bin/sh -c apt-get update && apt-get install 122.6 MB
<missing> 5 months ago /bin/sh -c apt-get update && apt-get install 44.28 MB
<missing> 5 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B
<missing> 5 months ago /bin/sh -c #(nop) ADD file:2ed80fd333c2052add 125.1 MB
There is a known issue with the Docker data space growing up to 64GB, even if all images and containers are destroyed. It has to do with Docker for Mac's usage of a block device which doesn't reclaim deleted space. To see how much space is being used:
$ du -s -h ~/Library/Containers/com.docker.docker
Until the problem is resolved, the workaround is to dump the CHORDs database, and re-install Docker for Mac. There is a button for this in the application badge. Not that this will wipe out all containers and images, not just the ones being used for CHORDS. So be careful and save anything you need.
Project Management
- Stakeholders
- Communication
- Use Cases
- Requirements
- Deliverables
- Milestones Associated with a Release
- "Sandbox" Milestones
- Github Workflow
AWS
- AWS Portal Migrations
- Amazon Appliance Workflow
- Overview
- Bringing up a new CHORDS Portal
- Cloud Formation
- EC2 Costing and Memory Constraints
Docker
- Running CHORDS
- Docker on AWS
- Duplicating Docker/Influxdb portals
- Docker on Raspberry Pi
- Docker Details and Tips
- Running CHORDS on Windows 10
Influxdb
Data Formats
Google Maps
Ingest Utilities
Miscellaneous
- Recovery from a full disk
- Github/Dockerhub release scheme
- CHORDS gh-pages and jeykll
- Bootstrap
- CHORDS Portal Web Site
- Dashboard Helper Refactor
- Development Notes
- Heroku
- Meteobridge
- Migrating from mysql to mysql/influxdb portals
- NCAR Wx Stations into CHORDS
- PAWS to CHORDS
- Post Get Query Syntax
- Postgres Testing
- Rails Tips
- Ruby and Rails Resources
- CUAHSI Archive
Historical Archive