Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
Add support for developing the rails app within Docker
Some notes about the rationale behind some of the decisions made in implementing this: - Initially got a version of this working with rails in a container with a mounted volume for the app code, but it's really really slow. Appears to be a known problem when using a mounted volume on OSX https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/276. Added docker-sync configuration and instructions on how to install/use. - Building the image will copy the full application code into /project_root within the container instead of /app. This is in order to disambiguate this from the app directory within the root of the project. Most of the docs for docker-sync will reference /app, so just be aware of this - Added sync excludes for common unneeded directories that may exist in dev workdirs. If you do begin using this, it would be a good idea to clean out any temp files you have in your directory to speed up build/syncs - There are some shenanigans I have to play with the bundled gems and the Gemfile. The rails container will bundle gems into a /bundle directory. This is to preserve any changes bundler will make to the Gemfile.lock, so that we can copy it back to the host once sync is started. I'm running an additional command at the end of the Dockerfile to copy of the lock file back to project_root in the case when someone is not using docker-sync - Had to add a USER env to the compose to reproduce the "developer is an admin" behavior that we all expect - Added install of fits (and the java runtime) to the rails image so that rake curatend:ci would run correctly - The rails_entry script will additionally create the test db for you to simplify running tests in the container. We may want to remove this in the future and separate the test scripts/compose/etc from dev - The rails app is not very fault tolerant if, at start time, mysql isn't ready. Initially I added a healthcheck on mariadb and a rails depends_on mysql healthy condition, thinking this should prevent rails from starting until mysql is healthy. Unfortunately, the healthy dependency condition is no longer available in v3 of docker compose. Instead, this required copying the wait-for-it script referenced in peter-evans/docker-compose-healthcheck#3 and adding a rails entry script that will handle all the things associated with first starting the rails app, including waiting for mysql to start. Left the healthy script there since it may still prove useful - Broke out the readme into separate readme files now that there is an entirely different set of instructions when using Docker and Docker-sync. README.md will continue to give instruction on how to install/run locally on a host. README_DOCKER.md will contain instructions on how to do these things with Docker - Finally, I just want to again note that this only solves the problem of using Docker in development. We need to do things differently if we want to use Docker for CI and other deploys. There are pieces of this that can be reused, but how we separate those out into a common base dockerfile/compose files would need to be discussed Ideally, adding this would have required making no changes to existing app code, but I did have to make several changes: - Running rails app under compose needs to use network names for external services, ie: specs and configs can't assume things are on localhost. Changed solr/fedora/database configs to use env if available, but fallback to localhost if not there to continue supporting development on the host without Docker (such as in the Travis ci atm) - Had to add config.allow_http_connections_when_no_cassette to spec_support. The config was already ignoring localhost, which worked when developing locally on a host. In docker this won't work because it will make calls to the "jetty" and "mysql" hosts and will fail because it couldn't find a cassette that matches. Likewise, changed the id_service_spec to just match on path and method for the cassette so that it would successfully match when a spec made the call to jetty. - Added a byebug initializer so that we can debug remotely, since this is technically a remote machine - Added .docker-sync to gitignore for cases when someone runs the sync as a daemon - Now have multiple Dockerfiles, so had to do a little clean up of the file locations. Note: This docker-compose is still for development only. We'll need to break this out later if we ever decide to use this for ci/staging/etc
- Loading branch information
Justin Gondron
committed
Jul 27, 2018
1 parent
efea632
commit 75f89dd
Showing
20 changed files
with
373 additions
and
57 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
* text=auto | ||
*.sh text eol=lf |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -34,3 +34,4 @@ spec/jetty_generator | |
.tags | ||
.byebug_history | ||
.DS_Store | ||
.docker-sync |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
# Developing CurateND with Docker | ||
|
||
## Installation Notes | ||
Install Docker: https://www.docker.com/ | ||
|
||
Install docker-sync: http://docker-sync.io/ | ||
|
||
Start the stack: | ||
```console | ||
docker-sync-stack start | ||
``` | ||
|
||
## Testing | ||
To execute the full test suite, run | ||
```console | ||
docker-compose exec rails bundle exec rake | ||
``` | ||
To run individual specs, run | ||
```console | ||
docker-compose exec rails bundle exec rspec spec/path/to/file_spec.rb | ||
``` | ||
|
||
## Debugging the application | ||
```console | ||
docker-compose exec rails bundle exec byebug --remote localhost:9876 | ||
``` | ||
|
||
## Rebuilding curate-jetty Docker image | ||
|
||
To rebuild the Docker image for running jetty, use the following command: | ||
|
||
```console | ||
docker build . -t ndlib/curate-jetty | ||
``` | ||
|
||
To push your image to Dockerhub: | ||
|
||
```console | ||
docker login | ||
docker push ndlib/curate-jetty | ||
``` | ||
|
||
To rebuild the image with pre-generated seed data: | ||
```console | ||
# First reset to the base image | ||
docker stop curate-jetty && docker rm curate-jetty | ||
docker run -p 8983:8983 -d --name curate-jetty -t ndlib/curate-jetty | ||
# Run seed scripts from the project root directory | ||
bundle exec rake db:schema:load db:seed:dev | ||
# Commit these changes to your image | ||
docker commit curate-jetty ndlib/curate-jetty-devseed | ||
# Push it up as the dev seed image | ||
docker login | ||
docker push ndlib/curate-jetty-devseed | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
if Rails.env.development? | ||
require 'byebug/core' | ||
#Byebug.wait_connection = true | ||
port = ENV.fetch("BYEBUG_SERVER_PORT", 9876).to_i | ||
print "Starting byebug server for remote debugging on port #{port}\n" | ||
Byebug.start_server 'localhost', port | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,15 +1,48 @@ | ||
version: '3' | ||
services: | ||
mysql: | ||
image: mariadb | ||
build: | ||
context: ./docker/ | ||
dockerfile: Dockerfile.mariadb | ||
ports: | ||
- "3306:3306" | ||
environment: | ||
MYSQL_ROOT_PASSWORD: password | ||
MYSQL_DATABASE: curate_nd_development | ||
volumes: | ||
- ./docker/mysql-utf8.cnf:/etc/mysql/conf.d/mysql-utf8.cnf | ||
healthcheck: | ||
interval: 30s | ||
timeout: 10s | ||
retries: 5 | ||
jetty: | ||
image: ndlib/curate-jetty-devseed | ||
ports: | ||
- "8983:8983" | ||
rails: | ||
build: | ||
context: . | ||
dockerfile: docker/Dockerfile.rails | ||
command: bash docker/rails_entry.sh | ||
environment: | ||
BUNDLE_PATH: "/bundle" | ||
FEDORA_HOST: jetty | ||
FEDORA_PORT: 8983 | ||
MYSQL_HOST: mysql | ||
SOLR_HOST: jetty | ||
SOLR_PORT: 8983 | ||
# Need to pass the user running docker into the container so that | ||
# config/admin_usernames.yml pulls in the user as an admin. This is | ||
# to replicate existing behavior on OSX and may not work correctly on | ||
# another OS | ||
USER: #{USER} | ||
ports: | ||
- "3000:3000" | ||
volumes: | ||
- curate_nd_rails_sync:/project_root:nocopy | ||
depends_on: | ||
- mysql | ||
- jetty | ||
volumes: | ||
curate_nd_rails_sync: | ||
external: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
version: "2" | ||
|
||
options: | ||
verbose: true | ||
syncs: | ||
curate_nd_rails_sync: | ||
src: '.' | ||
sync_excludes: ['.git', 'tmp', 'jetty', 'vendor/bundle', 'log'] |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
FROM mariadb | ||
|
||
COPY mysql-healthcheck /usr/local/bin/ | ||
|
||
HEALTHCHECK CMD ["mysql-healthcheck"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
FROM ruby:2.1.10 | ||
|
||
RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs unzip | ||
|
||
# Install Fits (which requires java) | ||
RUN apt-get install -y default-jre | ||
RUN mkdir /fits | ||
WORKDIR /fits | ||
RUN wget https://projects.iq.harvard.edu/files/fits/files/fits-0.6.2.zip | ||
RUN unzip fits-0.6.2.zip | ||
RUN chmod 755 /fits/fits-0.6.2/fits.sh | ||
ENV PATH="/fits/fits-0.6.2:${PATH}" | ||
|
||
# Put the installed gems outside of project_root so that the sync volume won't interfere | ||
RUN mkdir /bundle | ||
COPY Gemfile /bundle | ||
COPY Gemfile.lock /bundle | ||
WORKDIR /bundle | ||
RUN bundle install --without headless --path /bundle | ||
|
||
RUN mkdir /project_root | ||
WORKDIR /project_root | ||
COPY . /project_root | ||
|
||
# Gemfile.lock may have changed after bundling, copy it back into the project_root | ||
RUN cp /bundle/Gemfile.lock /project_root/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
docker-compose exec rails bundle exec byebug --remote localhost:9876 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
#!/bin/bash | ||
set -eo pipefail | ||
|
||
if [ "$MYSQL_RANDOM_ROOT_PASSWORD" ] && [ -z "$MYSQL_USER" ] && [ -z "$MYSQL_PASSWORD" ]; then | ||
# there's no way we can guess what the random MySQL password was | ||
echo >&2 'healthcheck error: cannot determine random root password (and MYSQL_USER and MYSQL_PASSWORD were not set)' | ||
exit 0 | ||
fi | ||
|
||
host="$(hostname --ip-address || echo '127.0.0.1')" | ||
user="${MYSQL_USER:-root}" | ||
export MYSQL_PWD="${MYSQL_PASSWORD:-$MYSQL_ROOT_PASSWORD}" | ||
|
||
args=( | ||
# force mysql to not use the local "mysqld.sock" (test "external" connectibility) | ||
-h"$host" | ||
-u"$user" | ||
--silent | ||
) | ||
|
||
if select="$(echo 'SELECT 1' | mysql "${args[@]}")" && [ "$select" = '1' ]; then | ||
exit 0 | ||
fi | ||
|
||
exit 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
# All the things that will execute when starting the rails service | ||
# Copy the generated lock file back into the project root. This is to sync this change back to the | ||
# application after the container has mounted the sync volume | ||
cp /bundle/Gemfile.lock ./ | ||
bash docker/wait-for-it.sh mysql:3306 | ||
bundle exec rake db:schema:load | ||
# If we find people are ok with just starting a shell in the container and running the db create/load before running specs, then we can remove this | ||
RAILS_ENV=test bundle exec rake db:create db:schema:load | ||
|
||
exec bundle exec thin start -p 3000 --ssl --ssl-key-file dev_server_keys/server.key --ssl-cert-file dev_server_keys/server.crt |
Oops, something went wrong.