Skip to content


Subversion checkout URL

You can clone with
Download ZIP
A platform for tools that do stuff with data
CoffeeScript HTML CSS JavaScript Shell

Merge pull request #716 from scraperwiki/add-dockerfile

Add Dockerfile and docker-compose
latest commit 9966b77116
@pwaller pwaller authored
Failed to load latest commit information.
.ssh Fix custard .ssh/config.
bin Meh
client Corporate signup only
cron Remove old crontab cleaning code
docker Merge pull request #671 from scraperwiki/docker-name-switch
migration Remove useless disk quota code
server Disable self service signup
shared Remove twitter help
test Change in help headings reflected in test
util migration: Oops. There are multiple views per dataset
.envrc Add instructions for direnv along with an .envrc
.gitignore Add tmp files found on prod to .gitignore
Cakefile Ignore tests in nodemon restart
Dockerfile Add Dockerfile and docker-compose
Procfile add a Procfile for foreman Update README for new Selenium
activate Re-source before nvm use
docker-compose.yml Add Dockerfile and docker-compose
fixtures.js Give box fixture a publish token
package.json Add Dockerfile and docker-compose pre-commit hook which will run unit tests
server.js fix permissions
tang.hook Time the tang.hook

scraperwiki / custard

A platform for tools that do stuff with data.

Together with cobalt, custard powers the new ScraperWiki platform.

AGPL Licenced (see LICENCE file).

Initial setup (mongo, redis, nvm)

 On Debian or Ubuntu:

sudo apt-get install -y mongodb
sudo apt-get install -y redis-server

You'll need a bunch of node stuff which we get using nvm:

curl | sh

nvm must be runnable, so add this line to your .profile:

. ~/.nvm/

start a new bash terminal (in order to get nvm which is a shell function)

nvm install 0.10

 On Mac OSX:

Install Node via nvm:

curl | sh
nvm install 0.10

Install Mongo and Redis via Homebrew:

brew install mongodb
brew install redis

Then on all platforms:

Clone this repo (inside ~/sw preferable) and from within the custard directory:

mkdir mongo
npm install pow-mongodb-fixtures -g
mongod --dbpath=mongo # might be running already
# redis-server # probably don't need this, as it's already running

Alongside custard, you will need to git clone swops-secret

git clone blah blah blah

Automatically getting the correct environment with direnv

direnv can be used to automatically "activate" the environment when you enter the directory. Briefly:

# (install go and set up your GOPATH and PATH sensibly)
go get
go install

# append direnv setup to your bash profile
echo 'eval "$(direnv hook $0)"' >> ${HOME}/.bashrc

# Then cd to the custard directory and enable the .envrc with direnv:
:~$ cd sw/custard/
.envrc is not allowed

:~/sw/custard$ direnv allow
direnv: loading ~/sw/.envrc
connect-assets not found, please run npm install

The first time and then every now and then (npm updates)

# cd into the custard directory
. activate
npm install

Every time you need to develop custard

If you have foreman installed (try gem install foreman) you can go:

foreman start

If you don't have foreman installed you need to do a few things by hand:

# Start a web server (best done in a new window)
. activate # don't need to do this again if you've already done it in this terminal
cake dev

The cake dev above starts custard the web server. It's listening on a local port (usually 3001), so you should be able to visit localhost:3001 in a web browser.


We love them.

First, download Selenium Server and download and unzip Chromedriver, and put them in your custard directory or the next level up:

You need to unzip chromedriver, but not selenium.

If you're wondering about versions, right now we're having luck with:

Then start the Selenium server using cake se:

. activate && cake se

(this is a shortcut for running java -jar selenium-server-standalone-2.*.0.jar

To run the tests:

. activate && cake dev       # runs a local webserver, as above

Or one of these:

mocha test/unit
mocha test/integration

Optional: running Selenium on a Windows machine

On the Windows machine

  • Make sure Java is installed
  • Download the selenium-server jar file (from
  • Download, unpack and add to your path the IE driver (avaiable on the same download page)
  • Run Selenium Server using the command java -jar selenium-server-standalone-2.*.0.jar
  • Run ipconfig in a cmd window to find out the IP address (e.g.

On your local machine

  • export SELENIUM_HOST=
  • export SELENIUM_PORT=4444
  • export BROWSER=iexplorer
  • export CU_TEST_SERVER=<local ip address>
  • run mocha as above and it will use the Windows machine to run the tests

Optional: disabling startup services

You may wish to disable redis-server and mongodb services from autostarting on boot when not developing custard.

(Tested on Ubuntu 12.04)

Disable mongo service:

echo "manual" | sudo tee /etc/init/mongodb.override

Enable mongo service:

sudo rm /etc/init/mongodb.override

Disable redis-server service:

sudo update-rc.d redis-server disable

Enable redis-server service:

sudo update-rc.d redis-server enable
Something went wrong with that request. Please try again.