The Guardian’s new image management system
Scala JavaScript HTML CSS Shell Python
Latest commit 3963b15 Jan 19, 2017 @akash1810 akash1810 committed on GitHub Merge pull request #2028 from guardian/aa-feedback-form
replace feedback form
Permalink
Failed to load latest commit information.
auth Use permissions API (#2000) Oct 10, 2016
cloud-formation Add extra info to readme, make stack-name script work on os x (#2004) Dec 8, 2016
cloudwatch-logs-to-logstash deploy.sh -> build.sh for clarity Dec 22, 2015
collections Many apps systemd (#1877) Apr 28, 2016
common-lib/src Merge branch 'master' into add-usage-info-media-atom-poster-images Dec 16, 2016
cropper Use permissions API (#2000) Oct 10, 2016
docker Add extra info to readme, make stack-name script work on os x (#2004) Dec 8, 2016
docs Minor fix Jan 6, 2017
elasticsearch Remove supplier weight sort plugin from ES startup (#1995) Sep 15, 2016
image-loader add unique png temp file name to prevent race conditions (#1991) Aug 24, 2016
imgops ensure user has nginx tap for installation of full version Mar 16, 2016
kahuna replace feedback form Jan 19, 2017
leases Batch leases front end (#1935) Jun 27, 2016
media-api Make Quotas configurable via file in S3 (#2020) Dec 8, 2016
metadata-editor Many apps systemd (#1877) Apr 28, 2016
packer update packer readme May 5, 2016
project Riff raff yam usages and add deployment for usage-stream Dec 22, 2016
reaper do not validate SSL. Sep 29, 2015
s3watcher bump to node-riffraff-artefact version with configurable buffer Feb 16, 2016
scripts updating EsScripts to use the current alias or default to "writeAlias" ( May 23, 2016
thrall Service changes for Leases in ES (#1930) Jun 16, 2016
usage Updated README on deployment process for usage and usage-stream Dec 22, 2016
.dockerignore upgrade to Docker 1.10 and Compose 1.6 Feb 9, 2016
.editorconfig editorconfig for .yml files Feb 11, 2016
.gitignore dot-properties helper script Jan 22, 2016
COPYING Add Apache 2 licence Jul 3, 2015
Procfile Adds Image Leases (#1845) Apr 18, 2016
README.md Removing 'create-dev-stack' advice in readme (#2017) Dec 8, 2016
TROUBLESHOOTING.md troubleshooting updates (#1911) May 27, 2016
VISION.md Update README and vision to make the repo more public-friendly. Jul 3, 2015
docker-compose.yml add auth service to docker Feb 11, 2016
nginx-mappings.yml Adds Image Leases (#1845) Apr 18, 2016
run.sh add leases to run script Apr 18, 2016
sbt SBT launcher & TeamCity test reporting plugin Aug 22, 2013
sbt-launch.jar Replace old SBT launcher with 0.13 version Sep 25, 2013
sbt-tc Update references to Java 7 Jun 23, 2015
setup.sh Call kahuna setup from master setup Jan 28, 2015
the_pingler.sh use healthcheck endpoint for ping Apr 19, 2016

README.md

Grid

Join the chat at https://gitter.im/guardian/grid

Grid is the Guardian’s new image management system, which provides a universal and fast experience accessing media that is organised and using it in an affordable way to produce high-quality content.

See the Vision document for more details on the core principles behind this project.

Screenshot of Grid search

Grid runs as a set of independent micro-services (Scala and Play Framework) exposed as hypermedia APIs (argo) and accessed using a rich Web user interface (AngularJS).

Grid relies on Elasticsearch for blazing-fast searching, and AWS services as additional storage and communication mechanisms.

Running the applications

Requirements

You will need to install:

  • sbt
  • JDK 8
  • Nginx
  • GraphicsMagick sudo apt-get install graphicsmagick or brew install graphicsmagick --with-little-cms2.

If you get a permissions issue when running brew install graphicsmagick --with-little-cms2 you can install little cms (brew install little-cms2 ) and then compile graphicsmagick from source, it will automatically pick up on the little cms installation

Nginx

To run correctly in standalone mode we run behind nginx, this can be installed as follows:

  1. Install nginx:

    • Linux: sudo apt-get install nginx
    • Mac OSX: brew install nginx
  2. Make sure you have a sites-enabled folder under your nginx home. This should be

    • Linux: /etc/nginx/sites-enabled
    • Mac OSX: /usr/local/etc/nginx/
  3. Make sure your nginx.conf (found in your nginx home) contains the following line in the http{} block: include sites-enabled/*;

    • you may also want to disable the default server on 8080
  4. Get the dev-nginx repo checked out on your machine

  5. Set up certs if you've not already done so

  6. Configure the app routes in nginx

    sudo /setup-app.rb /nginx-mapping.yml

Elasticsearch

You can run setup.sh to install and start Elasticsearch. You can use the script to start up Elasticsearch even if it's already installed.

Alternatively you can do these steps manually:

Run the Elasticsearch installer from the elasticsearch directory:

    $ cd elasticsearch/
    $ ./dev-install.sh

Start Elasticsearch from the elasticsearch directory:

    $ cd elasticsearch/
    $ ./dev-start.sh

Generate .properties files

Guardian devs can follow the guide in the dev-utils folder of this private repo. This will give you some tips about our specific configuration.

The general process is as follows:

Generate your .properties files for the various media-service services using the dot-properties generator

This will also create a panda.properties file that configures the pan-domain authentication

This file will be used by the different applications to share auth config, so that CORS is enabled across APIs.

Make sure you put the generated .properties files in /etc/gu/ instead of ~/.gu/ as many apps do.

Run the services

There are a few different ways to run the services, please see here.

NGINX, Play & SNI

As the Play Framework does not yet support SNI NGINX can't always work out which certificate to send where there are multiple services on the same IP. This might result in NGINX sending the incorrect certificate. This will result in a HostnameVerifier Exception appearing in application logs.

Resolution

When the correct cert to send is ambiguous NGINX simply sends the first cert it sees in it's configuration (which is loaded from config files in alphabetical order) to resolve this problem, prefix your grid config filename with 0-!