HTTP image processing server based on imgflo
CoffeeScript JavaScript HTML Makefile
Permalink
Failed to load latest commit information.
bin applications: Code and tool for adding new app Jul 27, 2016
components Update to GEGL 0.3.10 Nov 14, 2016
doc
examples UI: Make url copy/pasteable Nov 17, 2016
graphs Process video jobs in a dedicated worker Nov 17, 2016
migrations
runtime @ 9d9d76d runtime: Update dependencies to v101, with ffmpeg Nov 16, 2016
spec Tests: Add a video testcase Nov 16, 2016
src Process video jobs in a dedicated worker Nov 17, 2016
.buildpacks Try to bringback Heroku button support Jan 11, 2017
.gitignore .gitignore: Add build/ Nov 29, 2016
.gitmodules
.travis.yml Revert "Travis: Enable verbose tests" Nov 16, 2016
.vendor_urls dependencies: Bump v104, built on OSX now Nov 17, 2016
Aptfile
CHANGES.md CHANGES: Couple more mentions Feb 25, 2016
Makefile Process video jobs in a dedicated worker Nov 17, 2016
README.md
app.json
gitmodules
imgflo-server.creator
imgflo-server.files demo: Update TODO comment Nov 17, 2016
index.js server: Make sure New Relic is registered first Apr 8, 2015
knexfile.coffee knex: Always use DB config based on configured URL Jan 11, 2017
newrelic.js 403 Forbidden is also not an error we want to get alarms for Nov 11, 2015
package.json NPM: Also have sqlite3 in deps Jan 11, 2017
update-version-info.js Make version error non-fatal Sep 24, 2014
webpack.config.js webpack: Use external for newrelic Jun 7, 2016
worker.js Add msgflo based worker Apr 1, 2015
worker.webpack.js webpack: Initial config for worker Jun 7, 2016

README.md

Build Status

imgflo-server

imgflo-server is an image-processing server with HTTP built using the imgflo dataflow runtime.

Deploy

Changelog

See ./CHANGES.md

License

MIT

Note: GEGL itself is under LGPLv3.

API

imgflo-server provides a HTTP API for processing images. The entire processing request is described using an URL, and the processing can be triggered by a HTTP GET. This means no special integration is needed in order to use an image processed by imgflo-server in an application.

Process image and get results

Request

GET /graphs/mygraph?input=http://example.com/input-image.png&attr1=value1&....

Response

HTTP 301
Location: https://s3.aws.com/bucket/er132123sder12.png

If the image has been processed before, it will come straight from cache (fast).

Process image without waiting for result

Request

POST /graphs/mygraph?input=http://example.com/input-image.png&attr1=value1&....

Response

HTTP 301
Location: https://s3.aws.com/bucket/er132123sder12.png

Since the image is not processed by the time the response arrives, accessing the Location immediately will likely fail (with a 404 or 403). If the image processing failed, it may fail forever. Use a GET with the same imgflo URL to get the error.

Get available image processing graphs

The inports describe which parameters are available for each graph.

Request

GET /graphs

Response

HTTP 200 application/json
{
  "graphs": {
    "customgrey": {
      "inports": {
        "input": {
        },
        "height": {
          "metadata": {
            "description": "Requested output height",
            "type": "int",
            "maximum": 2000,
            "minimum": 0
          }
        },
        "width": {
          "metadata": {
            "description": "Requested output width",
            "type": "int",
            "maximum": 2000,
            "minimum": 0
          }
        }
      },
      "outports": {
        "output": {
        }
      }
    }
}

Specifying output format

By specifying an extension on the graph, you can

/graph/gradientmap.png?width=300....

If the extension is omitted, a default image format is inferred. This is currently the same image format as the input.

Authentication

To prevent people from using your deployment to process their images, there exists a version of the processing URL scheme which uses encryption to secure it. Note that this only prevents people from forming new image processing requests. It does not prevent them from requesting an already processed image.

Each client has its own KEY and SECRET.

Given a request with a set of parameters (like gradientmap.png&width=300&input=...), a token is constructed by taking the md5sum of the SECRET concatenated with the parameters:

token=md5(parameters+secret)
GET /graph/$KEY/$token/$params

Upon getting such a request, the server will use the KEY to find the matching SECRET, do the same construction of token, and validate that there is a match. If not, request will fail with a 403 Permission Denied.

Adding new tokens for a client/application

Use the helper script

./bin/imgflo-server-application-add

It will generate client key and secret, insert it to the database, and output the details to stdout.

Changing quotas

If processing_quota field is 0, the client cannot process new images, but can still request existing cached/processed images.

Currently processing quota is not otherwise enforced.

To disallow clients from also accessing cache/processed images, set enabled to false/0.

There is currently no tools for manipulating quota, have to use SQL on the applications table.

Errors

  • 504: Unable to fetch the specified input URL
  • ...

API client libraries

These libraries make it it easier to use imgflo-server, by providing a programmatic API which handles the details of constructing authenticated URLs.

JavaScript / node.js / browser


  1. Forming valid imgflo urls: imgflo-url
  2. Creating responsive images using media-query: rig

Java / Android

imgflo-url-java

Swift / iOSs

ImgFlo.swift

Testing UI

imgflo-server ships with a simple testing UI served at /. It can be used to see the available graphs, and make test requests and see the results.

TODO: add picture

Use for debugging a request

Given a regular imgflo URL, you can add &debug=1 at the end to open it in the UI. It will extract the parameters, including removing the urlencoding on the input URL.

Creating new image processing graphs

imgflo-server can easily be extended with new image processing pipelines, either using a text-based DSL or Flowhub node-based visual IDE.

See Adding Graphs

System architecture

For an in-depth look at how the system is implemented, see system architecture

Hosted public instance

Currently our deployed instance is only for The Grid. If you are interested in access to hosted version, send us an email: support@thegrid.io

Deploying to Heroku

Server

Register/log-in with Heroku, and create a new app. First one is free.

After creating the app, login at Heroku:

heroku login

Clone imgflo-server:

git clone https://github.com/imgflo/imgflo-server.git
cd imgflo-server

Add YOURAPP as remote:

heroku git:remote -a YOURAPP

Specify the multi buildpack with build-env support, either at app creation time, in Heroku webui or using

heroku config:set BUILDPACK_URL=https://github.com/mojodna/heroku-buildpack-multi.git#build-env

Configure some environment variables (hostname, port and local image cache):

heroku config:set HOSTNAME=YOURAPP.herokuapp.com
heroku config:set PORT=80

Deploy, will take ~1 minute

git push heroku master

You should now see the imgflo server running at http://YOURAPP.herokuapp.com

If everything is OK you should be able to see a generative image at http://YOURAPP.herokuapp.com/graph/delaunay_triangles?seed=foobar&height=800&width=600

Developing and running locally

Note: imgflo-server is only tested on GNU/Linux systems. imgflo (the runtime) has experimental support for OSX. However, all underlying dependencies (node.js, RabbitMQ, GEGL etc) are commonly used also on other platforms, like Windows.

Root is not needed for any of the build.

Pre-requisites

imgflo requires git master of GEGL and BABL, as well as a custom version of libsoup. It is recommended to let make setup this for you, but you can use existing checkouts by customizing PREFIX. You only need to install the dependencies once, or when they have changed.

git submodule update --init --recursive
make dependencies

Install node.js dependencies

npm install

Build

Now you can build & install imgflo-server itself

make install

To verify that things are working, run the test suite

make check

Running server

node index.js

You should see your server running at http://localhost:8080