GraphQL Template API
This is a service template for a GraphQL-based API. It is a small stateless HTTP API that aggregates, proxies and transforms downstream APIs. In particular, this template exposes a subset of the Star Wars API.
It aims to provide a simple, consistent, beginner- to intermediate-level stack, aimed at getting a small HTTP-based service up & running quickly with some things we care about in a production system, including:
- A GraphQL API exposed over HTTP;
- System monitoring;
- Error reporting;
This template uses other open source code from Redbubble:
- rb-scala-utils - Miscellaneous utilities (common code) for building Scala-based services, using Finch (on which this project depends).
- finch-sangria - A simple wrapper for using Sangria from within Finch;
- finagle-hawk - HTTP Holder-Of-Key Authentication Scheme for Finagle.
Redbubble also makes available a purely HTTP version of this template (on which this is based).
The architecture of the app is essentially composed vertically, representing a request's flow through the system. Each layer basically only talks to its adjacent layers.
For simplicity of the template (i.e. you may not really do this), the top-level packages are grouped into their functional areas, and are as follows:
api- The HTTP API that we expose to clients. Decodes incoming JSON GraphQL queries & sends them to be executed. Also handles caching at a query-level.
graphql- The data that is exposed, via GraphQL to a client. Basically these should only do marshalling to & from GraphQL. As these are exposed to clients, we call the classes that make up this layer
APIs. to a service.
services- High level business logic; compose fetches to produce a result & also run the fetches. Usually expose a cleaner API than the underlying backend or fetcher.
fetch- Understand how to fetch data from backend datasources. Includes lower level caching of fetched data.
backends- Clients for talking to backend or downstream services. Expose an API that directly mirrors the backend they talk to.
util- Any other code that doesn't fall neatly into one of the above functional buckets.
The API uses GraphQL. GraphiQL, an interactive browser, is available locally (though not in production): http://localhost:8080/v1/explore
There is also simple API documentation available. Where possible, prefer the GraphiQL online documentation as it will be up to date.
To setup for local development, run these steps. Note that most steps assume that you've changed directory to
Install Java 1.8 from Oracle. You will need a JDK (not a JRE), as of the time of writing this is "Java SE Development Kit 8u92". There is also documentation available (handy for linking into your IDE).
$ cd app $ ./sbt > update
Note. You could also
brew install sbtif you'd prefer a system version.
etc/development-envcontains a template
.envfile that you can use to configure your environment variables locally:
$ cd app $ cp etc/development.env .env
See sbt-dotenv for more information. Do not check
.env into source control!.
Note that if you change this while an sbt session is running, you will need to
reload sbt for the new settings to
take effect (as it's an sbt plugin).
- If you're using IntelliJ for development, grab the Scala plugin, and import the project (File->Open, or Import from
the welcome screen) as an SBT project. If you want to run tests, you will need to add the environment variables (in
.env) to the run configurations (specs2).
To run using sbt:
$ ./sbt run
You can also use Revolver for restarts when code changes (recommended!):
$ ./sbt ~reStart
To run using the Heroku tools (requires deployment setup as above), i.e. like it's run in production:
$ ./sbt compile stage $ heroku local # which uses [forego](https://github.com/ddollar/forego)
To run in debug mode:
$ ./sbt run -jvm-debug 5005
You can then connect a remote debugger to the JVM.
Deploying to production will restart the app servers (Heroku Dynos) automatically. However if you wish to restart manually you can:
$ heroku restart -a rb-ios-api
$ ./sbt test
This will start the
sbt REPL, from where you can issue commands.
test- Runs all tests;
test-only com.redbubble.gql.core.CurrencySpec- Runs a single test;
test-only com.redbubble.gql.core.*- Runs all tests in the
test-only *CurrencySpec- Runs
test:compile- Compiles your test code.
~ to the start of any sbt command will run it continuously; for example to run tests continuously:
Note. There are incompatibilities with Gatling & the Netty that Finagle uses at present, the performance tests may not work. If you want to use them, the recommendation is to move back down to Scala 2.11 (for all deps) and use that Gatling version.
Performance testing uses Gattling, and live in the
perf/src/it directory. You can run
all performance tests using the following:
Or individual scenarios as:
> gatling-it:test-only com.redbubble.perf.scenarios.AppStartup > gatling-it:test-only *AppStartup
For more information see: http://gatling.io/docs/2.2.4/extensions/sbt_plugin.html
Note. If you run tests against a local server you will need to start it first.
Deployment is to Heroku. There is a Docker container for testing & runtime, that could be repurposed for deployment.
Deployment to Heroku can be done using:
Before you do though, there is a one-off setup for deployment.
Get an account on Heroku.
Install heroku toolbelt.
Add the git remote (this is a one-off step).
$ heroku git:remote -a graphql-template-api
The deployment uses the SBT Native Packager to package up the artifacts
for deployment. You can run this locally using the
$ ./sbt compile stage
Here is more information on deployment:
To simulate locally what Buildkite runs (
test.sh), you can run the following:
$ make test
All logs go to standard out locally when developing.
You can get the logs using:
$ heroku logs -t -a graphql-template
This will only store the last 1500 lines, if you want to view more you can enable the PaperTrail plugin.
The system is monitored via New Relic. Note that via Heroku we don't get system level metrics such as CPU & memory.
Metrics are exposed via Twitter Metrics. See
com.redbubble.gql.util.metrics.Metrics for the entry point.
Metrics are available locally on the admin server: http://localhost:9990/admin
Metrics into production are bridged to New Relic (sent every minute) as custom metrics and are available on the New Relic insights dashboard.
Note that when sending to New Relic, we do filter out some metrics that are collected locally; in particular the JVM metrics (as NR collects these already), as well as tools we don't use (e.g. Zipkin). We also only send the 75th, 95th, 99th & 99.9th percentiles, along with 1, 5 & 15 minute weighted moving averages (for the appropriate metrics type where supported).
This section is aimed at developers on the project, and gives a quick overview of the features & lbraries used:
- HTTP stack, using Finch;
- Authentication using Hawk, a HMAC-style protocol;
- JSON encoding & decoding, using Circe, including reasonable error handling;
- Caching, batching & fetching using Fetch;
- Downstream service clients using Featherbed;
- Metrics, logged to New Relic;
- System monitoring via New Relic;
- Error reporting to Rollbar
- Testing using specs2 & ScalaCheck;
- Logging to stdout;
- Deployment packaging using SBT Native Packager.
- Getting started with Finagle
- An introduction to Finagle
- Finagle examples
- Other information on Finagle
You can uninstall everything you installed for this project by:
$ rm -rf ~/.sbt $ rm -rf ~/.ivy2
Then, if you want, you can uninstall Java by following the instructions here: https://docs.oracle.com/javase/8/docs/technotes/guides/install/mac_jdk.html#A1096903