For What It's URF
Live Site: For What It's URF
Created for Riot Games API Challenge, what started as a fantasy scoring application (hence the name 'urfantasy') ended up turning into a data exploration application. Rather than using strictly familiar technologies, my duo queue partner for this project, dbanksdesign, and I also incorporated new/trendy stuff (at least to us) such as Docker, ReactJS, D3.js, and Postgres for JSON persistence. I plan to write a retrospective of this development experience on my blog.
The goal of our application is to provide interesting information about URF without denying our users the experience to be their own explorers. We feel that once users get accustomed to the treemap interface, they can delight in their discoveries and have fun reviewing the URF data. As a project, the code can be a valuable resource to anyone trying to learn how to import, transform, and present data by interacting with queues, the database, and the frontend stack. As for the code itself, we don't recommend anyone using it as a template (gets bad towards deadline), but would probably best serve as a reference for any of the technologies we're using.
- Ruby 2.2.1
- Rails 4.2 (JSON API)
- Postgres (persistence,
- Sidekiq (queueing)
- Redis (backed Sidekiq)
- ReactJS, Backbone, Semantic-UI, D3.js (frontend)
- Docker (dev/prod environment)
- Puma (rails server)
- Nginx (reverse proxy)
- memcached (caching)
While it was available, the Riot API Challenge endpoint provided a randomized
list of match ids in 5 minute intervals for each region. The endpoint required
beginTime parameter in Unix epoch time to the nearest 5 minutes. With a
simple queuing setup, the import process was able to fetch over 490,000 matches
as well as perform aggregations on fields of interest.
Zilean, the Scheduler
Since the import tools were still under development after the endpoint was made
available, a Sidekiq queue was used to schedule jobs for a worker named
Zilean matching every possible
beginTime. I used 1427866200..1429340400
(contest end date, but URF ended before that) in 300 second intervals with a
600 second offset (in case Riot was slow). This was, in my mind, the easiest way
reconcile everything I missed to when it finally catches up.
Nunu, Challenge Endpoint Consumer
Zilean job runs, it delegates the consuming of the the challenge
endpoint for each region to
Nunu, and feasting of the match ids response to
Chogath. All aspects of the import process were developed with idempotency in
mind, so if
Zilean had to re-run (perhaps due to implementation changes),
checks are made between
Chogath to ensure nothing gets inserted twice.
Chogath, Match Endpoint Feaster
Chogath iterates through the list of match IDs provided by
and feasts on the JSON responses. The requests used to be handled one by
one, which was fine at first (or with a dev API key), but was somewhat
inefficient. The solution was to modify my Ruby gem Taric to handle parallel
HTTP requests, so that simultaneous requests can be made, and
batch insert the responses into a
jsonb type column in Postgres.
Bard, the Collector
To minimize database load, a worker named
Bard was created and scheduled to
aggregate an hours worth of data on an hourly basis. Like
Zilean, jobs were
scheduled to cover the entire spectrum of the challenge endpoint's existence.
Bard's job is not idempotent since aggregation
is a 'destructive' process, and there's a chance that new matches that weren't
available at request time (e.g. KR endpoint is down) become available in a rerun
ReactJS touts itself as being the V of MVC, and since neither of us were familiar with one-way data binding via Flux, Danny settled on Backbone for the routing aspect. He was able to write React components in CoffeeScript thanks to a gem called sprockets-coffee-react. Once he was familiar with React component development with Backbone, we agreed on the single page application approach. With only a week left to finish the project, we decided the fantasy application was no longer feasible (plus URF ended), but since we had a lot of data, we decided a data exploration application would have a better shot at being completed in time. Danny had prior experience with D3.js and suggested a treemap overview of the aggregated data. The SQL queries were actually pretty fast, but I stored the results in memcached anyway to keep the site running smoothly. Danny also used Semantic-UI, which is a semantic i.e. natural language based components framework.
I have a small VPS running on Digital Ocean, and while our application is light enough to run off Heroku, I wanted to practice running services in Docker. I was able to create a separate docker-compose configuration for the production environment that gets linked to another docker-compose group of containers that has nginx. Setting up the reverse proxy to the rails server, puma, was pretty easy, and will be documented in my blog in the near future.
Installation (URF Mode)
This repository contains a Dockerfile to build the image the web app will run in, and a docker-compose.yml file to link the dependencies:
- Rails 4.2
- PostgreSQL 9.4
- Redis 2.8
Get an API key from the Riot Games Developer site.
Install Docker and Docker Compose
- If you already have Docker, be sure it's updated to v1.5.
- If you're using Kitematic on OSX, use the terminal from there.
- Docker Compose:
- A native version of Compose is not available, but you can try it in a Compose Docker container.
Setup .env file
From the project's root folder, copy the
.env.template file to
.env and change as necessary. Be sure to define your
From the project's root folder, build:
After building, run it (ctrl+c to exit):
Open a separate terminal instance and run migrations:
docker-compose run app rake db:create docker-compose run app rake db:migrate
Visit http://localhost:3000 to see if it's running.
# queuing jobs docker-compose run app rake urfantasy:queue_all # starting workers docker-compose -f docker-compose-zilean up
# queuing jobs docker-compose run app rake urfantasy:aggregate # starting workers docker-compose -f docker-compose-bard up
# console docker exec -it urfantasy_app_1 rails c # migrations docker exec -it urfantasy_app_1 rake db:migrate # postgres docker exec -it urfantasy_postgres_1 psql -U postgres urfantasy_dev