Skip to content
Comic Cruncher is the Golang repository for getting and serving a comic book character's appearances per year.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Fix gopath May 7, 2019
.github Wooohooo. Oct 7, 2018
build Update nginx config and security header. May 15, 2019
cerebro Update imports for new repository name. May 6, 2019
cmd Update imports for new repository name. May 6, 2019
comic Update imports for new repository name. May 6, 2019
dc
imaging Update imports for new repository name. May 6, 2019
internal
marvel Update imports for new repository name. May 6, 2019
search Update imports for new repository name. May 6, 2019
storage
web Update nginx config and security header. May 15, 2019
.gitattributes Ignore Oct 27, 2018
.gitignore Remove from ignore list Oct 27, 2018
Dockerfile Fixes for Docker setup for golang 11 and circleci config. May 7, 2019
LICENSE.txt
Makefile Fixes for Docker setup for golang 11 and circleci config. May 7, 2019
README.md
codecov.yml Threshold change Dec 18, 2018
docker-compose.test.yml Disable jwt middleware for now & add back temp stuff. Jan 16, 2019
docker-compose.yml Fixes for Docker setup for golang 11 and circleci config. May 7, 2019
go.mod Upgrade to echo v4. May 15, 2019
go.sum Upgrade to echo v4. May 15, 2019

README.md

💥 Comic Cruncher

CircleCI codecov CircleCI Go Report Card

❗️ Currently WIP!

This repository contains the backend code responsible for generating all the character and appearance data as well as serving the REST API for Comic Cruncher.

Comic Cruncher started as an idea from my old Emma Frost site that ranks the character's appearances per year (you can check her graph here). I was curious what the graphs of other characters looked like, so I ran a small Go script on my computer, and then it turned into this huge thing ... so, here's the code for all who are curious.

The purpose of open sourcing this is more for show and tell 💅, and there is a private dependency required to use some of the packages, so don't expect this to work out of the box.

Getting Started

You can run this locally with or without Docker, but, again, the purpose of open sourcing this is more for show and tell. 💅

You'll need docker and docker-compose if you wanna make things easy. Otherwise you can run this locally on your computer without Docker (but no documentation is provided for it).

  1. Run GITHUB_ACCESS_TOKEN=X make netrc to install the private dependency that hosts the library for external issue sources. (The docker-compose.yml will mount the file to the $HOME directory of the container.)
  • NOTE: You'll need this access token if you plan on touching most of the cerebro package.
  1. Run make docker-up to create the Docker container.
  2. Run make docker-dep-ensure to install the dependencies.
  3. Run make docker-migrations to run the database migrations.
  4. If you wanna interact with the packages that use 3rd party sources, then SET YOUR ENVIRONMENT VARIABLES on your local machine (env vars defined in docker-compose.yml):
  • CC_MARVEL_PUBLIC_KEY and CC_MARVEL_PRIVATE_KEY: the access tokens provided by the Marvel.com API.
  • CC_AWS_ACCESS_KEY_ID, CC_AWS_SECRET_ACCESS_KEY, CC_AWS_BUCKET, CC_CDN_URL: The AWS access tokens and CDN url for uploading character images to the remote storage facility.
  • CC_AWS_REGION and CC_AWS_SQS_QUEUE: The AWS SQS queue for the messaging package (which I have yet to actually use. Idea was to sync characters across servers). Note that the access tokens from AWS must have permissions for the specified SQS.

Package Information

🎭 Comic

The comic package contains the model definitions, repositories, and services for everything that makes up a comic book: publishers, characters, issues, etc - not really a comedian as the emoji suggests!

The persistence layer is backed by a Postgres database, and Redis is used to cache the simple appearances per year for a character.

🧠 Cerebro

cerebro is the application that generates all the characters, character issues, character sources, and appearances per year and sends them to our comic persistence! I don't manually create the characters and issues, as that would be tedious work.

Check the package README for details.

DC / Marvel

The dc and marvel packages are just small wrappers for hitting the DC and Marvel APIs and getting a list of available characters. cerebro uses this package.

✉️ Messaging

Currently a WIP. This uses the AWS SQS and sends a message to a queue when a character should be synced. For now I'm just using a cron job to sync character appearances and will come back to this package later.

🔎 Search

The search service is used currently for searching characters in our persistence layer. The implementation just uses the trigram similarities for now.

🗄 Storage

Stores images into the remote AWS S3 repository.

🌐 Web

The web package is the REST API with endpoints for characters, their appearances per year, and a character search tool. The Comic Cruncher frontend interacts with this API to serve the frontend.

Tests

Running tests

  1. Run make docker-up-test to create the Docker container for testing.
  2. Run make docker-migrations-test to create the test database.
  3. Run make docker-test to run the tests.

Writing tests

The test coverage could be improved!

Limit database tests to the repository layers only. Any other tests that use the repositories must be mocked. The package internal/mocks contains generated mocks.

You can use make docker-mockgen or make mockgen (run mockgen locally) to regenerate mocks.

Deployment

Deployment is janky: no Puppet, no Terraform, no Kubernetes -- 10's are for work! Jobs in .circleci/config.yml run the tests, build the application binaries, and pop the binaries directly onto the server...

The latest tagged release is production (not master), so a CircleCI build that deploys to production is triggered any time a release occurs.

Pull Requests

Pull requests are currently closed for non-collaborators. You can read why at .github/CONTRIBUTING.md.

You can’t perform that action at this time.