💥 Comic Cruncher
This repository contains the backend code responsible for generating all the character and appearance data as well as serving the REST API for Comic Cruncher.
Comic Cruncher started as an idea from my old Emma Frost site that ranks the character's appearances per year (you can check her graph here). I was curious what the graphs of other characters looked like, so I ran a small Go script on my computer, and then it turned into this huge thing ... so, here's the code for all who are curious.
The purpose of open sourcing this is more for show and tell
You can run this locally with or without Docker, but, again, the purpose of open sourcing this is more for show and tell.
docker-compose if you wanna make things easy. Otherwise you can run this locally on your computer without Docker (but no documentation is provided for it).
GITHUB_ACCESS_TOKEN=X make netrcto install the private dependency that hosts the library for external issue sources. (The
docker-compose.ymlwill mount the file to the
$HOMEdirectory of the container.)
- NOTE: You'll need this access token if you plan on touching most of the cerebro package.
make docker-upto create the Docker container.
make docker-dep-ensureto install the dependencies.
make docker-migrationsto run the database migrations.
- If you wanna interact with the packages that use 3rd party sources, then SET YOUR ENVIRONMENT VARIABLES on your local machine (env vars defined in
CC_MARVEL_PRIVATE_KEY: the access tokens provided by the Marvel.com API.
CC_CDN_URL: The AWS access tokens and CDN url for uploading character images to the remote storage facility.
CC_AWS_SQS_QUEUE: The AWS SQS queue for the
messagingpackage (which I have yet to actually use. Idea was to sync characters across servers). Note that the access tokens from AWS must have permissions for the specified SQS.
comic package contains the model definitions, repositories, and services for everything that makes up a comic book: publishers, characters, issues, etc - not really a comedian as the emoji suggests!
The persistence layer is backed by a Postgres database, and Redis is used to cache the simple appearances per year for a character.
cerebro is the application that generates all the characters, character issues, character sources, and appearances per year and sends them to our comic persistence! I don't manually create the characters and issues, as that would be tedious work.
Check the package README for details.
DC / Marvel
marvel packages are just small wrappers for hitting the DC and Marvel APIs and getting a list of available characters.
cerebro uses this package.
Currently a WIP. This uses the AWS SQS and sends a message to a queue when a character should be synced. For now I'm just using a cron job to sync character appearances and will come back to this package later.
search service is used currently for searching characters in our persistence layer. The implementation just uses the trigram similarities for now.
Stores images into the remote AWS S3 repository.
web package is the REST API with endpoints for characters, their appearances per year, and a character search tool. The Comic Cruncher frontend interacts with this API to serve the frontend.
make docker-up-testto create the Docker container for testing.
make docker-migrations-testto create the test database.
make docker-testto run the tests.
The test coverage could be improved!
Limit database tests to the repository layers only. Any other tests that use the repositories must be mocked. The package
internal/mocks contains generated mocks.
You can use
make docker-mockgen or
make mockgen (run
mockgen locally) to regenerate mocks.
Deployment is janky: no Puppet, no Terraform, no Kubernetes -- 10's are for work! Jobs in
.circleci/config.yml run the tests, build the application binaries, and pop the binaries directly onto the server...
The latest tagged release is production (not master), so a CircleCI build that deploys to production is triggered any time a release occurs.
Pull requests are currently closed for non-collaborators. You can read why at