Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add redis-cache and other non-kube services docs (#9)
We've talked before about how our non-kube services need ops documentation here, too, so this is my first cut at putting that info in. It's surely far from perfect, but I wanted to make sure we have a good redis config documented since I just re-built it for edgi-govdata-archiving/web-monitoring#119 (see also 77a5dc3). This also means other people can critique and improve it :)
- Loading branch information
Showing
2 changed files
with
1,376 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
# Manually Managed Services | ||
|
||
While most of our infrastructure is managed with Kubernetes, several stateful services and other components are managed separately. At current, these are managed manually, but we are considering the best tools for this, e.g. Ansible. (If you have expertise or time to contribute here, let us know in the issues or on [Slack](https://archivers-slack.herokuapp.com/)!) | ||
|
||
|
||
## Redis Cache | ||
|
||
We use Redis for caching and for queuing in Rails (that is, in [web-monitoring-db][-db]). Our Redis caching instance is currently hand-managed as an EC2 machine on AWS. Its current configuration (excepting the password) can be found in [`redis-cache.conf`](./redis-cache.conf). It also has an elastic IP. | ||
|
||
The VM and other related resources (e.g. Elastic IP) are named `wm-cache-*`. | ||
|
||
This should *probably* be a cluster (and maybe managed via AWS Elasticache?), but this has been working alright for now. | ||
|
||
|
||
## Redis Queues | ||
|
||
The asynchronous jobs that are part of the [web-monitoring-db][-db] codebase (for importing data, sending e-mails, and auto-analyzing versions) are based on Redis. At current, they are managed within Kubernetes, but that means they are prone to losing their state. These should be moved to AWS Elasticache or some manually managed system in the future (we may also move off Redis for our queues, too). | ||
|
||
TBD | ||
|
||
|
||
## Postgres via RDS | ||
|
||
[Web-monitoring-db’s][-db] database is a PostgreSQL database managed via RDS. | ||
|
||
TBD | ||
|
||
|
||
## ETL | ||
|
||
We currently run scheduled scripts for extracting data from external services (Versionista, the Wayback Machine) and sending it to [web-monitoring-db][-db] to be imported. These are managed via `cron` on a single EC2 VM. | ||
|
||
TBD | ||
|
||
|
||
[-db]: https://github.com/edgi-govdata-archiving/web-monitoring-db |
Oops, something went wrong.