Skip to content

Commit

Permalink
Add redis-cache and other non-kube services docs (#9)
Browse files Browse the repository at this point in the history
We've talked before about how our non-kube services need ops documentation here, too, so this is my first cut at putting that info in. It's surely far from perfect, but I wanted to make sure we have a good redis config documented since I just re-built it for edgi-govdata-archiving/web-monitoring#119 (see also 77a5dc3). This also means other people can critique and improve it :)
  • Loading branch information
Mr0grog committed Dec 3, 2018
1 parent b1e8a9e commit 5c107d1
Show file tree
Hide file tree
Showing 2 changed files with 1,376 additions and 0 deletions.
36 changes: 36 additions & 0 deletions manually-managed/README.md
@@ -0,0 +1,36 @@
# Manually Managed Services

While most of our infrastructure is managed with Kubernetes, several stateful services and other components are managed separately. At current, these are managed manually, but we are considering the best tools for this, e.g. Ansible. (If you have expertise or time to contribute here, let us know in the issues or on [Slack](https://archivers-slack.herokuapp.com/)!)


## Redis Cache

We use Redis for caching and for queuing in Rails (that is, in [web-monitoring-db][-db]). Our Redis caching instance is currently hand-managed as an EC2 machine on AWS. Its current configuration (excepting the password) can be found in [`redis-cache.conf`](./redis-cache.conf). It also has an elastic IP.

The VM and other related resources (e.g. Elastic IP) are named `wm-cache-*`.

This should *probably* be a cluster (and maybe managed via AWS Elasticache?), but this has been working alright for now.


## Redis Queues

The asynchronous jobs that are part of the [web-monitoring-db][-db] codebase (for importing data, sending e-mails, and auto-analyzing versions) are based on Redis. At current, they are managed within Kubernetes, but that means they are prone to losing their state. These should be moved to AWS Elasticache or some manually managed system in the future (we may also move off Redis for our queues, too).

TBD


## Postgres via RDS

[Web-monitoring-db’s][-db] database is a PostgreSQL database managed via RDS.

TBD


## ETL

We currently run scheduled scripts for extracting data from external services (Versionista, the Wayback Machine) and sending it to [web-monitoring-db][-db] to be imported. These are managed via `cron` on a single EC2 VM.

TBD


[-db]: https://github.com/edgi-govdata-archiving/web-monitoring-db

0 comments on commit 5c107d1

Please sign in to comment.