Mike Perham edited this page Aug 22, 2018 · 3 revisions

Faktory 1.0 will switch from RocksDB storage to an "embedded" version of Redis. Redis gives us several advantages over RocksDB:

  • Copious documentation - RocksDB is a black box and hugely painful to configure correctly. No one outside of Facebook knows how to properly administer it.
  • Go-native - RocksDB is C++ and requires CGo to compile, making binaries larger and platform-specific. Builds will be much faster and easier to debug with Redis.
  • Real-time replication - RocksDB supports point-in-time backups but does not provide real-time stream of changes to a remote replica. This makes it easy to lose data if running in an unreliable environment (e.g. most cloud systems).

It is possible replication will be a commercial feature of Faktory Pro. My thinking: if you need high availability and reliability, you should have a budget for good tools and support.

Drawbacks:

  • The Redis command set is not specifically tuned for Faktory so some current Faktory features are harder than necessary or will need to be removed. A reliable pop operation from multiple queues is overly difficult, job prioritization is a O(log N) operation not O(1), etc.
  • Performance is harder to ensure if using a non-localhost Redis.

Implementation

  • Faktory will require a named socket connection to Redis, ensuring that Redis is running on localhost and therefore guaranteed to be fast.
  • Redis 3.2+ is required.

Configuration

  • faktory_redis_dir - directory which holds the RDB, defaults to /var/lib/faktory/default/
  • faktory_redis_sock - location of named socket, defaults to /var/run/faktory/default.sock
  • faktory_redis_bin - location of Redis binary, defaults to searching PATH

Faktory will start the Redis binary itself with a custom redis.conf. Faktory will not use an existing, running Redis instance; this is so Faktory controls its own Redis and access to the data. I want to avoid shared tenancy, misconfigurations and any associated support issues.

You will still be able to connect redis-cli to the named socket to muck about but beware that Faktory is running too so data can change in real-time.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.