Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate job management system to Redis #2

Closed
let4be opened this issue Jun 10, 2021 · 0 comments
Closed

Migrate job management system to Redis #2

let4be opened this issue Jun 10, 2021 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@let4be
Copy link
Owner

let4be commented Jun 10, 2021

While current "queue-like system" on top of clickhouse worked quite well for testing it's no near as good as required for any serious high-volume use

Recently I did some testing on a beefy AWS hardware and fixed some internal bottlenecks(not yet merged) and in some testing scenarios where I could temporary alleviate the last left bottleneck - job distribution(writing new/updating completed/selecting), Crusty was capable of doing over 900MiB/sec - a whooping 7+gbit/sec! on 48 core(96 logical) c5.metal with a 25gbit/s port

New job queue should be solely redis-based using redis modules: https://redis.io/topics/modules-intro
rust has good enough library to allow writing redis module logic: https://github.com/RedisLabsModules/redismodule-rs

We will use pre-sharded queue(based on addr_key)

Atomic operations:

  1. Enqueue jobs
  2. Dequeue jobs
  3. Finish jobs

using correct underlying data types(mostly sets and bloom filter for history) + batching and pipelining we can have solid throughput, low cpu usage per redis node, decent reliability and scalability
careful expiration could help to avoid memory overflow on redis node - we always discover domains faster than we can process them

@let4be let4be self-assigned this Jun 10, 2021
@let4be let4be added the enhancement New feature or request label Jun 10, 2021
@let4be let4be changed the title Migrate job management system to Apache Kafka + Redis(Bloom Filter) Migrate job management system to Redis Jun 10, 2021
@let4be let4be closed this as completed in 20afb5d Jun 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant