Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Fix our download count infrastructure. #1089
We need to save download counts for gems in a fast and reliable way.
Our current setup
Issues with the current setup
Stat-update save download counts on redis, which makes harder for us to mix that data with the data on Postgresql on the Rails side of things.
My propose fix
Uses redis as a way to offload the counts, and not as a data store for the counts.
With this approach, we wont overload Postgresql with updates for download counts, which is not a fast operation for the DB, and we wont make nginx calls slower, waiting on DB IO calls. We also can scale the ruby side workers if the queue is getting behind.
Yes, I am sure Postgres is capable of handling those updates, specially because they will be out-of-band, so we can even throttle them if we need to. The number of workers will dictate that update ratio in our DB.
Another option I was wondering we could do, to make the ruby side even easier would be to use something like Sidekiq for the LPOP part.