Queue and QueueStatus for rapns #86

Closed
wants to merge 1 commit into
from

Projects

None yet

3 participants

@shaokun
Contributor
shaokun commented Dec 9, 2012

Specifying a queue parameter when starting up a rapns instance, in order to support multiple instances of rapns in the same server or multiple servers.

Also added a QueueStatus ActiveRecord to monitor the sent_count, failed_count and heart_beart information of the rapns instance.

Added a AppSync object to sync the apps every 60 seconds and also update the heart beat of the instance.
I think it's easier to let the Instance sync the apps by itself when there are tens of instances rather than do kill -USR1 pid.

In my project, I start 10 instances in each server and there are 3 servers totally, all the 30 instances working fine so far.

This is basically a draft of my idea to run the rapns in multiple servers and I failed to run the rake test.
So no spec test for this yet.

Please let me know if this idea work or not. And then I could start to add test cases for it.

One problem is that there is probably no need to start feedback service for every instance.

Below it's the problem when I run ADAPTER=mysql2 bundle exec rake.

Using mysql2 adapter.
-- drop_table(:rapns_notifications)
-- create_table(:rapns_notifications)
/Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/mysql_adapter.rb:411:in `real_connect': Access denied for user ''@'localhost' to database 'rapns_test' (Mysql::Error)
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/mysql_adapter.rb:411:in `connect'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/mysql_adapter.rb:131:in `initialize'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/mysql_adapter.rb:38:in `new'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/mysql_adapter.rb:38:in `mysql_connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:315:in `new_connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:325:in `checkout_new_connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:247:in `block (2 levels) in checkout'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `loop'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:242:in `block in checkout'
    from /Users/wushaokun/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:239:in `checkout'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:102:in `block in connection'
    from /Users/wushaokun/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:101:in `connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_pool.rb:410:in `retrieve_connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_specification.rb:171:in `retrieve_connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/connection_adapters/abstract/connection_specification.rb:145:in `connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/migration.rb:452:in `connection'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/migration.rb:465:in `block in method_missing'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/migration.rb:438:in `block in say_with_time'
    from /Users/wushaokun/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/benchmark.rb:295:in `measure'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/migration.rb:438:in `say_with_time'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/migration.rb:458:in `method_missing'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/activerecord-3.2.9/lib/active_record/migration.rb:334:in `method_missing'
    from /Users/wushaokun/workspace/rapns/lib/generators/templates/create_rapns_notifications.rb:3:in `up'
    from /Users/wushaokun/workspace/rapns/spec/unit_spec_helper.rb:50:in `block in <top (required)>'
    from /Users/wushaokun/workspace/rapns/spec/unit_spec_helper.rb:47:in `each'
    from /Users/wushaokun/workspace/rapns/spec/unit_spec_helper.rb:47:in `<top (required)>'
    from /Users/wushaokun/workspace/rapns/spec/acceptance_spec_helper.rb:1:in `require'
    from /Users/wushaokun/workspace/rapns/spec/acceptance_spec_helper.rb:1:in `<top (required)>'
    from /Users/wushaokun/workspace/rapns/spec/acceptance/gcm_upgrade_spec.rb:1:in `require'
    from /Users/wushaokun/workspace/rapns/spec/acceptance/gcm_upgrade_spec.rb:1:in `<top (required)>'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/configuration.rb:789:in `load'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/configuration.rb:789:in `block in load_spec_files'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/configuration.rb:789:in `each'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/configuration.rb:789:in `load_spec_files'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/command_line.rb:22:in `run'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/runner.rb:80:in `run'
    from /Users/wushaokun/.rvm/gems/ruby-1.9.2-p320@rapns/gems/rspec-core-2.12.1/lib/rspec/core/runner.rb:17:in `block in autorun'
@shaokun shaokun added Queue config to rapns, support multiple queue for rapns;
added Queue attribute to notifications;
added a QueueStatus to monitor the sent_count, failed_count and heart_beat of the queue.
8e4d87d
@mattconnolly
Contributor

Why would you need multiple instances of rapns? It is designed to support multiple applications simultaneously... For high bandwidth (sending many messages quickly) rapns supports multiple connections per application. However, you're still likely to be ultimately be limited by your network bandwidth to Apple's servers.

If you have multiple rapns instances for the same application, there could be race conditions where notifications may be delivered twice.

I agree with the feedback point. I think the feedback service would only need to run for a time period (perhaps 48 hours??) after a message was sent.

@ileitch
Owner
ileitch commented Dec 9, 2012

I do like the idea of restricting rapns to a single "queue".

  • Have you benchmarked how long it takes to deliver 0.5m (or a subset) notifications with a single server using, say 10 connections? (with --no-error-checks and increase your pool setting in database.yml).
  • I agree that signaling -HUP on each server is not ideal. I need to think more about your AppSync idea.
  • Yes, Feedback is an issue...

Specs are not working because it looks like it could not detect your login? What does the output of this give you?

require 'etc'
puts Etc.getlogin
@shaokun
Contributor
shaokun commented Dec 10, 2012

@mattconnolly when create a notification, somehow I need to assign them to a instance by setting the new attribute "queue" in the notification record. So the delivered-twice shouldn't be a problem, correct?

I realized that there is already an queue concept in rapns, so probably the new "queue" I added should be renamed.

But there will be problem for QueueStatus, the increment for sent_count & failed_count, I do need a db lock to make sure the update is atomic.

@ileitch
Owner
ileitch commented Dec 10, 2012

I think a reflection API that you can then hook up to statsd would be a better approach for monitoring.

Something like this...

Rapns.reflect do |relfect|
  reflect.on_successful_delivery do |notification, duration|
    Stats.increment('notifications.delivered.count')
    Stats.timer('notifications.delivered.duration', duration)
  end

  reflect.on_delivery_failure do |notification, duration|
    Stats.increment('notifications.failed.count')
    Stats.timer('notifications.failed.duration', duration)
  end

  reflect.on_apns_connection_reset do
    Stats.increment('apns_connection_reset.count')
  end
end

I'm planning on adding a reflection API in 3.1

@shaokun
Contributor
shaokun commented Dec 10, 2012

@ileitch the reflection API looks good to me. Thanks!

I haven't being able to run this in production env yet, my project will be in production before Christmas.
Will post here more information here later.

I could run the rake test now. There are several failing after my changes currently. :( Looking into them.

@mattconnolly
Contributor

What is the real problem that queues solve? Is it that you can't send messages fast enough? Then where is the bottleneck, is it rapns or the network connection to Apple?

I'm just trying to understand why.

If this is simply about running multiple servers and load balancing between them, wouldn't this be better served by using a job queue where the rapns instances pulled jobs out of the queue? Having to explicitly set a queue requires knowledge of which instance will used to send the notification.

@shaokun
Contributor
shaokun commented Dec 11, 2012

@mattconnolly Very good questions. The problem I am trying to solve is to make rapns could scale easily because I will need to deliver push to a lot of devices (currently 0.5m), and the number is increasing.

Given that the connection of rapns is basically handled by a green thread. I am worried that I am going to face a lot of problem in the future. It would be good that I could simply start a new server and run more processes when really needed.

A job queue is a good suggestion. I was thinking about redis. When huge number of pushes needed, there is probably not good to store notifications in DB. We probably just need to make the notification.to_binary and push in to a job queue, and then the rapns instance just pull from the queue. We lose the ability to keep track of the errors in this case. That's probably another running mode of the rapns.

Currently, I make the queue selection really simple, just randomly select from an array I set. So the result will be evenly distributed.

No matter what, these changes are made without any benchmark yet. I mainly want to push here and gain more comments from you guys to see if this way makes sense.

@mattconnolly
Contributor

I like the idea of redis, too. Although I think that it would be best to put as small an amount of information into redis as possible (redis cluster is still a work in progress, and redis running out of memory is /bad/). For example, perhaps just the notification id's could be added to redis, and the actual notifications could be read from the database or saved to a distributed memcache for faster access.

I think the ultimate solution would be to use event machine sockets to connect to apple (no threads!) and redis as a queue to load balance across connections. (eg: you might have 3 rapns instances, each with 5 connections to apple, so you could send 15 notifications simultaneously.)

Still, you might find that your database becomes the bottleneck: inserting 0.5m notification records might be very very slow.

If you really need a much higher throughput, then perhaps skipping the database layer all together and using just redis might be best. In this case, you might be stepping away from rapns as a whole and perhaps just leverage the connections from your own event loop. Perhaps even separate the logic of the Rapns::Notification class into an ActiveModel class so that you can access the same validations and binary serialisation without actually saving a record to the database at all.

More food for thought....

@ileitch
Owner
ileitch commented Dec 11, 2012

If you're using 1.9 then threads are not "green". Yes, only a single thread can an run at once, BUT if a thread is blocked on I/O then another thread will run. Most of the work rapns does is I/O; db, read/write APNs connections so lock contention may not be much of an issue.

You should do some benchmarks.

  • How slow is notification creation?
  • Using development mode, how long does it take to deliver N notifications, how does that time decrease as you add more connections?

I have always wanted to add a Redis backend, maybe 3.1?

@ileitch ileitch closed this May 22, 2013
@mattconnolly
Contributor

While you're thinking about Redis, I've recently been learning a lot about ZeroMQ and that could be just as good as Redis, for "triggered" events in a remote app, but without the need to run another application. ie: Rails can talk directly to Rapns, skip the redis server. Alternatively, a "pluggable" method might be good too, just like how the threadqueue thing is currently plugged for ruby 1.8 vs 1.9...

@ileitch
Owner
ileitch commented May 22, 2013

Storage backends are also pluggable, so ZeroMQ should be possible. It is just a message transport though, so retries are not possible. Also a custom failure handler would need to be hooked up using the reflection API.

https://github.com/ileitch/rapns/wiki/Implementing-your-own-storage-backend

@ileitch
Owner
ileitch commented May 22, 2013

I also started an ActiveRecord like Redis gem a while ago: https://github.com/ileitch/modis

It's incomplete and I haven't worked on it in a long time...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment