New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using asterisk to send to all takes too long to prepare #27

Closed
cmabastar opened this Issue Jul 12, 2013 · 15 comments

Comments

Projects
None yet
2 participants
@cmabastar

cmabastar commented Jul 12, 2013

Hi,

We got a pretty huge number of users and we try to send the * to send to all. Unfortunately doing an http requests will timeout the client because the server will try to prepare its stuff.

I like how the gcm push works because its so fast. Except for APNS which is so slow.

@ghost ghost assigned monnand Jul 12, 2013

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 12, 2013

Member

I see. I can partly answer the question that why GCM is so fast. GCM supports multicast, which allows users to send multiple notifications through one single HTTP request. The user only needs to specify the registration IDs inside the request.

On the other hand, the user has to send multiple requests, one per notification, through APNS. However, this should not be that bad because all notifications are sent through one single TCP connection, meaning you don't need to re-establish the connection every time.

Let me see how can I improve the performance on my side. Can you provide me some rough numbers, like number of receivers, estimated duration of sending all notifications through APNS, etc. Just a rough estimation would be enough. I will work on it during this weekend.

Member

monnand commented Jul 12, 2013

I see. I can partly answer the question that why GCM is so fast. GCM supports multicast, which allows users to send multiple notifications through one single HTTP request. The user only needs to specify the registration IDs inside the request.

On the other hand, the user has to send multiple requests, one per notification, through APNS. However, this should not be that bad because all notifications are sent through one single TCP connection, meaning you don't need to re-establish the connection every time.

Let me see how can I improve the performance on my side. Can you provide me some rough numbers, like number of receivers, estimated duration of sending all notifications through APNS, etc. Just a rough estimation would be enough. I will work on it during this weekend.

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 12, 2013

Alright,

We currently have around 67k users, both apns/gcm. I'm not sure if we
implemented this right way but please do provide tips on how to improve our
sending of push messages
. We didn't use * to send to all. Currently doing an http request to
uniqush to send 1 message to 1 user. Using celery to do the request, we
have around 4 workers and doing 4 concurrency each.

It's doing around 2 message per sec and takes around 3hours+ to complete.

Cheers,

Christopher Marlon S. Abastar

On Fri, Jul 12, 2013 at 2:06 PM, monnand notifications@github.com wrote:

I see. I can partly answer the question that why GCM is so fast. GCM
supports multicast, which allows users to send multiple notifications
through one single HTTP request. The user only need to specify the
registration IDs inside the request.

On the other hand, the user has to send multiple request, one per
notification, through APNS. However, this should not be that bad because
all notifications are sent through one single TCP connection, meaning you
don't need to re-establish the connection every time.

Let me see how can I improve the performance on my side. Can you provide
me some rough numbers, like number of receivers, estimated duration of
sending all notifications through APNS, etc. Just a rough estimation would
be enough. I will work on it during this weekend.


Reply to this email directly or view it on GitHubhttps://github.com/uniqush/uniqush-push/issues/27#issuecomment-20859955
.

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender
immediately by email if you have received this email by mistake and delete
this email from your system. If you are not the intended recipient, you are
notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.

cmabastar commented Jul 12, 2013

Alright,

We currently have around 67k users, both apns/gcm. I'm not sure if we
implemented this right way but please do provide tips on how to improve our
sending of push messages
. We didn't use * to send to all. Currently doing an http request to
uniqush to send 1 message to 1 user. Using celery to do the request, we
have around 4 workers and doing 4 concurrency each.

It's doing around 2 message per sec and takes around 3hours+ to complete.

Cheers,

Christopher Marlon S. Abastar

On Fri, Jul 12, 2013 at 2:06 PM, monnand notifications@github.com wrote:

I see. I can partly answer the question that why GCM is so fast. GCM
supports multicast, which allows users to send multiple notifications
through one single HTTP request. The user only need to specify the
registration IDs inside the request.

On the other hand, the user has to send multiple request, one per
notification, through APNS. However, this should not be that bad because
all notifications are sent through one single TCP connection, meaning you
don't need to re-establish the connection every time.

Let me see how can I improve the performance on my side. Can you provide
me some rough numbers, like number of receivers, estimated duration of
sending all notifications through APNS, etc. Just a rough estimation would
be enough. I will work on it during this weekend.


Reply to this email directly or view it on GitHubhttps://github.com/uniqush/uniqush-push/issues/27#issuecomment-20859955
.

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender
immediately by email if you have received this email by mistake and delete
this email from your system. If you are not the intended recipient, you are
notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 12, 2013

Member

2 messages per sec? This is ridiculously slow. I will definitely dig into this problem.
I see no issues with your implementation.

Another thing which worth to notice is that APNS is in fact an asynchronous service. While uniqush-push provides a synchronize service, this means it will responds until it gets a result from APNS (or time out, which by default is 6s. Any time out will be considered a success delivery according to APNS' document.) When you send messages through uniqush, you don't need to wait the reply to send the next message. You are safe to send them simultaneously.

Which version of uniqush are you using right now? If you are using 1.4.3, then the poor performance is a known bug and fixed in the exp branch, which will be 1.4.4.

In issue #24, I said twice to download the recent builds. They have same URLs but with different content (I uploaded them twice with recent update). To make things easier, I will use different version numbers (hence different URLs) indicate different builds. They will be 1.4.4a1, 1.4.4a2, etc. If you can re-install uniqush on your server, please check out the most recent builds:

The version is 1.4.4a1.

Member

monnand commented Jul 12, 2013

2 messages per sec? This is ridiculously slow. I will definitely dig into this problem.
I see no issues with your implementation.

Another thing which worth to notice is that APNS is in fact an asynchronous service. While uniqush-push provides a synchronize service, this means it will responds until it gets a result from APNS (or time out, which by default is 6s. Any time out will be considered a success delivery according to APNS' document.) When you send messages through uniqush, you don't need to wait the reply to send the next message. You are safe to send them simultaneously.

Which version of uniqush are you using right now? If you are using 1.4.3, then the poor performance is a known bug and fixed in the exp branch, which will be 1.4.4.

In issue #24, I said twice to download the recent builds. They have same URLs but with different content (I uploaded them twice with recent update). To make things easier, I will use different version numbers (hence different URLs) indicate different builds. They will be 1.4.4a1, 1.4.4a2, etc. If you can re-install uniqush on your server, please check out the most recent builds:

The version is 1.4.4a1.

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 12, 2013

Yes, I upgraded our uniqush to the latest build you specified. This is the
wget url we used:
http://uniqush.org/downloads/uniqush-push-1.4.4-1.x86_64.rpm

Thanks for the info :)

Cheers,

Christopher Marlon S. Abastar

On Fri, Jul 12, 2013 at 3:19 PM, monnand notifications@github.com wrote:

2 messages per sec? This is ridiculously slow. I will definitely dig into
this problem.
I see no issues with your implementation.

Another thing which worth to notice is that APNS is in fact an
asynchronous service. While uniqush-push provides a synchronize service,
this means it will responds until it gets a result from APNS (or time out,
which by default is 6s. Any time out will be considered a success delivery
according to APNS' document.) When you send messages through uniqush, you
don't need to wait the reply to send the next message. You are safe to send
them simultaneously.

Which version of uniqush are you using right now? If you are using 1.4.3,
then the poor performance is a known bug and fixed in the exp branch, which
will be 1.4.4.

In issue #24 #24, I said
twice to download the recent builds. They have same URLs but with different
content (I uploaded them twice with recent update). To make things easier,
I will use different version numbers (hence different URLs) indicate
different builds. They will be 1.4.4a1, 1.4.4a2, etc. If you can re-install
uniqush on your server, please check out the most recent builds:

The version is 1.4.4a1.


Reply to this email directly or view it on GitHubhttps://github.com/uniqush/uniqush-push/issues/27#issuecomment-20862027
.

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender
immediately by email if you have received this email by mistake and delete
this email from your system. If you are not the intended recipient, you are
notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.

cmabastar commented Jul 12, 2013

Yes, I upgraded our uniqush to the latest build you specified. This is the
wget url we used:
http://uniqush.org/downloads/uniqush-push-1.4.4-1.x86_64.rpm

Thanks for the info :)

Cheers,

Christopher Marlon S. Abastar

On Fri, Jul 12, 2013 at 3:19 PM, monnand notifications@github.com wrote:

2 messages per sec? This is ridiculously slow. I will definitely dig into
this problem.
I see no issues with your implementation.

Another thing which worth to notice is that APNS is in fact an
asynchronous service. While uniqush-push provides a synchronize service,
this means it will responds until it gets a result from APNS (or time out,
which by default is 6s. Any time out will be considered a success delivery
according to APNS' document.) When you send messages through uniqush, you
don't need to wait the reply to send the next message. You are safe to send
them simultaneously.

Which version of uniqush are you using right now? If you are using 1.4.3,
then the poor performance is a known bug and fixed in the exp branch, which
will be 1.4.4.

In issue #24 #24, I said
twice to download the recent builds. They have same URLs but with different
content (I uploaded them twice with recent update). To make things easier,
I will use different version numbers (hence different URLs) indicate
different builds. They will be 1.4.4a1, 1.4.4a2, etc. If you can re-install
uniqush on your server, please check out the most recent builds:

The version is 1.4.4a1.


Reply to this email directly or view it on GitHubhttps://github.com/uniqush/uniqush-push/issues/27#issuecomment-20862027
.

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error, please notify the system manager.
This message contains confidential information and is intended only for the
individual named. If you are not the named addressee, you should not
disseminate, distribute or copy this email. Please notify the sender
immediately by email if you have received this email by mistake and delete
this email from your system. If you are not the intended recipient, you are
notified that disclosing, copying, distributing or taking any action in
reliance on the contents of this information is strictly prohibited.

monnand added a commit that referenced this issue Jul 15, 2013

monnand added a commit that referenced this issue Jul 15, 2013

monnand added a commit that referenced this issue Jul 16, 2013

monnand added a commit that referenced this issue Jul 16, 2013

monnand added a commit that referenced this issue Jul 16, 2013

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 16, 2013

Member

@cmabastar-gumi I just uploaded a new build. It should be much faster than previous versions. However, it is not fully tested yet, please use caution.

Download links:

The version is 1.4.4a2. Please let me know if there is any problem with this build.

In this version, uniqush-push uses a connection pool to connect to APNS, which means it will maintain more than one connection with APNS for each cert/key pair. This change should improve the performance, with better concurrency support. Moreover, it will not wait for the status code from APNS. This is sort of a trade-off. Because there is some useful information in the status code (e.g. invalid device token length.) However, most of the errors can be detected before sending to APNS, so I would rather not wait for the status code, which takes most of the time when sending lots of messages.

NOTE: this version does not have feedback service support. I will add it in the next build.

Member

monnand commented Jul 16, 2013

@cmabastar-gumi I just uploaded a new build. It should be much faster than previous versions. However, it is not fully tested yet, please use caution.

Download links:

The version is 1.4.4a2. Please let me know if there is any problem with this build.

In this version, uniqush-push uses a connection pool to connect to APNS, which means it will maintain more than one connection with APNS for each cert/key pair. This change should improve the performance, with better concurrency support. Moreover, it will not wait for the status code from APNS. This is sort of a trade-off. Because there is some useful information in the status code (e.g. invalid device token length.) However, most of the errors can be detected before sending to APNS, so I would rather not wait for the status code, which takes most of the time when sending lots of messages.

NOTE: this version does not have feedback service support. I will add it in the next build.

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 16, 2013

@monnand Thanks monnand, We'll try to integrate it soon in our system. I will let you know how it goes.

cmabastar commented Jul 16, 2013

@monnand Thanks monnand, We'll try to integrate it soon in our system. I will let you know how it goes.

monnand added a commit that referenced this issue Jul 16, 2013

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 16, 2013

Member

@cmabastar-gumi No problem. I hope this is not annoying: New version has just been uploaded. I added feedback service support and fixed some minor bugs.

@nevo and I are testing this version now. It is feature complete but may contain bugs.

Member

monnand commented Jul 16, 2013

@cmabastar-gumi No problem. I hope this is not annoying: New version has just been uploaded. I added feedback service support and fixed some minor bugs.

@nevo and I are testing this version now. It is feature complete but may contain bugs.

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 16, 2013

Thanks @monnand We will be testing this new build tomorrow. Probably around 10k users with 2 uniqush servers. I hope its stable and won't crash :D

cmabastar commented Jul 16, 2013

Thanks @monnand We will be testing this new build tomorrow. Probably around 10k users with 2 uniqush servers. I hope its stable and won't crash :D

monnand added a commit that referenced this issue Jul 16, 2013

monnand added a commit that referenced this issue Jul 16, 2013

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 16, 2013

Member

@cmabastar-gumi According to @nevo 's test result, alpha 3 works fine. However, I'm not sure what it will behave on 10K users. Let's hope and pray :)

Member

monnand commented Jul 16, 2013

@cmabastar-gumi According to @nevo 's test result, alpha 3 works fine. However, I'm not sure what it will behave on 10K users. Let's hope and pray :)

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 17, 2013

RPM file is not found. :(

cmabastar commented Jul 17, 2013

RPM file is not found. :(

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 17, 2013

Member

Sorry, it was a wrong url. Here we go:

Member

monnand commented Jul 17, 2013

Sorry, it was a wrong url. Here we go:

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 17, 2013

Thanks @monnand . Will be testing it now...

cmabastar commented Jul 17, 2013

Thanks @monnand . Will be testing it now...

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 17, 2013

hi @monnand , The latest uniqush is build, we were able to send 22k messages in less than 3mins and we were also able to test around 68k users to be subscribed in around 5mins. Thanks for this build! It's damn fast!

cmabastar commented Jul 17, 2013

hi @monnand , The latest uniqush is build, we were able to send 22k messages in less than 3mins and we were also able to test around 68k users to be subscribed in around 5mins. Thanks for this build! It's damn fast!

@cmabastar

This comment has been minimized.

Show comment
Hide comment
@cmabastar

cmabastar Jul 17, 2013

Just more info, we had around 8x8=64 celery workers and doing around 350-370 messages per second :D

cmabastar commented Jul 17, 2013

Just more info, we had around 8x8=64 celery workers and doing around 350-370 messages per second :D

@monnand

This comment has been minimized.

Show comment
Hide comment
@monnand

monnand Jul 17, 2013

Member

@cmabastar-gumi Wonderful! Thank you very much!

I will make some more tests and release 1.4.4 by this weekend. It won't happen without your help!

Member

monnand commented Jul 17, 2013

@cmabastar-gumi Wonderful! Thank you very much!

I will make some more tests and release 1.4.4 by this weekend. It won't happen without your help!

@monnand monnand closed this Jul 17, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment