-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Many parallel subscriptions in one process => DEADLINE_EXCEEDED upon ack #550
Comments
@ctavan as always, thank you for the detailed write up!
I suspect this is exactly it. I'm a little fuzzy on the internal workings of grpc, but if you're re-using the same You could limit the number of underlying streams per subscription via const sub = pubsub.subscription('my-sub', {
streamingOptions: {maxStreams: 1}
}); Alternatively, you could try and create a new PubSub instance per Subscription. If you don't mind testing this out, that would be awesome! Otherwise I'll try and put together a reproduction case this week. |
@callmehiphop thanks for the swift reply, I think we're getting closer! The magic number where it starts failing in my setup seems to be Reducing I now even artificially restricted the number of concurrent streams to exactly So even though I'm not 100% certain that my test setup is ideal since not all of the subscriptions that I subscribe to actually receive messages during the time of testing, the seemingly exact limit of I see a few options to move forward:
Then:
WDYT @callmehiphop ? For the time being I can live with the workaround of reducing |
Came across this issue grpc/grpc-go#1514 which then lead me to this option. Sounds like grpc might default to 100, could you give the following a test run? Assuming that is the issue, I think the solution here would be to unbound the number of concurrent streams. const pubsub = new PubSub({
'grpc.max_concurrent_streams': 115
}); |
@callmehiphop unfortunately I have built a simple reproduction case that you can use to further debug this issue: https://gist.github.com/ctavan/2298c2bd965e902d4d8ad2788ffcb0b8 Just play with the two constants on top to see the errors appearing. |
I should maybe mention that I was running the reproduction case from my local machine, not GKE: So the issue doesn't seem to be environment-related. |
@kir-titievsky does the PubSub API limit the number of concurrent streams per channel? |
friendly ping @kir-titievsky @jganetsk |
Is 'grpc.max_concurrent_streams' per channel? Maybe there's another property that's specific to channels? |
I believe that is per channel, @murgatroid99 might be able to help here |
Hey @ctavan 👋 small world. Just dusting off this issue, which has gotten a little bit stale; are you still bumping into issues? any updates. |
@bcoe I'm going to go ahead and say yes, looks like there's a magical limit that I'm guessing gRPC sets, but it is pretty unclear to me how it is set and if it is configurable. I'm hoping @murgatroid99 can shed some light here |
Hey @bcoe & @callmehiphop: I just re-ran my reproduction case and can confirm that the issue still persists. Picking up my suggestions from above I see two routes moving forward:
While I think the first option is preferable I also believe that opening 100 gRPC streams on a single For the second option we should make sure to check:
I'll see if I can extend my reproduction case to answer these two questions. |
@callmehiphop @bcoe I've extended my reproduction case a bit to show that:
|
I changed this issue from a "bug" to a "feature request" of supporting multiple subscriptions in the same client. That isn't the standard use of the client, but we would like to support that feature. There's a hard limit of 100 streams per channel, and we can't do anything about that. There are a number of options to fix this:
Option 4 is a long term enhancement. Options 1-3 make the user code a bit more complicate, but are solutions that a user can implement. |
@sduskis thank you for confirming the 100 streams per channel limit. As outlined in my comment above I believe that my usage of opening more than 100 subscription streams per client is rather an edge case usage and I don't believe that it's really worth the extra complexity to implement your option 4 above. However I still believe that now that we know the 100 streams limit exists and that there is nothing we can do about it I think we should consider route 2. from my comment above:
The issue for me wasn't really that I'm not able to use more than 100 streams on one pubsub client instance, I can easily create a new client instance for each subscription. My issue was that the 100 streams limit is not documented and that the So can we agree that the client should keep track of the amount of streams and throw a descriptive error as soon as a user tries to open the 100th stream plus document this in the API docs? |
@sduskis regarding your comment:
I don't think that there is any problem with this approach, it worked very well for me. What part of my comment were you referring to? |
@ctavan, option 4 is actually still on the table, since there's work being done towards channel pooling for other products. I agree that we should give an error at some point. I'm not sure what the actual limit ought to be. 100 streams in use for receiving messages means that the messages can't be acked on that connection. I'd say that erroring out after 50 streams is safer, even if it's not the absolute limit. |
@sduskis totally fine with a safe limit of 50. Just whatever error message that is more descriptive than I furthermore suggest to move option 4 into a separate issue and keep this one open only until the error message has been added. |
@callmehiphop, can you please add the error? |
@sduskis I actually think this isn't the right place to solve this problem. If grpc receives a limit from the server and accepts an option to adjust said limit, shouldn't we ask grpc to provide a better error instead of implementing similar logic in the client? |
@callmehiphop, I think this is purely a client-side logic thing. In most cases, a client sending more than 100 streams is fine. The behavior out to be to keep a queue that waits until one of the streams is done. In unary call systems or short stream calls, which is a vast majority of use cases, that's a perfectly reasonable solution. 100 Pub/Sub subscriber streams are a special case since they stay open for an infinite amount of time. Even for Pub/Sub, Node.js (and other dynamic languages) is a special case. In Java or Go, the client would have a channel pool anyway which would allow ack-ing on a different channel. I don't think that the Pub/Sub servers should cut off more than x < 100 streams. I believe that's a pub/sub non-channel pooled client's job. The real solution is to enhance the client with channel pooling so that this scenario works. IMHO, the short term fix is to tell the user to fix it. |
@sduskis I think in that case we should look into adding support for grpc-js in grcp-gcp. I'd be happy to try and take that on if there are no objections. Otherwise I would suggest documenting this as a quirk of the PubSub client. |
+1 on a long term plan for +1 on a short term docs fix. |
Since there's nothing we can do to resolve this (until |
Environment details
@google-cloud/pubsub
version: 0.28.1Steps to reproduce
This is a follow-up report to #240, the symptoms are the same however I think I have a better understanding of what might go wrong this time.
I have made another take on finally upgrading the node pubsub library from
v0.18.0
tov0.28.1
.Here's what the problematic pod does:
DEADLINE_EXCEEDED
error event on one of the subscriptions where acks have been sent (and I crash the process).Here's the error:
For the affected subscriptions, the
num_undelivered_messages
stackdriver metric just keeps on rising:Stackdriver also indicates that there are indeed
streaming_pull_message_operation_count
operations (they keep on rising because my pod keeps on crashing and restarting, thus pulling the rising amount of unacked messages over and over again):However stackdriver doesn't show any
pull_ack_message_operation_count
operations (ignore the spike to the right, this is where I started the pod with only one subscription where it worked):Just for comparison, with version
v0.18.0
of this client library, that I had been using before, it was still usingstreamingAcks
(see discussion in #240) and everything was working well, even with the 23 parallel subscriptions in one node process:All screenshots above show the same subscription and the same timeframe, at time ~4:30pm I deployed the new version, the point where the amount of undelivered messages starts rising.
To me it looks like the amount of parallel subscriptions determines whether acks work as expected or whether I consistently receive
DEADLINE_EXCEEDED
errors as a result to theack
batches. It's interesting that acks either work for all subscriptions handled by the affected process or none.I doubt that it is CPU related because the process does basically only I/O (pipe the messages from Pubsub to BigQuery) and the CPU-graphs of GKE also show no significant CPU usage. CPU usage of the same process with
v0.18.0
of the library was also <1%.Is the number of parallel subscriptions that can be handled in a single node process (all subscription message handlers registered with a single
new PubSub()
client instance) somehow limited?Any pointer into what direction I should be looking to would be highly appreciated!
(Next thing I'd like to try out is see if I can narrow down the number of parallel subscriptions in one process where it starts to break… However due to this npm registry issue I have currently trouble deploying new code to the GKE cluster: https://status.npmjs.org/incidents/3p30y0pkblsp)
The text was updated successfully, but these errors were encountered: