pubsub-emulator throw error when publish 50k messages. #992
Comments
product-auto-label
bot
added
the
api: pubsub
meredithslota
added
priority: p2
|
@mrdulin Thanks for the detailed write-up! Just based on the Java error trace at the top, I'm guessing this is something with the Pub/Sub emulator, but I'll try out your repro steps. |
|
@feywind ok. Thanks. But I also hit this issue in the production environment. If I publish 50K messages without using const n = 50 * 1000;
for (let i = 0; i < n; i++) {
const data = `message payload ${i}`;
const dataBuffer = Buffer.from(data);
await topic.publish(dataBuffer)
}
// some logic ...
res.json(data);Any suggestions about how to publish a huge amount of messages without blocking the execution of subsequent logic? Do I need to use |
|
Realistically you should be able to just fire and forget (unless the Node process is terminated or something) so it does kind of seem like there's some library issue there. We're going to be pulling in some much newer versions of the pieces under the Pub/Sub library soon with the 2.0 release of the Pub/Sub library. I'm hoping that will help with some of these issues. I think the plan is to release that this week. |
|
|
|
@feywind ok. thanks. I will try it. |
|
@mrdulin Just checking in to see if you're still experiencing these issues? |
|
There hasn't been any movement on this for a while, so I'm going to go ahead and close it. (Please feel free to re-open if you need any other help with it.) |
|
@stephenplusplus @feywind The issue is gone when I upgraded the client library to |
Environment details
uname -a=>Darwin US_C02WG0GXHV2V 17.7.0 Darwin Kernel Version 17.7.0: Thu Jan 23 07:05:23 PST 2020; root:xnu-4570.71.69~1/RELEASE_X86_64 x86_64node -v=>v10.16.2npm -v=>6.14.4@google-cloud/pubsubversion:"@google-cloud/pubsub": "^1.7.3"Description
When I tried to publish 50k messages, the pubsub-emulator throw below error infinitely:
Error stack
My publisher also throws many same errors:
{ Error: Retry total timeout exceeded before any response was received at repeat (/Users/ldu020/workspace/xxx/xxx-master/workflow/node_modules/google-gax/src/normalCalls/retries.ts:83:23) at Timeout.setTimeout (/Users/ldu020/workspace/xxx/xxx-master/workflow/node_modules/google-gax/src/normalCalls/retries.ts:124:13) at ontimeout (timers.js:436:11) at tryOnTimeout (timers.js:300:5) at listOnTimeout (timers.js:263:5) at Timer.processTimers (timers.js:223:10) code: 4 } { Error: Retry total timeout exceeded before any response was received at repeat (/Users/ldu020/workspace/xxx/xxx-master/workflow/node_modules/google-gax/src/normalCalls/retries.ts:83:23) at Timeout.setTimeout (/Users/ldu020/workspace/xxx/xxx-master/workflow/node_modules/google-gax/src/normalCalls/retries.ts:124:13) at ontimeout (timers.js:436:11) at tryOnTimeout (timers.js:300:5) at listOnTimeout (timers.js:263:5) at Timer.processTimers (timers.js:223:10) code: 4 }My subscriber throw below errors:
[2020-05-06T06:29:08.932Z]Received message 46689: Data: message payload 3998 Attributes: {} [2020-05-06T06:29:08.932Z]Received message 46690: Data: message payload 3999 Attributes: {} ERROR: Error: Failed to "acknowledge" for 500 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to "acknowledge" for 100 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to "acknowledge" for 200 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to "modifyAckDeadline" for 400 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to "modifyAckDeadline" for 500 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to "modifyAckDeadline" for 100 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to "modifyAckDeadline" for 200 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline exceeded ERROR: Error: Failed to connect to channel. Reason: Failed to connect before the deadlineActually, I don't know it's a bug or I am using pubsub incorrectly. So I decide to post this as a github issue. I found there are some questions on stackoverflow and issues on github:
But didn't find a solution.
Steps to reproduce
I made a minimal code example for reproducing it. Please tell what's going on? Thanks!
Repo: https://github.com/mrdulin/nodejs-gcp/tree/master/src/pubsub/pubsub-emulator
gcloud beta emulators pubsub start --project=$PROJECT_IDAdditional information
I also got the error:
Retry total timeout exceeded before any response was receivedin production environments. So it may not be a problem of pubsub-emulator@google/pubsubversion:"@google-cloud/pubsub": "^1.6.0"That's why I am trying to make an example to reproduce it. But can't figure out what's going on.
The only way I can reproduce this error now is to post 50k messages.
Test cases: https://gist.github.com/mrdulin/79f1689a9baaafaef90fcad42646bf6d
The text was updated successfully, but these errors were encountered: