Not receiving all messages

[vcrepin]

• OS: Google Container Engine (Kubernetes)
• Node.js version: gcr.io/google_appengine/nodejs
• npm version:
• @google-cloud/pubsub version: 0.16.2

I have a very simple subscription on a topic that runs in Container Engine. The code is the same as your examples:

const pubsub = require('@google-cloud/pubsub')();

const projectId = process.env.GOOGLE_CLOUD_PROJECT;
const topic = pubsub.topic('projects/' + projectId + '/topics/security-logs');
const subscription = topic.subscription('projects/' + projectId + '/subscriptions/securitySubscription');

subscription.on('error', function (err) {
    "use strict";
    console.error("Subscription error: " + err);
});

function onMessage(message) {
    "use strict";
    // Called every time a message is received.
    logger.log("Subscription message received: " + JSON.stringify(message));
    const data = Buffer.from(message.data).toString('utf-8');
    logger.log("message.data: " + data);
    message.ack();
}

subscription.on('message', onMessage);

The problem is that not all messages are received. I would say a little more than 50% are.

I see in my logs that messages are successfully sent by publishers (I get messageIds) but the subscription does not get them all. I have only one subscriber.

Any idea what might be wrong ?

[stephenplusplus]

@callmehiphop any thoughts?

[callmehiphop]

We've had issues with Kubernetes becoming unresponsive in the past - when you say you don't receive all the messages, would you also say its accurate that they stop coming in altogether?

[vcrepin]

No. IT stays responsive but some messages are missing.

[callmehiphop]

Interesting. @kir-titievsky any thoughts on why that might happen?

[vcrepin]

Is There à way to get some insight into what pubsub is doing ? Some kind of debug logs in stackdriver ? I did not find some...

[kir-titievsky]

@vcrepin The first thing would be to look at the Pub/Sub metrics in Stackdriver:

• Can you check if your backlog is growing (look for Pub/Sub subscriber metrics)?
• Are you seeing errors on pull or ack?
• Are you seeing errors on publish? Meaning, you may be logging publish calls but some of them are not succeeding.
  Next: is there anything in logs from the container? Any errors?

If you send a note to cloud-pubsub@google.com, we can work through some basic possibilities directly. If you pay for a support package with Google cloud, submitting a case is an even better starting point.

It could be that there is something tragic going on with the service, but that's not usual. We should eliminate that possibility and go from there. Other causes for this kind of behavior that I've seen: users accidentally leave another instance of their app running in the background after multiple iterations of writing the code. So half of the messages end up going to an app they do not see.

[vcrepin]

Hello and thank you for your response.

Well, I must admit that I'm ashamed... when I read your post I realized that I effectively has another old instance of my subscriber running in another namespace...

This instance was simply getting messages and outputting them on the console...

So Thank you very much and really sorry for this.

[stephenplusplus]

It happens, @vcrepin! Thanks for the help, @kir-titievsky.

[sharif9876]

We've had issues with Kubernetes becoming unresponsive in the past - when you say you don't receive all the messages, would you also say its accurate that they stop coming in altogether?

@callmehiphop I'm having this issue now, do you know of any solution for this?

[callmehiphop]

@sharif9876 What version of the PubSub client are you running? I think you might be getting hit by #770, which we released a patch for just yesterday.

[sharif9876]

@callmehiphop That seems to have fixed it! Thanks!

[callmehiphop]

@sharif9876 awesome, thanks for stopping by and letting us know!