Issues with explicitely ack() messages remaining in queue

[PILLOWPET]

Environment details

• OS: Mac OS Sonoma
• Node.js version: 18.18.1
• npm version: 9.8.1
• @google-cloud/pubsub version: 4.0.7

Steps to reproduce

1. I'm using the library locally to develop and I have a development environnement on PubSub.
2. I have a Post endpoint n my app that emits one event myEvent to a given topic foo on PubSub.
3. For this topic I have a bar subscription that is supposed to change a field in my database
4. When the DB operation is done, I call message.ack() and then the function finishes.
5. I kill the app locally and relaunch it, re-subscribing to the topic
6. I receive the same message even though it was already acked in the previous session
7. Messages are never acked and keep piling up in the Pubsub dashboard

There is likely something that I'm missing, I don't have exactly once delivery enabled but I don't understand why a message that I explicitely ack remains in the queue and is sent again when I add my subscription again to the topic by relaunching the app.
My PubSub options:

import { PubSub } from "@google-cloud/pubsub";

export const getDefaultSubscriptionOptions = (): Parameters<PubSub["createSubscription"]>[2] => ({
    ackDeadlineSeconds: 60,
    retryPolicy: {
        maximumBackoff: {
            seconds: 180,
        },
        minimumBackoff: {
            seconds: 60,
        },
    },
    flowControl: {
        maxMessages: 1,
    },
    deadLetterPolicy: {
        deadLetterTopic: `projects/dev/topics/deadletter`,
        maxDeliveryAttempts: 5,
    },
});

Thanks!

[feywind]

@PILLOWPET Did you try await sub.close() on your Subscription object before shutting down? It's likely the ack is queued but not sent in time.

[Onzeka]

Hello @feywind,

I work with @PILLOWPET on the same project and I managed to narrow down the issue. Even without shutting down the app we can see that the message is not removed from the subscription. Actually in our tests we only shut down the app to make sure that all the messages in the queue are pulled again after restarting it.

I noticed that this issue (being able to receive messages but not to acknowledge them) occurs depending on the number of active subscriptions we have in our app, if we have a number of 20 or above active subscriptions in our app the acknowledgment does not work as expected, if we have a number 19 or under active subscriptions it works fine. It may be also worth to mention that we, currently for this app, have for each distinct subscription - one distinct associated topic.

[Onzeka]

Looks like this: #1705 and this behaviour is also mentioned here https://cloud.google.com/nodejs/docs/reference/pubsub/latest/pubsub/subscription-class

We will try to implement a fix using the links mentioned above to see if it fixes our issue but if that is the case I believe that the library should raise a warning or an error when the maximum number of stream is reached or exeeded.

[feywind]

I agree that that's annoying. I'll have to check with the other language authors, because we're trying to keep them more in sync going forward, but it seems like a reasonable request.

[feywind]

Closed in favour of #1917