PubSub: Synchronous pull raises 504 after a few seconds if no messages available

[plamut]

Forked from googleapis/google-cloud-python#7709 (comment).

Environment details

Tested with the following:

• PubSub v1.0.2
• Python 3.7
• Ubuntu 18.04.

It is reported that the issue does not occur with in PubSub version v0.45.0.
Update: However, this could not be confirmed.
Confirmed by @HemangChothani

Steps to reproduce

1. Create a topic.
2. Create a subscription.
3. With no messages published to the topic, perform a synchronous pull.

Actual result:
After a few seconds a 504 DeadlineExceeded exception is raised.

Expected result:
Assuming that the network functions normally, there should be no error, and the subscriber.pull() should simply return an empty list of messages.

Mind that there is no error if the return_immediately=True flag is used, but the latter is set to be deprecated and should thus not be used.

Code sample

The following script might be useful:

reproduce.py

import logging
import os
import threading
import time

from google.cloud import pubsub_v1


script_dir = os.path.dirname(os.path.realpath(__file__))
log_filename = os.path.join(script_dir, "subscriber.log")

log_format = (
    "%(levelname)-8s [%(asctime)s] %(threadName)-33s "
    "[%(name)s] [%(filename)s:%(lineno)d][%(funcName)s] %(message)s"
)

logging.basicConfig(
    level=logging.DEBUG,
    format=log_format,
)

logger = logging.getLogger(__name__)

PROJECT_NAME = "my-project"
TOPIC_NAME = "my-events"
SUBSCRIPTION_NAME = f"subscription_{TOPIC_NAME}"


def main():
    subscriber = pubsub_v1.SubscriberClient()

    subscription_path = subscriber.subscription_path(PROJECT_NAME, SUBSCRIPTION_NAME)
    topic_path = subscriber.topic_path(PROJECT_NAME, TOPIC_NAME)
    # TODO: if topic does not exist, you must first create it
    subscriber.create_subscription(name=subscription_path, topic=topic_path)

    print("Starting a pull...")
    response = subscriber.pull(
        subscription_path, max_messages=3,
        # return_immediately=True
    )

    received_messages = response.received_messages

    if not received_messages:
        print("No messages received")
        return

    print("response >>", response)

    for message in response.received_messages:
        print(message.message.data.decode("utf-8"))
    
    print("Sending ACK...")
    subscriber.acknowledge(
        subscription_path, [msg.ack_id for msg in response.received_messages]
    )
    print("... ACK sent.")


if __name__ == "__main__":
    main()

[plamut]

For the reference, I tried running the script both with PubSub version 1.0.2 and 0.45.0, and got the same result (an error occurs).

[HemangChothani]

@plamut I have tried running the script with many versions and master branch it only passes with 0.45.0 and 1.0.0 and it fails with 1.0.1 and 1.0.2 and master branch so I found the change that has been done in PR 9279 and it's related to initial_rpc_timeout_millis parameter of messaging in pubsub/google/cloud/pubsub_v1/gapic/subscriber_client_config.py file. I have changed it to the 25000 as previous and it works fine for me.

[plamut]

@HemangChothani Thanks for the research! (cc: @hongalex ).

Will add this to the agenda of the next PubSub meeting.

[cesarmarroquin]

Hello, I have been running into an issue on my project when using this pub sub library. I am seeing the same error google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded that @plamut ran into. After some research, this thread was the best thing I could find that seemed relevant, and matched the error that I ran into.

I am using:
PubSub v1.1.0
Python 3.7
Ubuntu 18.04.

The only difference with me is that I got it after calling:
pubsub_v1.PublisherClient().list_topics

I would appreciate some help, I am not sure if my issue is related to this one, but the error messages do match up which led me to believe this might be the cause.

[hongalex]

Ack. Will be reverting initial_rpc_timeout_millis back to 25000 shortly.

[plamut]

@cesarmarroquin Listing topics is a different method on the backend, and it's likely not related to the synchronous pull. The 504 Deadline Exceeded errors can occur randomly due to temporary network issue or other reasons.

Does the error occur often, or just every now and then when using the list topics method? Does your application otherwise work, generally speaking? Any additional info would be appreciated, thanks!

[plamut]

This should have been fixed here by reverting the timeout to the previous value, closing the issue.