Market Open/Close Data Delay #56
Description
I am using the NASDAQ Python SDK client to consume market data for the NLSUTP topic. The data is fetched using the NCDSClient and processed in real-time. I send the data received from the consumer via WebSocket. During normal market hours, I get multiple responses from WebSocket within 1 second, which works as expected. However, during market opening and closing timestamps, I experience significant delays in receiving messages. These delays are sometimes more than 15 seconds, and occasionally up to a few minutes, which is problematic for my real-time application.
Code Snippet
Below is the relevant portion of my code:
def init_nasdaq_kafka_connection(topic):
security_cfg = {
"oauth.token.endpoint.uri": os.getenv("NASDAQ_KAFKA_ENDPOINT"),
"oauth.client.id": os.getenv("NASDAQ_KAFKA_CLIENT_ID"),
"oauth.client.secret": os.getenv("NASDAQ_KAFKA_CLIENT_SECRET"),
}
kafka_cfg = {
"bootstrap.servers": os.getenv("NASDAQ_KAFKA_BOOTSTRAP_URL"),
"auto.offset.reset": "latest",
"socket.keepalive.enable": True,
}
ncds_client = NCDSClient(security_cfg, kafka_cfg)
consumer = ncds_client.ncds_kafka_consumer(topic)
logger.info(f"Success to connect NASDAQ Kafka server for topic {topic}.")
return consumer
# Usage
consumer = init_nasdaq_kafka_connection("NLSUTP")
while True:
messages = consumer.consume(num_messages=1000000, timeout=0.25)
if messages:
logger.info(f"Received {len(messages)} messages from Kafka topic {topic}.")
response = makeRespFromKafkaMessages(messages)
# Sending response via WebSocket During normal hours I received data like this where you can see I get multiple records in 1 second:
2025-01-02 15:48:33,471 - INFO - Received 551 messages from Kafka topic NLSUTP.
2025-01-02 15:48:33,688 - NFO - Received 516 messages from Kafka topic NLSCTA.
2025-01-02 15:48:33,804 - INFO - Received 391 messages from Kafka topic NLSUTP.
2025-01-02 15:48:34,012 - INFO - Received 518 messages from Kafka topic NLSCTA.
2025-01-02 15:48:34,116 - INFO - Received 299 messages from Kafka topic NLSUTP.
2025-01-02 15:48:34,393 - INFO - Received 386 messages from Kafka topic NLSCTA.
2025-01-02 15:48:34,471 - INFO - Received 519 messages from Kafka topic NLSUTP.
2025-01-02 15:48:34,763 - INFO - Received 523 messages from Kafka topic NLSCTA.
2025-01-02 15:48:34,797 - INFO - Received 339 messages from Kafka topic NLSUTP.
2025-01-02 15:48:35,144 - INFO - Received 361 messages from Kafka topic NLSUTP.
2025-01-02 15:48:35,153 - INFO - Received 429 messages from Kafka topic NLSCTA.
2025-01-02 15:48:35,676 - INFO - Received 418 messages from Kafka topic NLSUTP.
2025-01-02 15:48:35,684 - INFO - Received 544 messages from Kafka topic NLSCTA.
2025-01-02 15:48:36,211 - INFO - Received 790 messages from Kafka topic NLSUTP.
2025-01-02 15:48:36,233 - INFO - Received 847 messages from Kafka topic NLSCTA.
2025-01-02 15:48:36,667 - INFO - Received 636 messages from Kafka topic NLSUTP.
2025-01-02 15:48:36,708 - INFO - Received 955 messages from Kafka topic NLSCTA.
2025-01-02 15:48:36,999 - INFO - Received 489 messages from Kafka topic NLSUTP.
2025-01-02 15:48:37,073 - INFO - Received 660 messages from Kafka topic NLSCTA.
2025-01-02 15:48:37,334 - INFO - Received 482 messages from Kafka topic NLSUTP.
2025-01-02 15:48:37,416 - INFO - Received 612 messages from Kafka topic NLSCTA.
2025-01-02 15:48:37,648 - INFO - Received 390 messages from Kafka topic NLSUTP.
2025-01-02 15:48:37,760 - INFO - Received 444 messages from Kafka topic NLSCTA.
2025-01-02 15:48:37,938 - INFO - Received 331 messages from Kafka topic NLSUTP.
2025-01-02 15:48:38,061 - INFO - Received 409 messages from Kafka topic NLSCTA.
2025-01-02 15:48:38,243 - INFO - Received 363 messages from Kafka topic NLSUTP.
2025-01-02 15:48:38,364 - INFO - Received 427 messages from Kafka topic NLSCTA.
2025-01-02 15:48:38,555 - INFO - Received 498 messages from Kafka topic NLSUTP.
2025-01-02 15:48:38,745 - INFO - Received 375 messages from Kafka topic NLSCTA.
2025-01-02 15:48:38,879 - INFO - Received 371 messages from Kafka topic NLSUTP.
2025-01-02 15:48:39,055 - INFO - Received 505 messages from Kafka topic NLSCTA.
2025-01-02 15:48:39,186 - INFO - Received 505 messages from Kafka topic NLSUTP.
2025-01-02 15:48:39,362 - INFO - Received 407 messages from Kafka topic NLSCTA.
2025-01-02 15:48:39,512 - INFO - Received 665 messages from Kafka topic NLSUTP.
2025-01-02 15:48:39,817 - INFO - Received 1535 messages from Kafka topic NLSCTA.
2025-01-02 15:48:39,867 - INFO - Received 429 messages from Kafka topic NLSUTP.
2025-01-02 15:48:40,360 - INFO - Received 827 messages from Kafka topic NLSUTP.
2025-01-02 15:48:40,365 - INFO - Received 1243 messages from Kafka topic NLSCTA.
2025-01-02 15:48:40,807 - INFO - Received 716 messages from Kafka topic NLSUTP.
2025-01-02 15:48:40,850 - INFO - Received 1100 messages from Kafka topic NLSCTA.
2025-01-02 15:48:41,357 - INFO - Received 881 messages from Kafka topic NLSUTP.
2025-01-02 15:48:41,380 - INFO - Received 936 messages from Kafka topic NLSCTA.
2025-01-02 15:48:41,800 - INFO - Received 570 messages from Kafka topic NLSCTA.
But during market close I get this:
2025-01-02 15:54:32,483 - INFO - Received 5021 messages from Kafka topic NLSCTA.
2025-01-02 15:54:33,279 - INFO - Received 9576 messages from Kafka topic NLSUTP.
2025-01-02 15:54:34,778 - INFO - Received 6613 messages from Kafka topic NLSCTA.
2025-01-02 15:54:35,818 - INFO - Received 7784 messages from Kafka topic NLSUTP.
2025-01-02 15:54:37,750 - INFO - Received 9807 messages from Kafka topic NLSCTA.
2025-01-02 15:54:38,574 - INFO - Received 8654 messages from Kafka topic NLSUTP.
2025-01-02 15:54:41,174 - INFO - Received 10448 messages from Kafka topic NLSCTA.
2025-01-02 15:54:41,369 - INFO - Received 7315 messages from Kafka topic NLSUTP.
2025-01-02 15:54:45,762 - INFO - Received 15863 messages from Kafka topic NLSCTA.
2025-01-02 15:54:49,316 - INFO - Received 10563 messages from Kafka topic NLSCTA.
2025-01-02 15:54:49,977 - INFO - Received 28275 messages from Kafka topic NLSUTP.
2025-01-02 15:54:53,229 - INFO - Received 11273 messages from Kafka topic NLSCTA.
2025-01-02 15:54:59,452 - INFO - Received 18081 messages from Kafka topic NLSCTA.
2025-01-02 15:55:00,635 - INFO - Received 33904 messages from Kafka topic NLSUTP.
2025-01-02 15:55:04,286 - INFO - Received 13621 messages from Kafka topic NLSCTA.
2025-01-02 15:55:12,597 - INFO - Received 26616 messages from Kafka topic NLSCTA.
2025-01-02 15:55:17,115 - INFO - Received 51512 messages from Kafka topic NLSUTP.
2025-01-02 15:55:23,031 - INFO - Received 31851 messages from Kafka topic NLSCTA.
......
2025-01-02 15:56:14,999 - INFO - Received 65342 messages from Kafka topic NLSCTA.
2025-01-02 15:56:33,856 - INFO - Received 95397 messages from Kafka topic NLSUTP.
2025-01-02 15:56:40,135 - INFO - Received 87480 messages from Kafka topic NLSCTA.
2025-01-02 15:57:03,987 - INFO - Received 100067 messages from Kafka topic NLSUTP.
2025-01-02 15:57:07,468 - INFO - Received 93486 messages from Kafka topic NLSCTA.
2025-01-02 15:57:39,012 - INFO - Received 113623 messages from Kafka topic NLSUTP.
2025-01-02 15:57:41,337 - INFO - Received 100002 messages from Kafka topic NLSCTA.
2025-01-02 15:58:10,056 - INFO - Received 100027 messages from Kafka topic NLSCTA.
2025-01-02 15:58:10,730 - INFO - Received 100028 messages from Kafka topic NLSUTP.
2025-01-02 15:58:37,849 - INFO - Received 100201 messages from Kafka topic NLSUTP.
2025-01-02 15:58:39,215 - INFO - Received 100095 messages from Kafka topic NLSCTA.
2025-01-02 15:59:05,271 - INFO - Received 100345 messages from Kafka topic NLSUTP.
2025-01-02 15:59:23,013 - INFO - Received 156137 messages from Kafka topic NLSCTA.
2025-01-02 15:59:37,753 - INFO - Received 117444 messages from Kafka topic NLSUTP.
2025-01-02 15:59:49,769 - INFO - Received 100063 messages from Kafka topic NLSCTA.
2025-01-02 16:00:14,354 - INFO - Received 118266 messages from Kafka topic NLSUTP.
Observations
- During normal market hours, the WebSocket receives multiple responses within 1 second, which works as expected.
- During market opening and closing timestamps:
- The data flow is delayed significantly.
- Delays range from 15 seconds to a few minutes.
- The WebSocket responses also experience delays during this period.
Expected Behavior
The data should have minimal delay, even during the high-activity periods of market opening and closing. The WebSocket should receive multiple responses within 1 second, similar to normal market hours.
Questions
- Is this delay expected behavior for the NASDAQ Python SDK client during market open/close times?
- Are there any configuration options or optimizations to reduce these delays?
- Could this issue be related to server-side throttling, Kafka settings, or network latency?
Environment Details
- NASDAQ Python SDK version: Latest
- Python version: 3.10.6
- OS: Windows 10