Messages dropped when using PUB/SUB over high-frequency in mql4zmq

[forexindicator]

Hi @AustenConrad 👋

Thanks for the great library! I’ve been using mql4zmq to integrate MT4 with external data feeds via ZeroMQ, and hit a problem that seems like it might be a bug (or at least needs better handling/documentation).

---

Environment:

• Repository: AustenConrad/mql4zmq :contentReference[oaicite:0]{index=0}
• Version: 2.0.0-pre (latest beta) :contentReference[oaicite:1]{index=1}
• MT4 Build: 600+ :contentReference[oaicite:2]{index=2}
• OS: Windows 10 x64
• ZeroMQ version: v4.x

---

Issue:
When using the PUB/SUB pattern at high message rates (e.g. publishing from external application every 100 ms or less), some messages are never received by the MT4 subscriber. The behavior is intermittent, not deterministic, so sometimes all messages arrive, sometimes one or two are dropped without any error or warning.

---

Expected Behavior:

• MT4 subscriber should receive all messages from the publisher in order (or at least reliably, without silent drops) up to the limits of ZeroMQ’s guarantees.
• If messages are dropped, it should be either documented or detectable (e.g. by sequence numbering or logging).

---

Actual Behavior:

• Some messages are not received.
• No error or corruption is reported.
• Sometimes the message receive loop inside MT4 appears to skip or ignore a message (e.g. subscriber sees messages Newer zmq version #1,2,4,5, missing Update v1.0.1 and v2 readme's #3) even though publisher sent them.

---

Code / Reproduction Steps:

// MT4 Expert: subscriber.mq4
#include <mql4zmq.mqh>

void OnStart()
{
    int socket = zmq_context_socket(ZMQ_SUB);
    zmq_setsockopt(socket, ZMQ_SUBSCRIBE, "", 0);
    zmq_connect(socket, "tcp://127.0.0.1:5555");

    while(true) {
        string msg = zmq_recv(socket);
        Print("Received: ", msg);
        Sleep(100);  // or Sleep(50)
    }
}

[AustenConrad]

Thank you for the kind words. I'm actually surprised that it still works at all! I haven't touched the code base in 12 years.

What you're experiencing is expected ZeroMQ behavior if you're using a Pub/Sub because if the publisher exceeds the rate at which the subscriber can consume the messages then ZeroMQ by default will drop messages once the buffer is full. This is because it's designed to optimize for low latency over guarantees.

To guarantee that every message is handled you would need to use ZeroMQ's Request/Reply pattern instead. However if you're exceeding Metatrader's ability to process the messages, which you likely will at a high velocity of messages, then your whole trading system will be lagging behind the market.

Might want to start by trying to set a higher high water mark on the publisher and subscribers to see if that increased buffer will handle the message rate you're sending.