Message Loss JeroMQ 0.5. due to zmq security

[RekikMohamed]

I am subscribing to an external data source.
I used two subscriber:

1. .Net6 with NetMQ
2. Java JeroMQ

In both i am subscribing to a topic and printing the data.
The message count in .Net6 is 33
The message count in Java JeroMQ is less than 10

 @Override
 public void run() {
        final String ID_MSG = "DataZMQSubscriber : ";
        int messageCount = 0;
        try {
            this.closeSocket();
            Thread.sleep(100);
            for (String topic : this.topics) {
                this.addTopic(topic);
            }
            this.getSubscriberSocket().setRcvHWM(0);
            while (this.listening) {
                try {
                    String data = this.getSubscriberSocket().recvStr();
                    if (data == null || data.isEmpty()) {
                        continue;
                    }
                    if(data.contains("\"Instance1\"")) {
                        messageCount ++;
                        System.out.println("raw data:  " + data);
                        System.out.println("message Count:  " + messageCount);
                    }

                } catch (Exception exception) {
                }
            }
        } catch (Exception exp) {
        } finally {
        }
    }

I also tried a sample application using JeroMQ and still having a massive data loss:

import org.zeromq.ZContext;
import org.zeromq.ZMQ;

public class main {
    private static Object SocketType;
    private static ZMQ.Curve.KeyPair serverPair = new ZMQ.Curve.KeyPair("P7*N&z+Nd#:h}bHBM0pjAQx}}>-G:g]vx$T=yiQr", "}r!y</A*iP[iAE-1feWqiJL@/NfKy6.eN?o@Pn{L");

    public static void main(String[] args) {
        try (ZContext context = new ZContext()) {
            System.out.println("Starting Z Context");
            ZMQ.Socket sub = context.createSocket(org.zeromq.SocketType.SUB);
            sub.connect("tcp://192.168.7.240:31554");
            int messageCount = 0;
            sub.subscribe("d37f6e3d-1cc8-4532-8c5f-cff565509182".getBytes());
            sub.setRcvHWM(0);
            sub.setCurveServerKey(serverPair.publicKey.getBytes());

            sub.setCurveSecretKey(serverPair.secretKey.getBytes());
            sub.setCurvePublicKey(serverPair.publicKey.getBytes());
            System.out.println("Subscribed");

            while (true) {
                String message = sub.recvStr();
                if(message.contains("\"Instance1\"")) {
                    messageCount ++;
                    System.out.println("raw data:  " + message);
                    System.out.println("message Count:  " + messageCount);
                }
            }
        }
    }
}

Note: After disabling the zmq security, no data loss was observed

[rajeshnair]

This looks similiar to the one I filed as #1009

[RekikMohamed]

This looks similiar to the one I filed as #1009

Yes it is similar, I think the only thing we are doing different is I am setting the HWM to 0.
One of the solution, I foud is to batch the sent data from the publisher side. However, this only reduces the data loss and does not solve it.

[rajeshnair]

One of my goals when I faced the issue was to isolate it to JeroMQ by trying similiar code in libzmq, but could not get around it .
@RekikMohamed
Do you have a standalone Java program to replicate it?
I can try looking into it

[RekikMohamed]

@rajeshnair, I used the sample application provided in the issue.
Honestly, I did not try other libraries. I tried NetMQ in .net.
However, after several tests, I discovered that removing the CurveMQ solve the issue of data loss.
Maybe it is the level of encryption used that may cause data loss