Periodic NullPointerException on unregister and reissue when using payload view in EMA Java Api

[ligasgr]

We've recently started using ViewPayload in RT-SDK EMA (version 3.7.0.0 and 3.6.1.2 on Linux) to limit the number of fields that we receive in two of our applications. This works ok for receiving the data but unfortunately for reissue and during unregistration we are seeing issues in the API the end up with exceptions. It looks like there may be a race condition or a bug in the logic of the library (unless we are using it wrong, but we followed the examples from: https://github.com/Refinitiv/Real-Time-SDK/blob/master/Java/Ema/Examples/src/main/java/com/refinitiv/ema/examples/training/consumer/series300/ex360_MP_View/Consumer.java)

Please note that we are NOT using batch request.

Sample code that is equivalent of what we do during registration is:

OmmConsumerClient ommConsumerClient = ...;// custom consumer client
ItemRequest itemRequest = ...; // custom class keeping domainType, serviceName, itemName
Set<String> fields = ....;
Map<String, Integer> fieldIdLookup = ...;
 
Request request = EmaFactory.createReqMsg();
request.clear();
ElementList viewElementList = EmaFactory.createElementList();
OmmArray ommArray = EmaFactory.createOmmArray();
ommArray.clear();
ommArray.fixedWidth(2);
int fieldPosition = 0;
for (String field : fields) {
  if (fieldIdLookup.containsKey(field)) {
    ommArray.add(EmaFactory.createOmmArrayEntry().intValue(fieldIdLookup.get(field)));
    fieldPosition++;
  } else {
    LOG.warn("[{}] requested field {} was not found in dictionary for {}", name, field, itemRequest);
  }
}
if (fieldPosition > 0) {
  ElementEntry viewType = EmaFactory.createElementEntry().uintValue(EmaRdm.ENAME_VIEW_TYPE, 1);
  viewElementList.add(viewType);
  viewElementList.add(EmaFactory.createElementEntry().array(EmaRdm.ENAME_VIEW_DATA, ommArray));
  request.payload(viewElementList);
}
request
  .domainType(itemRequest.domainType())
  .serviceName(itemRequest.serviceName())
  .name(itemRequest.itemName())
  .interestAfterRefresh(interestAfterRefresh);
long itemHandle = ommConsumer.registerClient(request, ommConsumerClient, closure);

On the reissue side what we do is:

// reusing the same request, using the itemHandle obtained in the code above
request.clear();
ommConsumer.reissue(request, itemHandle);

And unregister:

// using the itemHandle obtained in the code above
ommConsumer.unregister(itemHandle);

The issues do not occur every time and there is no specific pattern of them happening but when they happen they produce results like the ones below:

Reissue (using version 3.7.0.0):

java.lang.NullPointerException: null
        at com.refinitiv.eta.valueadd.reactor.WlViewHandler.removeRequestView(WlViewHandler.java:381) 
        at com.refinitiv.eta.valueadd.reactor.WlItemHandler.removeRequestView(WlItemHandler.java:3816) 
        at com.refinitiv.eta.valueadd.reactor.WlItemHandler.handleReissue(WlItemHandler.java:1030) 
        at com.refinitiv.eta.valueadd.reactor.WlItemHandler.submitRequest(WlItemHandler.java:228) 
        at com.refinitiv.eta.valueadd.reactor.Watchlist.submitMsg(Watchlist.java:162) 
        at com.refinitiv.eta.valueadd.reactor.ReactorChannel.submit(ReactorChannel.java:802) 
        at com.refinitiv.ema.access.SingleItem.rsslSubmit(ItemCallbackClient.java:3026) 
        at com.refinitiv.ema.access.SingleItem.modify(ItemCallbackClient.java:2892) 
        at com.refinitiv.ema.access.ItemCallbackClient.reissue(ItemCallbackClient.java:2320) 
        at com.refinitiv.ema.access.OmmBaseImpl.reissue(OmmBaseImpl.java:581) 
        at com.refinitiv.ema.access.OmmConsumerImpl.reissue(OmmConsumerImpl.java:273) 
...

Unregister (using older version 3.6.1.2):

java.lang.NullPointerException: null
        at com.refinitiv.eta.valueadd.reactor.WlViewHandler.removeRequestView(WlViewHandler.java:373) ~[etaValueAdd-3.6.1.2.jar!/:etaj3.6.1.G1.all.rrg]
        at com.refinitiv.eta.valueadd.reactor.WlItemHandler.removeRequestView(WlItemHandler.java:3727) ~[etaValueAdd-3.6.1.2.jar!/:etaj3.6.1.G1.all.rrg]
        at com.refinitiv.eta.valueadd.reactor.WlItemHandler.removeUserRequestFromOpenStream(WlItemHandler.java:1601) ~[etaValueAdd-3.6.1.2.jar!/:etaj3.6.1.G1.all.rrg]
        at com.refinitiv.eta.valueadd.reactor.WlItemHandler.submitMsg(WlItemHandler.java:1466) ~[etaValueAdd-3.6.1.2.jar!/:etaj3.6.1.G1.all.rrg]
        at com.refinitiv.eta.valueadd.reactor.Watchlist.submitMsg(Watchlist.java:191) ~[etaValueAdd-3.6.1.2.jar!/:etaj3.6.1.G1.all.rrg]
        at com.refinitiv.eta.valueadd.reactor.ReactorChannel.submit(ReactorChannel.java:690) ~[etaValueAdd-3.6.1.2.jar!/:etaj3.6.1.G1.all.rrg]
        at com.refinitiv.ema.access.SingleItem.rsslSubmit(ItemCallbackClient.java:3055) ~[ema-3.6.1.2.jar!/:emaj3.6.1.G1.all.rrg]
        at com.refinitiv.ema.access.SingleItem.close(ItemCallbackClient.java:2907) ~[ema-3.6.1.2.jar!/:emaj3.6.1.G1.all.rrg]
        at com.refinitiv.ema.access.ItemCallbackClient.unregister(ItemCallbackClient.java:2301) ~[ema-3.6.1.2.jar!/:emaj3.6.1.G1.all.rrg]
        at com.refinitiv.ema.access.OmmBaseImpl.unregister(OmmBaseImpl.java:525) ~[ema-3.6.1.2.jar!/:emaj3.6.1.G1.all.rrg]
        at com.refinitiv.ema.access.OmmConsumerImpl.unregister(OmmConsumerImpl.java:165) ~[ema-3.6.1.2.jar!/:emaj3.6.1.G1.all.rrg]
...

Please let me know if you need any further details you may need for the investigation.

Thanks, Grzegorz

[MitchellKato]

Hi, a few questions:

Is this a multithreaded application?
How many items is the application requesting?
Which EMA dispatch mode are you using?

Thanks,
Mitchell

[MitchellKato]

One other thing, in the reissue example, the code is clearing the request message immediately prior to the calling OmmConsumer.reissue(). What is the intent behind this? The clear would imply that the view does not apply anymore, is that what the application wishes to do?

Mitchell

[ligasgr]

Hi @MitchellKato
Thanks a lot for quick response!
To answer your questions...

Is this a multithreaded application?

Yes, we run 8 concurrent OmmConsumer threads.

How many items is the application requesting?

We request one item at a time (not using any batching) but altogether these applications (two separate ones) request between 7.6k-11k items.

Which EMA dispatch mode are you using?

OperationModel.API_DISPATCH

One other thing, in the reissue example, the code is clearing the request message immediately prior to the calling OmmConsumer.reissue(). What is the intent behind this? The clear would imply that the view does not apply anymore, is that what the application wishes to do?

As mentioned in the original example code we are reusing same ReqMsg across all of the calls that we make (registration/reissue) to prevent memory pollution. I think I forgot to mention that the ReqMsg used by reissue call is a separate one from the one that we used when registerClient(). For the case of reissue we don't mind if we lose the view that we had (for the situation in which we use it).
For the case of unregister we are unregistering anyway so we definitely don't care about the view and it definitely should be removed as part of unregistration (but currently it fails).

[MitchellKato]

Thank you, we are looking into the issue.

[ligasgr]

Thanks @MitchellKato, please keep me posted!

[ligasgr]

Hi @MitchellKato
has there been any progress in trying to debug the issue?

[ayurov2]

Hi @ligasgr, we are still attempting to reproduce the problem you've identified. If you do have any additional code you can share that can recreate the issue more reliably, feel free to pass it along. Either way we want to make sure that we can reproduce it to find an appropriate solution.

[ayurov2]

Hi @ligasgr, we have found a way to accurately reproduce the issue and are looking into appropriate solutions to the problem.

[ligasgr]

That's great news, thanks a lot! Please let me know when you have something that we could test.

[vlevendel]

RTSDK 2.2.0.L1 addresses this issue. Please let us know if you have further concerns. Closing.