SAMZA-2517 : Adding handling and relevant exception message for null in oldest offset from system-admin#1353
Conversation
…set from system-admin
| Map<SystemStreamPartition, String> offsets = new HashMap<>(); | ||
| for (SystemStreamPartition ssp : systemStreamToSsp.get(systemStream)) { | ||
| if (partitionMetadata == null || partitionMetadata.get(ssp.getPartition()) == null) { | ||
| offsets.put(ssp, null); | ||
| } else { | ||
| offsets.put(ssp, partitionMetadata.get(ssp.getPartition()).getOldestOffset()); |
There was a problem hiding this comment.
This map seems redundant. Why not directly put to oldestOffsets?
| // Because of https://bugs.openjdk.java.net/browse/JDK-8148463 using lambda will NPE when getOldestOffset() is null | ||
| Map<SystemStreamPartition, String> offsets = new HashMap<>(); | ||
| for (SystemStreamPartition ssp : systemStreamToSsp.get(systemStream)) { | ||
| if (partitionMetadata == null || partitionMetadata.get(ssp.getPartition()) == null) { |
There was a problem hiding this comment.
SystemStreamMetadata#getSystemStreamPartitionMetadata is also used in CoordinatorStreamStore, ContainerStorageManager and other places, we should also do null handling there right?
Isn't it better to do at StreamMetadataCache level?
There was a problem hiding this comment.
the null values are on oldestOffset in the returned metadata-object not the returned objectl itself (returned val is by us and is not-null it appears).
|
@rmatharu the change lgtm once the minor comment on unit test is resolved |
|
Let's update description with https://cwiki.apache.org/confluence/display/SAMZA/SEP-25%3A+PR+Title+And+Description+Guidelines before merging. |
5 participants
Problem: When a Kafka broker returns null for oldest-offset, because of a OpenJDK bug (https://bugs.openjdk.java.net/browse/JDK-8148463), the collect() operation in TaskSideInputStorageManager can fail with a misleading null-pointer-exception message.
Note however that, Kafka broker cannot return null for oldest-offset.
Cause: Above.
Fix: The fix is to use the suggested workaround for the bug https://bugs.openjdk.java.net/browse/JDK-8148463
And update the exception and log messages in case the oldest-offset are returned as null from Kafka.
API changes: None
Upgrade Instructions: None