GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table

[abhijeetkushe]

Table Config

  "tableName": "workflowEvents",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "eventTimestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "workflowEvents",
    "replicasPerPartition": "1",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "365",
    "segmentPushType": "APPEND"
  },
  "tenants": {
    "broker":"DefaultTenant",
    "server":"DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kinesis",
      "stream.kinesis.topic.name": "prod-rel-cdp-dl-workflow-metrics-stream",
      "region": "us-east-1",
      "shardIteratorType": "LATEST",
      "stream.kinesis.consumer.type": "lowlevel",
      "stream.kinesis.fetch.timeout.millis": "30000",
      "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
      "realtime.segment.flush.threshold.size": "5000000",
      "realtime.segment.flush.threshold.time": "1d"
    }
  },
  "upsertConfig": {
    "mode": "FULL"
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "metadata": {
    "customConfigs": {}
  }
}

• I upgraded pinot to 0.9.1 version from 0.8.0 last month and after that I started seeing GetRecord.IteratorAgeInMilliseconds metrics displaying in correct values
• Yesterday I observed the data stopped being displayed in the Table from the Query Console even though the GetRecords metrics showed that records were read but where not read from the Table.There were some new data fields that were added to the record but the schema was not updated
• I decided to change Table config to AT_SEQUENCE_NUMBER and I did not see the changes immediately but this morning I was able to see those records but post that when records were added this morning they were not being reflected in the table
• I decided to switch the Table config back to LATEST.I also saw that the old segments got terminated and new segments were created but the records were still not reflected in the Table
  So both the data issue and GetRecord.IteratorAgeInMilliseconds still persist

[abhijeetkushe]

[KKcorps]

Looking into it.

[abhijeetkushe]

I have more updates.I do see the data in the table but not realtime. shardIteratorType as LATEST.I ran the below query
select $segmentName,ToDateTime(eventTimestamp,'yyyyMMddHHmm') from workflowEvents order by eventTimestamp desc limit 100

$segmentName	todatetime(eventTimestamp,'yyyyMMddHHmm')
workflowEvents__0__58__20220125T2005Z	202201260000
workflowEvents__1__60__20220125T2106Z	202201260000
workflowEvents__1__60__20220125T2106Z	202201251640
workflowEvents__1__60__20220125T2106Z	202201251640
workflowEvents__1__60__20220125T2106Z	202201251640
workflowEvents__1__60__20220125T2106Z	202201251640
workflowEvents__1__60__20220125T2106Z	202201251640
workflowEvents__1__60__20220125T2106Z	202201251640

8 Records are sent at ~1543

Kinesis Get Records shows that it is read at 1643

From the server logs records do get read at 1643 but I do not see corresponding events at 1543
16:43:05.029 INFO [LLRealtimeSegmentDataManager_workflowEvents__1__60__20220125T2106Z] [workflowEvents__1__60__20220125T2106Z] Consumed 12 events from (rate:0.19965394/s), currentOffset={"shardId-000000000001":"49622318632194742639402514501316476653895236003058155538"}, numRowsConsumedSoFar=43, numRowsIndexedSoFar=43