Text Index lag time for realtime consuming segments

[opschronicle]

I am having Text Index and and inverted Index . And for some data Text Index not able to pull out results. for the same search Inverted Index works all the time.

Eg: - The Text Index Search for following is not working
select * from mylog where text_match(log, '1DB7BAD2-406A-D773-B1DA-291664314C4F')

but it works for
select * from mylog where text_match(log, '0D82F520-62C8-9914-14B8-4C2331E54075')

it looks like somehow some data are skipped from text_index. However both search will works fine with regexp_like inverted index.

[opschronicle]

Partial data set for not-working search is

myservicev5applog 10:52:37,468 INFO  filters.AllCheckInitFilter - [1DB7BAD2-406A-D773-B1DA-291664314C4F] Begin Transaction @ 1/12/21 10:52 AM, associated with session [6E4B98FA819713F21BCEB7A772F4E77C].

myservicev5applog 10:52:37,468 INFO  filters.AllCheckInitFilter - [1DB7BAD2-406A-D773-B1DA-291664314C4F] Processing request: /checkiot.action?

myservicev5applog 10:52:37,468 DEBUG matcher.SimplePathRequestMatcher - [1DB7BAD2-406A-D773-B1DA-291664314C4F] Checking match of request : '/checkiot.action'; against '/AppMonPool'

myservicev5applog 10:52:37,468 DEBUG matcher.SimplePathRequestMatcher - [1DB7BAD2-406A-D773-B1DA-291664314C4F] Checking match of request : '/checkiot.action'; against '/iotPub/**'

myservicev5applog 10:52:37,468 DEBUG matcher.SimplePathRequestMatcher - [1DB7BAD2-406A-D773-B1DA-291664314C4F] Checking match of request : '/checkiot.action'; against '/services/**'

[opschronicle]

@kishoreg @siddharthteotia Could you guys help me to point to correct direction to fix this issue?

[opschronicle]

@siddharthteotia I just tried now and can see the query working, So that means there is a huge delay for text Index to index realtime data? or the results to appear in search. Are there any settings that controlling the text index timings?

[opschronicle]

@siddharthteotia The behavior I am seeing is that the text index not pulling results from a segment that is in CONSUMING / IN PROGRESS status..! I am posting the consuming segment metadata here

{
"segment.realtime.endOffset": "9223372036854775807",
"segment.start.time": "-1",
"segment.time.unit": null,
"segment.flush.threshold.size": "100000",
"segment.realtime.startOffset": "0",
"segment.end.time": "-1",
"segment.total.docs": "-1",
"segment.table.name": "mylog_REALTIME",
"segment.realtime.numReplicas": "1",
"segment.creation.time": "1610430115929",
"segment.realtime.download.url": null,
"segment.name": "mylog__0__0__20210112T0541Z",
"segment.index.version": null,
"segment.flush.threshold.time": null,
"segment.type": "REALTIME",
"segment.crc": "-1",
"segment.realtime.status": "IN_PROGRESS"
}

[opschronicle]

Further Analysis showed there is roughly 3 minutes delay in Text Index updates flowing in . Following are the timelines measured for TextIndex . So it looks TextIndex is doing a three minute bulk index update rather than incremental. Is there any way to reduce this ~3 mts to instant like happening for inverted Index.

2021-01-12 18:59:15.357000
2021-01-12 19:02:15.357000
2021-01-12 19:05:15.357000
2021-01-12 19:08:02.804000
2021-01-12 19:11:05.357000

[opschronicle]

The issue is we are seeing a delay in getting realtime segment ingested. in this case it is about 3 minutes.
As per @siddharthteotia suggestion I tried to reduce the number of columns on the text index to make the text index processing more faster. However reducing the text index columns are having no effect.

[opschronicle]

@siddharthteotia @kishoreg reducing the rowcount to 1000 from 100000 is giving me a better performance of 20 seconds delay. The timings are below. However this causes the segment count to grow x100 . Could you guys please provide me the direction to fix the issue. I have to assume this is a bug and might need a code change?

2021-01-14 22:51:06.986000
2021-01-14 22:51:26.987000
2021-01-14 22:51:46.986000
2021-01-14 22:52:06.986000
2021-01-14 22:52:26.987000

[siddharthteotia]

We need to optimize the text index snapshot refresh code. It uses a single background thread to refresh index across all consuming segments. We need to make the refresh multi threaded.

[siddharthteotia]

Let me think a little bit on the details of code changes. We can sync up then.

[siddharthteotia]

Also the issue is not due to the delay in getting realtime segment ingested. This is a lag/consistency issue in the text index. We need to make the refresh faster to reduce the lag. Data is getting consumed correctly and timely into the segment

[siddharthteotia]

The issue is we are seeing a delay in getting realtime segment ingested. in this case it is about 3 minutes.
As per @siddharthteotia suggestion I tried to reduce the number of columns on the realtime table to make the text index processing more faster. However reducing the text index columns are having no effect.

I had suggested to reduce the number of column s with text index. Not to reduce the number of columns in the table. This can potentially make the refresh code faster. How did you test this?

[opschronicle]

@siddharthteotia ,sorry for the misleading comment(corrected my comment above). I did not modified the table . I tested by reducing number of columns with text index. I deleted the text index definitions for the all columns except one i.e. log.

The Steps I followed:
a) Deleted the complete table config (which deleted all Segments)
b) Restarted the server
c) Executed the new config that have only single text index column.
d) Pinot started to create segments from the topic
e) Waited for sometime for Pinot to catch up with topic
f) Executed queries to check lag

However it is not making any difference. The lag still remains the same.

But, when I reduced the number of rows per segment it does reduced the lag time. However the query performance in searches reduced slightly. I think it is because of lots of small segments. So it will be better to fix the lag time code.

[opschronicle]

@siddharthteotia If I understand correctly the issue is because of the way how text index scans the CONSUMING segments. it keeps refreshing the entire CONSUMING segments. If the segment size is large then the refreshing takes longer which causes the lag. Please correct me if I am wrong.