[FEATURE] Optimize Storage of Large Query Sources

[ansjcy]

Is your feature request related to a problem?

Currently, we store the entire query source in top queries regardless of size, potentially causing large memory footprint issues.

What solution would you like?

• We can simply ignore storing queries with large source content, though this would result in empty query source displays in the detail page.
• Or, we can compress large query sources by extracting only essential features - specifically, store just the query shape for queries with large sources.

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[LilyCaroline17]

Currently this PR: #424 is trying to address the issue of source being too large for storage in the local index by converting it to a string. However, this has some possible issues with how existing nodes store source so we would have to migrate as well as having to unpack the string in memory (which may lead to truncation) and more bugs down the line.

Implementation details:

• Stored in memory: SearchSourceBuilder
• Stored in local index: String
• Index mapping definition : "source": { "type": "keyword", "index": false }
• Export: XContentHelper.convertToJson(source); or directly giving the source as a string
• Read: hit.getSourceAsMap().get("source") (may need to parse back to SearchSourceBuilder)
• Handle truncation - It would be fine if stored as a string, but anything that requires the json would lead to issues in conversion.
• Compatibilities - We would have to migrate the older versions so that they convert their local index sources to string.

Another alternate solution is to store the source as a string in a separate field for local index. If we are unable to store the source (like in this PR we see that #411 errors will show up), then we store the source as a string. With this case, we maintain backward compatibility and preserve the source data as much as possible while trying to reduce the processing that would come later. However, this is more complex and confusing to have two different versions of the same information possibly stored, which can lead to more bugs down the line and slower processing.

Implementation details:

• Stored in memory: SearchSourceBuilder
• Stored in local index: SearchSourceBuilder (default) or String (if can't store)
• Index mapping definition : "source": { "type": "object" } and "source_string": { "type": "keyword", "index": false }
• Export: Directly giving the SearchSourceBuilder or (XContentHelper.convertToJson(source); or directly giving the source as a string)
• Read: hit.getSourceAsMap().get("source") if not null, else hit.getSourceAsMap().get("source_string")
• Handle truncation - It would be fine if stored as a string, but anything that requires the json would lead to issues in conversion.
• Compatibilities - Old versions all have existing "source" as SearchSourceBuilder and can ignore "source_string"

A second alternate solution is we always store the source as a single string. This solution has a consistent storage format which is easier to implement, debug, and look at moving forward (expecting source to be the same). However, we would have to convert existing historical top n queries's sources to be in this format. We would have to convert source to string after the grouping calculation.

Implementation details:

• Stored in memory: String
• Stored in local index: String
• Index mapping definition : "source": { "type": "keyword", "index": false }
• Export: XContentHelper.convertToJson(source); or directly giving the source as a string
• Read: hit.getSourceAsMap().get("source") (may need to parse back to SearchSourceBuilder)
• Handle truncation - It would be fine if stored as a string, but anything that requires the json would lead to issues in conversion.
• Compatibilities - We would have to migrate the older versions so that they convert their local index sources to string.

[ansjcy]

We had a discussion around this, basically there are two issues we need to solve:

First, we need to create an upper limit for the source field to prevent indexing failures when sources are too large. Storing source as a string enables truncation and solves this issue - All proposed approaches address this.

Second, we need to handle backward compatibility when existing indices already have the source field defined as an object - During rolling upgrades, writing string data to indices with object mappings will cause conflicts, potentially losing data for that entire day.

IMO the dual-field approach (storing both source as object and source_string as keyword) can resolve backward compatibility with zero data loss. However, this introduces unnecessary storage overhead by duplicating the same data in two fields, which is also an anti-pattern and creates long-term technical debt.

We can improve the single-field approach by adding an "adaptive" write logic that detects the existing index mapping and writes in the appropriate format. If an index has the old object mapping, write source as an object (only for that specific day). If it has the new keyword mapping, write source as a string (this should also be the default). The read logic already handles both formats in the mentioned PR. This maintains backward compatibility without data duplication and without data loss as well.

[dzane17]

I like storing the source as string in both memory and local index. String will allow us to truncate large source values without needing to maintain a valid object structure.

A couple implementation pointers:

1. The top_queries index template which contains the local index mapping is created when there is no existing template. On upgrade from <3.4 to 3.4, the template will already exist so it won't pickup any new changes. See here. Some options are to 1) refresh the index template on node creation, 2) update the template daily during index creation, or 3) create a template with new name whenever we change the source code mapping.
2. We need to ensure changing the index mapping will not impact upgrades. Specifically migration of old local indices via shard copying, reindex, snapshot restore, etc... My understanding is that templates are applied during index creation so only reindex would be affected.