Track Lucene field usage

[ywelsch]

Adds a field usage API that reports shard-level statistics about which Lucene fields have been accessed, and which parts of the Lucene data structures have been accessed.

Field usage statistics are automatically captured when queries are runnning on a cluster. A shard-level search request that accesses a given field, even if multiple times during that request, is counted as a single use.

$ GET /my-index-000001/_field_usage_stats

{
    "_shards": {
        "total": 1,
        "successful": 1,
        "failed": 0
    },
    "my-index-000001": {
        "shards": [
            {
                "tracking_id": "MpOl0QlTQ4SYYhEe6KgJoQ",
                "tracking_started_at_millis": 1625558985010,
                "routing": {
                    "state": "STARTED",
                    "primary": true,
                    "node": "gA6KeeVzQkGURFCUyV-e8Q",
                    "relocating_node": null
                },
                "stats" : {
                    "all_fields": {
                        "any": "6", <1>
                        "inverted_index": {
                            "terms" : 1,
                            "postings" : 1,
                            "proximity" : 1, <2>
                            "positions" : 0,
                            "term_frequencies" : 1,
                            "offsets" : 0,
                            "payloads" : 0
                        },
                        "stored_fields" : 2,
                        "doc_values" : 1,
                        "points" : 0,
                        "norms" : 1,
                        "term_vectors" : 0
                    },
                    "fields": {
                        "_id": {
                            "any" : 1,
                            "inverted_index": {
                                "terms" : 1,
                                "postings" : 1,
                                "proximity" : 1,
                                "positions" : 0,
                                "term_frequencies" : 1,
                                "offsets" : 0,
                                "payloads" : 0
                            },
                            "stored_fields" : 1,
                            "doc_values" : 0,
                            "points" : 0,
                            "norms" : 0,
                            "term_vectors" : 0
                        },
                        "_source": {...},
                        "context": {...},
                        "message.keyword": {...}
                    }
                }
            }
        ]
    }
}

I've benchmarked this change on the http_logs track and could not see any substantial changes in performance. I think we will only get full validation of this by merging this change and having our nightly benchmarks catch any changes in performance.

Relates #73944

[jpountz]

I like how you are checking the flags that are used in TermsEnum#postings rather than wrapping the PostingsEnum, which helps avoid adding a wrapper class over something that is called in tight loops.

Do we need to actually check whether impacts are used? As a Lucene developer, I'd be curious about this, but I don't know if users would be since this isn't actionable (Lucene doesn't allow you to disable indexing impacts).

[ywelsch]

Do we need to actually check whether impacts are used? As a Lucene developer, I'd be curious about this, but I don't know if users would be since this isn't actionable (Lucene doesn't allow you to disable indexing impacts).

Good point. I wasn't sure whether those were configurable or not. I've removed them.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jpountz]

If I read the PR correctly, it currently returns raw counts which are impacted by the number of shards. E.g. a 2-shards index would have 2x the values of a single-shard index. Should we automatically divide the counts for one index by the number of shards it has so that numbers across different indices are easier to compare?

[ywelsch]

@dnhatn would love to have your thoughts on this ^^

I'm wondering if we could have TranslogLeafReader somehow embedded in SingleDocDirectoryReader so that we always apply the wrapper, but then lazily load the InMemoryLeafReader when there is access to more than what the TranslogLeafReader can offer.

[ywelsch]

I've built a solution for this. I won't push it to this PR though as it is quite a big change to how TranslogLeafReader works. Will make a separate PR tomorrow if test show all is fine with that solution.

[ywelsch]

I've opened #74722 for this.

[jpountz]

This looks good to me overall.

One question I wonder about is whether the response should return stats per-shard instead of merging them. If we were to dogfood this by indexing it into Elasticsearch to track field usage over time, having per-shard information would be better as we could more easily figure when resets happen due to shard relocation or node shutdowns. Said otherwise this could be treated as a source of metrics where the index name, shard number, field name and data structure are dimensions, and the number of times it's been used is a counter.

[jpountz]

I wonder if we should have an object for all data structures for consitency, e.g.

        "all_fields": {
            "any": "6",
            "inverted_index": {
                "terms" : 1,
                "postings" : 1,
                "proximity" : 1,
                "positions" : 0,
                "term_frequencies" : 1,
                "offsets" : 0,
                "payloads" : 0
            },
            "stored_fields" : {
                "any": 2
            },
            [...]

[ywelsch]

One question I wonder about is whether the response should return stats per-shard instead of merging them. If we were to dogfood this by indexing it into Elasticsearch to track field usage over time, having per-shard information would be better as we could more easily figure when resets happen due to shard relocation or node shutdowns. Said otherwise this could be treated as a source of metrics where the index name, shard number, field name and data structure are dimensions, and the number of times it's been used is a counter.

Good question. In that case, tracking per shard would not even be enough as we have multiple replicas for which we are already summing up the stats (those can come and go as well). It would have to be tracking per some kind of "shard copy".

We suffer here from the same issue as other existing stats, e.g., query_total in "search stats". Will think a bit more about it.

[ywelsch]

My current thinking is to change the API for now to not aggregate the data beyond the level of shard copies. As there is not a good identifier for shard copies (allocation id of a primary shard e.g. stays same after node restart), I would go with just a UUID for each shard instance, similar to how we have done it in the repository metering API (repository_ephemeral_id). This follows a similar idea s the repository metering API. Similar to the repository metering API, we can also keep a bit historical data on the node when a shard moves away to a different node, but otherwise (node restarts / crashes) will have to accept that the information is lost.

It would look like the following:

shards: [
  {
    "tracking_id" : "SOME_UUID",
    "tracking_started_at" : 123141414 // unix timestamp
    "tracking_stopped_at" : 123441414  // only available if shard stopped / moved away -> historical info
    "stats": {
      "terms" : 0,
      "proximity" : 0,
      "stored_fields" : 1,
      ...
  },
  {
    "tracking_id" : "SOME_OTHER_UUID",
    ...
  }
]

[ywelsch]

@jpountz this is ready for another review :)