Handle metadata fields in search 'fields' option

[jtibshirani]

Currently the search 'fields' option skips over metadata fields. It would be great to add support for them, to help make 'fields' the central place to retrieve document content.

Some specific improvements:

• Users would no longer need to remember to load _size from docvalue_fields (Document using docvalue_fields to retrieve _size field from mapper size plugin #57103).
• We recently started allowing some metadata fields to pull values from _source, like _doc_count (Add doc_count field mapper #58339). The 'fields' option should retrieve these like any other field value.

An open question: should only return metadata fields if they're explicitly requested, or allow wildcard patterns like * to match them?

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jtibshirani]

We discussed this as a group, and agreed we'd only like to return metadata fields if they're explicitly requested. They will not be returned when requesting "fields": ["*"].

[cbuescher]

I took a first look at how this might work and opened a PR with #70575. I don't know if it was already discussed, but my guess is that we want to decide which other metadata fields to support on a case-by-case basis. For example, I don't know if we want to support "_seq_no", in any case most of these field types currently don't return their own valueFetcher but throw an exception instead. We would need to change that for the fields we want to support.

[jtibshirani]

We didn't discuss what metadata fields to return explicitly. I'm curious to hear what others think, but from my perspective, your PR is the right first step -- it covers the helpful case of _size and makes sure that all metadata fields present in _source can be retrieved. It feels less valuable to support other metadata fields like _index, _id and _routing, since these are already returned on the search hit by default. It's also not straightforward right now, since they're 'stored fields'. For _seq_no and _primary_term, these can already be requested through the seq_no_primary_term parameter.

In the future, maybe we could consider unifying the retrieval for most metadata fields through the fields option? So we wouldn't need all these specialized options like version: true and seq_no_primary_term: true. This feels like a follow-up to me though, not in scope for this issue.

[jtibshirani]

I'm closing this out since we covered the issue focus:

• We removed logic to skip over metadata fields when loading fields
• We added support for fields that don't already have special handling on the search hit: _doc_count, _size, _tier