Add support for dynamic runtime fields that are not added to the mapping

[felixbarny]

Description

Scenario: say a user does not want to index all fields of a document (such as a log event) by default to save on disk space and ingest time. But they still want to be able to search by any field and potentially index the fields in the future.

This sounds like a perfect fit for setting dynamic: runtime. Well, almost. The issue is that this will still add all of the fields to the mapping which makes the index prone to mapping explosions (by default only 1000 fields are allowed). While bumping the field limit is a temporary workaround, the underlying issue of the cluster state taking up a lot of heap space still remains.

So ideally, we would not add dynamic: runtime fields to the mapping. Or we make dynamic: false behave in a way where it's still possible to search by fields that aren’t declared in the mapping and instead apply runtime-field like behavior by looking values _source or stored fields. It would be similar to the behavior of flattened fields, but for runtime fields.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

Thanks for opening this issue @felixbarny . By the way, runtime fields are currently not counted as part of the total fields limit, so you can currently get away with adding as many as you want, but this is an oversight that we want to fix (see #88265) so your issue is definitely valid.

This is effectively what was proposed in #81357 for unmapped fields as well as in #80504 for mapped fields that don't have doc_values: fallback to loading values from _source automatically. I will close this as duplicate if you don't mind. We have recently discussed this matter with the team in the context of the infrastructure that the scripting fields API needs to automatically fallback to _source. We have not reached a clear conclusion yet around automatically falling back to _source (see #80504 (comment)) but we will resume this discussion soon.