Solidify the approach to allowing typeless requests on an index with a custom type.

[jtibshirani]

In #35790, we introduced support for making typeless API calls against an index with a custom type name. For example, given an existing index with type my_type, we allow new documents to be indexed using the typeless endpoint index/_doc/id. This is functionality is important in supporting the move to typeless APIs, since many users are likely to be working with older indices that were created with a custom type.

The approach in #35790 is fairly fragile, and we would like to step back and consider if there is something cleaner and more solid we could do instead. To help document the current implementation, here is the list of PRs associated with this work:

• Make typeless APIs usable with indices whose type name is different from _doc. Make typeless APIs usable with indices whose type name is different from _doc #35790
• Rewrite SourceToParse with resolved docType. Rewrite SourceToParse with resolved docType #36921
• Make sure to resolve the type in DocumentMapperParser#extractMappings. Make sure to use the resolved type in DocumentMapperService#extractMappings. #37451
• Give precedence to the type from index creation when applying templates. Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. #37871
• Treat put-mapping calls with _doc as a top-level key as typed calls. Treat put-mapping calls with _doc as a top-level key as typed calls. #38032

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-search

[jtibshirani]

@jpountz I assigned this to you, but please feel free to ping me if we should find another owner.

[jpountz]

I looked into this over the past two weeks. Unfortunately I haven't been able to come up with an approach that would feel more solid. On the positive side over the months we have made more and more internal APIs typeless, such as moving from MapperService#documentMapper(String type) to MapperService#documentMapper(). Most call sites are now using the latter. This doesn't prevent bugs, but it helps.

I will keep looking into it over the next weeks, but I don't expect big changes, rather small changes in the same spirit as the one mentioned above.

[jtibshirani]

Thanks @jpountz, that makes sense to me! I mentioned this earlier, but I am fairly concerned about the amount of complexity we've been introducing into the mapping code related to mixed typed and typeless APIs. I was wondering if there were some smaller-scale refactors we could do to clarify or unify the points where we're resolving _doc to the custom type name. Although we have REST test coverage, it would also be good to have unit tests (I find it easier to get good coverage with unit tests, and they also help in quickly iterating on bugs).

[jpountz]

I've been thinking a bit more about this one recently. In general I like the current approach that resolves the type as high in the stack as possible a bit better as it makes things easier to reason about. Having it eg. in MapperService#documentMapper would feel a bit more dangerous to me as it has many call sites and not all of them expect _doc should be an alias of the current type of the index.

I'm leaning towards addressing this issue through testing. I've been reviewing places where we had gaps in testing. One clear one is MetaDataMappingService, which I tried to address via #40578.