Ability to explicitly pass an index/type for a document through it metadata

[ov7a]

Feature description

Start of discussion: #990 (comment)

Currently, if one wants to use complex logic for getting index name for document, there are two options:

1. Use patterns to construct an index name
2. Implement index name as part of transformation steps before persisting to Elasticsearch and add extra field to resulting document.

If logic is very complex then the first option is either to hard or impossible to implement. With the second option, extra field can be eliminated with es.mapping.exclude feature.

Unfortunately, es.mapping.exclude feature is ignored when es.input.json is specified. So, currently it is impossible to use complex logic, json input and have no extra fields in mapping.

I propose to add an INDEX option to document metadata to provide an explicit index for document.

Same thoughts can be applied to type of a document.

[jbaiera]

Related (loosely) to #956

[whitfin]

To work around es.mapping.exclude, I believe you can exclude the fields in your index mapping in ES itself. Wasted bytes on the wire, but at least it won't be persisted.

[jbaiera]

I'm closing this feature request out because it is not really an item that anyone will be working on and conflicts with a few features in the ES-Hadoop connector - primarily the feature of a writer "pinning" itself to a primary node - In the event that an index pattern is specified, we allow writers to pin to any node they want. This mode works with the "save with metadata" feature. The problem is that the moment you allow "save with metadata" to provide an index or type during runtime, we need to turn off the selective host targeting for any use of "save with metadata".

Adding more configurations to get around this feels clunky for a feature that is already supported with existing fields and configurations. If more control over the serialization is needed, we provide it already with es.mapping.exclude. While this indeed does not work with raw json data, the idea for writing raw json is to avoid the overhead of serialization logic in the connector.

If this gets enough push from the community, we can discuss a way forward, but for now I am closing this.