-
Notifications
You must be signed in to change notification settings - Fork 988
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unclear docs #990
Comments
This could certainly be cleared up a bit more:
This corresponds more toward using index and type names like
In this case, it explains that you can use a special pattern (denoted by curly braces) to have the connector determine which index and type to save documents to at runtime using the values stored in the documents' fields. This is different from the above because we are resolving the resource to a single target resource at write time for each document. If you were to use this pattern with something that does not resolve to a single index (
~ versus ~
This is mostly to highlight that you can use the Does that clear things up? I'll look into expanding the documentation around this to clarify the differences in "multiple indices" for each situation. |
Yes, it does clear the things up, thank you. Is it ok to create a feature-request for an ability to explicitly pass an index for a document through it metadata (alongside with id)? |
@ov7a I think you should be able to do this currently by just specifying a field pattern as the entire index path item, like |
What if index name depends on other things (not only fields) and I do not want to store them? |
E.g. |
@ov7a Even if we were to specify a configuration property that selects the entire index from the document field before writing it, it would be the exact same functionality as the existing pattern use case I explained above. In the even that you need to implement complex logic for selecting the index to send data to, I would suggest implementing that logic as part of your transformation steps before persisting to Elasticsearch. Finally, if you are concerned about adding unneeded fields to your index mappings, you can always mark those metafields to be excluded from the final document sent to Elasticsearch by using the |
I thought that way, but es.mapping.exclude feature is ignored when es.input.json is specified :( |
Yeah, es.input.json is meant to be a performance boosting option to avoid the serialization overhead. Since to include es.mapping.exclude with that would mean we would have to parse the JSON fragments to remove them, we thought it best to leave it off. |
So, if I want es.input.json and complex index logic I have two options:
The feature request I want to propose is to provide extra metadata for each document, so it would be possible to have both json input and complex logic. |
That's fine with me to open an enhancement ticket for that. Thanks! |
Partial copy of this thread:
https://discuss.elastic.co/t/writing-to-multiple-indices-and-documentation-about-it/82859
There are some unclear things in the documentation
At some point it says:
But the next paragraph is titled "Dynamic/multi resource writes" and states that
Is seems like these two statements contradict each other. I'm still not sure is it possible to write to multiple indices or not.
Secondly, the example with timestamp also looks unclear. I think about resource as 'index/type' (please fix me if I'm wrong). In the example
timestamp is a type right? But usually we separate indices by time, not data types. This seems wrong. I would expect timestamp to be a part of index:
Is it valid resource? Will it work as expected (i.e. write to multiple indices)?
P.S. If I misplaced the issue, please specify a proper place to report it. I had zero replies at forum for almost a month. Elasticsearch/docs says: "If you find an error in the documentation, you should open an issue or pull request on the repository which contains the docs"
The text was updated successfully, but these errors were encountered: