Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Allow indices_options parameter for datafeed configuration #48056

Closed
benwtrent opened this issue Oct 15, 2019 · 3 comments · Fixed by #52793
Closed

[ML] Allow indices_options parameter for datafeed configuration #48056

benwtrent opened this issue Oct 15, 2019 · 3 comments · Fixed by #52793
Assignees
Labels
>enhancement :ml Machine learning

Comments

@benwtrent
Copy link
Member

In certain situations it is valuable to allow missing indices (e.g. a well defined datafeed configuration that takes advantage of ECS). Right now, if one of the indices in the supplied index pattern does not exist, we throw an error on _start. If we exposed an indices_options type of parameter (that serializes to an IndicesOptions object), that will allow more advanced users to provide custom expansion options to handle their use case.

@benwtrent benwtrent added >enhancement :ml Machine learning labels Oct 15, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@benwtrent benwtrent self-assigned this Oct 15, 2019
@jasonrhodes
Copy link
Member

Related issue we're having on the logs/ML integration side: elastic/kibana#48219

@droberts195
Copy link
Contributor

One particular effect of not being able to configure indices options on the datafeed config is that it's currently impossible to configure a datafeed to search frozen indices because you have to explicitly set the ignore_throttled=false option to do this.

@benwtrent benwtrent self-assigned this Feb 25, 2020
benwtrent added a commit that referenced this issue Feb 27, 2020
This adds a new configurable field called `indices_options`. This allows users to create or update the indices_options used when a datafeed reads from an index. 

This is necessary for the following use cases:
 - Reading from frozen indices
 - Allowing certain indices in multiple index patterns to not exist yet

These index options are available on datafeed creation and update. Users may specify them as URL parameters or within the configuration object.
 
closes #48056
benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Feb 27, 2020
This adds a new configurable field called `indices_options`. This allows users to create or update the indices_options used when a datafeed reads from an index.

This is necessary for the following use cases:
 - Reading from frozen indices
 - Allowing certain indices in multiple index patterns to not exist yet

These index options are available on datafeed creation and update. Users may specify them as URL parameters or within the configuration object.

closes elastic#48056
benwtrent added a commit that referenced this issue Feb 27, 2020
This adds a new configurable field called `indices_options`. This allows users to create or update the indices_options used when a datafeed reads from an index.

This is necessary for the following use cases:
 - Reading from frozen indices
 - Allowing certain indices in multiple index patterns to not exist yet

These index options are available on datafeed creation and update. Users may specify them as URL parameters or within the configuration object.

closes #48056
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :ml Machine learning
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants