-
Notifications
You must be signed in to change notification settings - Fork 24.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pipeline processors fails when field is not present #19995
Comments
I can imagine this situation is pretty common. Wondering if |
Will open a PR for this shortly. a new |
Hitting this a lot across many processors. For example running into the issue with |
@djschny I am scared of there being too many ways to control pipeline flow using exception handling. Mind sharing how you are hitting this with other processors and why the |
Sure, consider the following simple example and processor:
In the above the field For many of these processors the absence of a field, a field with a null value, and potentially even an empty string field should all be ignored and not throw an error IMO. |
I'm augmenting the title to be generic as so far I have run into this issue when implementing pipelines and for the following processors:
|
@djschny I've update the PR to include these, except for |
Cool thanks @talevy However from an external user perspective, consistency would be better. Externally To have to set it one way on some processors and another way on others just seems odd IMO. |
I can't disagree there, I am all for consistency. It is just not the case that all processors
Although, this may be a common theme (extracting data from one field) for many (if not all) of the existing processors, this is more of a coincidence rather than a rule. Since processors are can be as flexible as "fetch data from somewhere else, and inject in document"(#20340) or "translate all field values to Spanish", the As you can see by the search-processor discussion, the scope of what processors can/will/should do is still contested. I would like to avoid introducing further generalizations that may become difficult to move away from in the future, as things stabilize. TL;DR I think the what do you think? |
I'm all fine with not making it a global/common property. No problem there and understand the concern. However for the |
++ I'll add it to the rename |
Saw that the PR merged, thanks! However I noticed Were there any particular considerations into why defaulting to |
because it is explicit. we tell you by default if there is a problem, but we give you the tools to deal with it as you need. |
Closing since this has been added to the discussed processors here: #20194 |
Looks like this is a problem on the |
Elasticsearch version: v5.0.0-alpha5
Plugins installed: []
JVM version: 1.8.0_92-b14
OS version: Mac 10.11.15
Description of the problem including expected versus actual behavior:
The
convert
pipeline processor throws an error if the target field to convert does not exist in the document. Ideally adding an option ofignore_missing
or similar would exist and would default to true so that way for the majority of users it just works. Theignore_failure
option would not work in this scenario because we would still want to fail in the situation where there was a parse error or other unknown error that happened.This can be illustrated with the very common/poster-child example of apache log data where we would have the equivalent Logstash config in an ingest node pipeline:
In this situation
bytes
is optional and not present in the case of HEAD/OPTIONS/DELETE requests. If thebytes
field is not present ideally the pipeline should still work for an end user. See related elastic/beats#2229 for original discovery and background information.CC @talevy as talked briefly about it earlier this morning
The text was updated successfully, but these errors were encountered: