Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add auto detection to convert processor #17139

Closed
martijnvg opened this issue Mar 16, 2016 · 5 comments

Comments

Projects
None yet
3 participants
@martijnvg
Copy link
Member

commented Mar 16, 2016

The ingest processor is able to convert string values to either a boolean, integer or float value.
This processor needs to be configured in a strict manner. For example field_a need to be converted to an integer. If the field doesn't exist or can't be its value can't converted to an integer this processor fails.

The idea is here the add an auto convert type option that tries to convert strings to either a integer/float or a boolean value based on simple rules:

  • true | false -> boolean
  • [-+]?[0-9] -> integer
  • [-+]?[0-9].?[0-9] -> float

If the auto convert type is unable to convert than it just does nothing and leaves that field untouched.

In order to auto detect a number fields it would be helpful if the field option would support wildcards too.
For all fields that match the convert processor will convert the values based on the configured convert type. If no fields match with the wildcard pattern than the convert processor does nothing.

I think if this gets implemented then this can supersede number detections in mappings?

@kimchy

This comment has been minimized.

Copy link
Member

commented Mar 16, 2016

another potential option for this is to to create a new field when converting, only if we manage to convert it. So, if you have field1, and we try to convert it to numeric, we will create field1_num with the numeric value, but if we can't convert, we can let ingest still work, just not create the field1_num field.

@martijnvg

This comment has been minimized.

Copy link
Member Author

commented Mar 17, 2016

Maybe we can add target_field option? Any conversion done by this processor will be put in that field.

@talevy talevy self-assigned this Mar 22, 2016

@talevy

This comment has been minimized.

Copy link
Contributor

commented Mar 22, 2016

^^^ kind of solves this. Missing target_field for now

I wanted to get something cleared away with target_field first. How should List-valued fields behave with this target field? I am ok with treating it just as another type and setting a whole new field with the new list value.

@martijnvg

This comment has been minimized.

Copy link
Member Author

commented Mar 23, 2016

How should List-valued fields behave with this target field?

I think like any other field, only here multiple conversions will take place and all the these converted values are placed into the new target field which will be an array too.

@talevy

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2016

that's what I thought, just wanted to double check. will update PR shortly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.