Convert feature of mutate filter does not work for a added field that was created using %

[mursabogdan]

Hello.

I have a problem regarding the mutate filter.

To be more specific, in a logstash config, I added a new field like this:

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{IPORHOST:remote_ip} %{IPORHOST:client_ip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] %{NUMBER:response_time} "%{NOTSPACE:request} %{NOTSPACE:route} %{NOTSPACE:protocol}" %{NUMBER:response_code} %{NUMBER:response_size} "%{DATA:referer}" "%{DATA:user_agent}"" }
}
mutate {
add_field => [ "response_code_%{response_code}", 1 ]
}
mutate {
convert => [ "response_code_%{response_code}", "integer" ]
}
mutate {
convert => {
"response_code" => "integer"
"response_size" => "integer"
"response_time" => "integer"
"offset" => "integer"
}
}
}
}

So, for example if the response_code is 200 I expect a new field "response_code_200" with value 1.
{
....
"response_code_200" => "1"
}

All good until here, since I know that by default when you add a numerical value to a new field it is made string.

The problem is in the next section of mutate, when I try to convert to integer. I can't manage to convert the value to integer under any circumstances.
I receive as output always a string.

{
....
"response_code_200" => "1"
}

I tried as well to add_field/ convert using hash syntax, still no result.

mutate {
add_field => { "response_code_%{response_code}" => 1 }
}
mutate {
convert => { "response_code_%{response_code}" => "integer" }
}

A big observation was that if I don't use %{response_code} in the key name in the conversion phase, everything works.

E.g: response_code = 200

mutate {
add_field => { "response_code_%{response_code}" => 1 }
}
mutate {
convert => { "response_code_200}" => "integer" }
}

OUTPUT:
{
....
"response_code_200" => 1
}

I look forward to hearing from you,
Bogdan Mursa.

[dmarkhas]

There doesn't seem to be any solution for this, except for using Ruby filter directly.

[GogLlundain]

Can we add this as a feature request? How do we do that?

[pallix]

I think it would be very useful to convert fields automatically or based on regexp matching the fields names. It would be a useful complement to the kv filter which create dynamically fields.