Nesting a key under an existing key results in duplicate keys

[robinkb]

Bug Report

Describe the bug

Attempting to nest a key under a key that already exists results in duplicate keys.
Elasticsearch does not accept duplicate keys, meaning that the record cannot be indexed.

To Reproduce

fluent-bit configuration to reproduce the problem:

[SERVICE]
    Flush        5
    Daemon       Off
    Config_Watch On

[INPUT]
    Name  dummy
    Tag   dummy
    Dummy {"key": "value", "nested": {"other_key": "value"}}

[FILTER]
    Name       nest
    Match      dummy
    Operation  nest
    Wildcard   key
    Nest_under nested

[OUTPUT]
    Name  stdout
    Match dummy

This results in the following data structure, as logged through stdout:

[0] dummy: [1551985434.000610724, {"nested"=>{"other_key"=>"value"}, "nested"=>{"key"=>"value"}}]

Expected behavior

No duplicate keys appear, resulting in the following data structure:

{"nested"=>{"other_key"=>"value","key"=>"value"}}

Your Environment

• Version used: 1.0.4

See the configuration snippet above.

Additional context

Because fluent-bit generates a data structure with duplicate keys, I cannot index some logs in Elasticsearch.

[fhoy]

This seems similar to the situation described in https://groups.google.com/d/msg/fluent-bit/bTEewmXEKFM/AM1zLr-VBgAJ.

[edsiper]

cc: @michiel

[michiel]

@fhoy, @edsiper This observation is correct - the nest filter currently does not nest into an existing map, only into a new one.

[edsiper]

FYI: I pushed fix 1d14886 to handle duplicated keys for plugins that converts data back to JSON