METRON-449 JSONMapParser should unfold maps to arbitrary depths

[ottobackwards]

The JSON Parser currently only unfolds maps at the root level. The parser would support a wider range of sources is it supported unfolding maps at any depth of the document.

METRON-449 adds a recursive function to unfold Maps as they are encountered when parsing. A new function was added because the enum lambdas cannot reference themselves.

[cestella]

I like it! Would you make one more test case that involves just a single map?

[cestella]

like

{ 
  "collection" : {
    "key" : "value"
  },
"key" : "value"
}

[ottobackwards]

Sure, I'll do that

[ottobackwards]

@cestella , from looking at the ES templates etc, it seems that ':' is used as the name separator and not '.', although the documented JSON Object uses '.'. Is there some conversion happening? Should the the unfolding use ':'?

[cestella]

No, ES uses ':' and modifies the keys upon write converting '.' to ':'.
Before that, '.' is the preferred source. I'm hoping we can go back to '.'
eventually. Also, non-ES indices are perfectly fine with '.'.

On Mon, Sep 26, 2016 at 11:59 AM, ottobackwards notifications@github.com
wrote:

@cestella https://github.com/cestella , from looking at the ES
templates etc, it seems that ':' is used as the name separator and not '.'.
Should the the unfolding not use the same?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#271 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAg-x-luOvx1Ebrf2a3eH3JTi9nEd5rrks5qt-v4gaJpZM4KFphg
.

[ottobackwards]

Never mind - just found the field name converter

[cestella]

+1, tested in single node vagrant.

Did the following:

• Created a config called custom at $METRON_HOME/config/zookeeper/parsers/custom.json:

{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"custom",
  "parserConfig" : {
    "mapStrategy" : "UNFOLD"
  }
}

• Created kafka topic for custom via /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic custom --partitions 1 --replication-factor 1
• Loaded the configs into zookeeper via $METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i /usr/metron/0.2.0BETA/config/zookeeper -z node1:2181
• Started the topology via $METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s custom
• Created some sample data in ~/test_data.dat:

{ "foo" : "bar" }
{ "map" : { "blah" : 1 } }
{ "foo" : "grok", "map" : { "blah" : { "tim" : "sort" } } }

• Loaded the sample data into the kafka queue via cat ~/test_data.dat | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic custom
• Checked in ES Head that I had a message with a field called map:blah:tim

Thanks a lot for your contribution, Otto!