METRON-1162 Apply Live Messages to the Profile Debugger

[nickwallen]

I want to be able to use PROFILER_APPLY with live messages from a Metron cluster. For example, I would like to be able to grab 10 messages from my production Metron and then apply them in my debugger. This would be tremendously helpful for iterating between the debugger and a live cluster when troubleshooting.

p := PROFILER_INIT(conf)
msgs := KAFKA_GET("indexing", 10)
PROFILER_APPLY(msgs, p)
PROFILER_FLUSH(p)

The PROFILER_APPLY function does not currently accept a list of messages. It accepts a String that is a JSON list, but not a Stellar list. For PROFILER_APPLY to play happy with KAFKA_GET (and other potential uses) it needs to accept a List.

This PR makes PROFILER_APPLY rather flexible in the types of arguments that it accepts.

• String containing one JSON message
• String containing an array of JSON messages
• List containing a JSONObject
• List containing a String of one JSON message
• List containing a String that is an array of JSON messages

Testing

This can be tested by launching the REPL and running the following.

Create a Profiler definition.

[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
  "profiles": [
    {
      "profile": "hello-world",
      "onlyif":  "exists(ip_src_addr)",
      "foreach": "ip_src_addr",
      "init":    { "count": "0" },
      "update":  { "count": "count + 1" },
      "result":  "count"
    }
  ]
}

Create a sample message.

[Stellar]>>> msg := SHELL_EDIT()
[Stellar]>>> msg
{
  "ip_src_addr": "10.0.0.1",
  "protocol": "HTTPS",
  "length": "10",
  "bytes_in": "234"
}

Initialize the Profiler.

[Stellar]>>> p := PROFILER_INIT(conf)
[Stellar]>>> p
Profiler{1 profile(s), 0 messages(s), 0 route(s)}

Apply 3 messages at a time by passing in a list.

[Stellar]>>> PROFILER_APPLY([msg, msg, msg], p)
Profiler{1 profile(s), 3 messages(s), 3 route(s)}

[cestella]

Just a very small nit, but could we make that an iterable instead of a List since that's all you really need? We now have SETs in stellar too and I think we may end up creating lazy iterable transformation functions too. (e.g. PROFILE_APPLY( LAZY_MAP( [ msg1, msg2 ], m -> REGEX_APPLY(...., m))

[nickwallen]

Yes, that's better. Will do.

[nickwallen]

Done and done.

[cestella]

+1 by inspection, lgtm!