Support non-string labels

[felixbarny]

APM has support for that but in ECS, labels are mapped as keyword

@pgomulka would you need non-string MapMessage values for Elasticsearch?

[pgomulka]

there are some fields that are non string but I format them as strings anyway so that is fine

{"type": "index_indexing_slowlog", "timestamp": "2019-06-24T15:01:11,415-04:00", "level": "WARN", "component": "i.i.s.index", "cluster.name": "esprod", "node.name": "esprod0", "message": "[foo/6cpavWDYRO6pyxezB0LepA]", "took": "3ms", "took_millis": "3", "doc_type": "t", "id": "1", "routing": "", "source": "{\"foo\":\"bar\"}", "cluster.uuid": "b28RGuVmRDimbAkNTVbacg", "node.id": "erxPlzmKQOGUdrDxGrww_g"  }
{"type": "index_indexing_slowlog", "timestamp": "2019-07-09T04:48:28,203-04:00", "level": "WARN", "component": "i.i.s.index", "cluster.name": "es1", "node.name": "es1_1", "message": "[foo/DubP3paOQJCtD1EkUmqxpw]", "took": "2.3ms", "took_millis": "2", "doc_type": "_doc", "id": "1", "source": "{\"foo\":\"bar\"}", "cluster.uuid": "xOp8Gs9TTa-SoSWcs6NZHg", "node.id": "BgSzU7SUTgeRYqzbiyf1sA"  }

[pgomulka]

@felixbarny I just checked and some fields are meant to be Arrays. At the moment these are strings, but the plan was to make them Arrays.
https://github.com/elastic/beats/pull/13179/files

Also I realised why MapMessage won't be feasible in my scenario. I still have to provide a message field which is not supported by MapMessage - unless done some tweaking.

[felixbarny]

I just checked and some fields are meant to be Arrays. At the moment these are strings, but the plan was to make them Arrays.
https://github.com/elastic/beats/pull/13179/files

I think we could make that work pretty easily as well and support things like new MapMessage(Map.of("foo", List.of("bar", "baz")))

Also I realised why MapMessage won't be feasible in my scenario. I still have to provide a message field which is not supported by MapMessage - unless done some tweaking.

Would it be an option to just use the message key in the map for that? For example, new MapMessage(Map.of("message", "hello world", "total_hits", 0))

[pgomulka]

I still maintain the previous text messages for plaintext logs. So with this example:

 indexLogger.warn( new IndexingSlowLogMessage(index, doc, tookInNanos, reformat, maxSourceCharsToLog));

This will create a JSON log line where a field message contains a string with index name

{
  "type": "index_search_slowlog",
  "timestamp": "2019-06-25T11:43:00,587-04:00",
  "level": "WARN",
  "component": "i.s.s.query",
  "cluster.name": "esprod",
  "node.name": "esprod0",
  "message": "[.monitoring-es-7-2019.06.25][0]",
  "took": "6.2ms",
  "took_millis": "6",
  "total_hits": "3 hits",
  "types": "",
  "stats": [],
  "search_type": "QUERY_THEN_FETCH",
  "total_shards": "1",
  "source": "{\"size\":10000,\"query\":{\"bool\":{\"filter\":[{\"term\":{\"type\":{\"value\":\"cluster_stats\",\"boost\":1.0}}},{\"range\":{\"timestamp\":{\"from\":null,\"to\":null,\"include_lower\":true,\"include_upper\":true,\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"sort\":[{\"timestamp\":{\"order\":\"desc\"}}],\"collapse\":{\"field\":\"cluster_uuid\"}}",
  "id": "",
  "cluster.uuid": "kZOAmikdTFKUOSxTGj59Bg",
  "node.id": "gfFy7VKWSXqOOOE1no4s0Q"
}

but if someone creates their own pattern layout, or uses old plain text pattern then it requires %m field. And to be backwards compatible it contains all these fields formatted in a structured log line.

[2018-06-29T10:06:14,933][INFO ][index.search.slowlog.query] [v_VJhjV] [metricbeat-6.3.0-2018.06.26][0] took[4.5ms], took_millis[4], total_hits[19435], types[], stats[group1,group2], search_type[QUERY_THEN_FETCH], total_shards[1], source[{"query":{"match_all":{"boost":1.0}}}],

It is worth testing if ECSLayout will support both thse log usages log.info(new MapMessage(...) and log.info("some message"). To make sure it won't duplicate the message field.

[felixbarny]

That works with this change: #30

MapMessage supports rendering the message as JSON which EcsLayout is now delegating the rendering to. So you could either use MapMessage, which would render the plain text message slightly differently (key="value"). Or you could create your own class which implements MultiformatMessage (or better MultiFormatStringBuilderFormattable) and which supports a JSON and a plain-text format.

[felixbarny]

closed in #30