Add support for CRI-O format

[jonasrutishauser]

Support container logs from nodes running with docker or with crio (https://cri-o.io/).

[jvassev]

Hi @jonasrosland, I have not environment with cri-o to test this. Is your change compatible with docker AND crio? Would Docker container logs still be processed correctly?

[jonasrutishauser]

Yes it is compatible with docker and cri-o. The docker container logs should still be parsed the same way.

The multiline parser contains 2 patterns which are tried one after the other:

• The first will match all cri-o logs and parse the corresponding fields
  

  kube-fluentd-operator/config-reloader/templates/kubernetes.conf

  Lines 13 to 15 in d79805d

  	# cri-o
  	format1 /(?<partials>.+ (stdout|stderr) P .+\n)*/
  	format2 /(?<time>.+) (?<stream>stdout|stderr) F (?<log>.*)/

• The second matches all json (docker) logs without parsing the json
  

  kube-fluentd-operator/config-reloader/templates/kubernetes.conf

  Lines 16 to 17 in d79805d

  	# docker
  	format3 /|(?<log>{.*})/

The next filter is only for cri-o logs and it will concatenate all partials.

kube-fluentd-operator/config-reloader/templates/kubernetes.conf

Lines 23 to 31 in d79805d

	<filter kubernetes.**>
	@type record_transformer
	@id filter_crio_container_logs
	enable_ruby true
	remove_keys partials
	<record>
	log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
	</record>
	</filter>

Then there is a filter which parses the json of docker logs.

kube-fluentd-operator/config-reloader/templates/kubernetes.conf

Lines 34 to 44 in d79805d

	<filter kubernetes.**>
	@type parser
	@id filter_docker_container_logs
	key_name log
	reserve_data true
	emit_invalid_record_to_error false
	<parse>
	@type json
	time_format %Y-%m-%dT%H:%M:%S.%NZ
	</parse>
	</source>

The only small issue with this solution is in cri-o logs.
If the log line is a json object it will be parsed too.

[jonasrutishauser]

I have tested with the following input:

2016-10-06T00:17:09.669794202Z stdout F The content of the log entry 1
{"log":"Log line is here\n","stream":"stdout","time":"2019-01-01T11:11:11.111111111Z"}
2016-10-06T00:17:09.669794202Z stderr P First line of log entry 2
2016-10-06T00:17:09.669794202Z stderr P Second line of the log entry 2
2016-10-06T00:17:10.113242941Z stderr F Last line of the log entry 2
2016-10-06T00:18:12.123456743Z stdout F {"foo":"some","bar":"other"}

The output just after the second filter was the following:

2016-10-06 00:17:09.669794202 +0000 kubernetes.var.log.containers.container.log: {"stream":"stdout","log":"The content of the log entry 1"}
2019-01-01 11:11:11.111111111 +0000 kubernetes.var.log.containers.container.log: {"log":"Log line is here\n","stream":"stdout"}
2016-10-06 00:17:10.113242941 +0000 kubernetes.var.log.containers.container.log: {"stream":"stderr","log":"First line of log entry 2Second line of the log entry 2Last line of the log entry 2"}
2016-10-06 00:18:12.123456743 +0000 kubernetes.var.log.containers.container.log: {"stream":"stdout","log":"{\"foo\":\"some\",\"bar\":\"other\"}"}

[jvassev]

Thank you for your PR!