Splunk logging driver truncates messages at 4K size when splunk-format is set to raw

[kyaparla]

Splunk logging driver truncates messages at 4K size, when splunk-format is setup to RAW. However, with splunk-format as inline, we are able to send messages >10K.

@sharonx, @chenziliang

[thaJeztah]

Only think I see using 4k is the stream channel;

moby/daemon/logger/splunk/splunk.go

Lines 57 to 58 in f653485

	// Number of messages allowed to be queued in the channel
	defaultStreamChannelSize = 4 * defaultPostMessagesBatchSize

These values can be overridden with environment variables though;

moby/daemon/logger/splunk/splunk.go

Lines 254 to 256 in f653485

	postMessagesBatchSize = getAdvancedOptionInt(envVarPostMessagesBatchSize, defaultPostMessagesBatchSize)
	bufferMaximum = getAdvancedOptionInt(envVarBufferMaximum, defaultBufferMaximum)
	streamChannelSize = getAdvancedOptionInt(envVarStreamChannelSize, defaultStreamChannelSize)

moby/daemon/logger/splunk/splunk.go

Lines 65 to 67 in f653485

	envVarPostMessagesBatchSize = "SPLUNK_LOGGING_DRIVER_POST_MESSAGES_BATCH_SIZE"
	envVarBufferMaximum = "SPLUNK_LOGGING_DRIVER_BUFFER_MAX"
	envVarStreamChannelSize = "SPLUNK_LOGGING_DRIVER_CHANNEL_SIZE"

[kyaparla]

@thaJeztah I think the environment variables here are to limit how often to send batch of events, and how many messages to store in cache..etc as documented here https://docs.docker.com/config/containers/logging/splunk/#advanced-options, the problem I see is for raw format messages, number of characters allowed per message is 4096, any thing bigger than is chopped. But for inline format message, messages are bigger than 10K bytes, but I heard the limit is 16K. This issue is to identify if there is any difference in the way raw messages are processed compared to inline.

[sharonx]

cc @luckyj5

There shouldn't be any difference between the how raw and inline messages are processed. I'll verify it. At the same time, you could try to increase the channel size through env variable SPLUNK_LOGGING_DRIVER_CHANNEL_SIZE and see if it helps.

[kyaparla]

@sharonx

I see these lines, in the code, not sure if "prefix.Bytes()" has anything to do with limiting message to 4K chars.

loggerWrapper = &splunkLoggerRaw{logger, prefix.Bytes()}

But not seeing prefix.Bytes for inline or json format.

loggerWrapper = &splunkLoggerJSON{&splunkLoggerInline{logger, nullEvent}}

[sharonx]

@kyaparla I actually cannot reproduce this issue. The messages I receive are truncated at 16kb which is docker daemon's limit.

I'm using Docker version 18.03.1-ce, build 9ee9f40

Command I run is

docker run --log-driver=splunk --log-opt splunk-url=https://splunk-server:8088 --log-opt splunk-token=XXXXXXX --log-opt splunk-insecureskipverify=true --log-opt splunk-format=raw -d test-logging

[dcvtruong]

@kyaparla I am seeing similar behavior where the logs from the log field getting truncated at 16kb. I can reproduce this consistently.

Is there a workaround for setting the log size to 64k? The log is being truncated into four separate log.

Running docker-ce 18.6.1 and I'm not using splunk.