[receiver/filelog] encoding not applied to multiline option

[DougManton]

Component(s)

receiver/filelog

What happened?

Description

SAP audit log files have utf-16le encoding, continuously updated with new logs, fixed length records, and no line termination. This should be supportable using the Filelog receivers encoding and multiline support, but I've got a reproducible bug and workaround.

Steps to Reproduce

1. Create a utf-16le file named auditlog.txt containing 10 SAP audit log records in the format:

2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        

2. Configure filelog receiver as following:

receivers:
  filelog/sap:
    include: [ auditlog.txt ]
    encoding: utf-16le
    multiline:
      line_start_pattern: '([23])[A-Z][A-Z][A-Z0-9]\d{14}00'
    preserve_trailing_whitespaces: true
    start_at: beginning

Expected Result

10 log events

Actual Result

1 log event

Workaround

I suspected the multiline processing is not honouring the file encoding, and therefore failing to match the pattern. To test this theory, I adjusted the multiline to only match on the first 8 bits of each 16 bit character:

receivers:
  filelog/sap:
    include: [ auditlog.txt ]
    encoding: utf-16le
    multiline:
      line_start_pattern: '([23]).[A-Z].[A-Z].[A-Z0-9].(\d.){14}0.0.'
    preserve_trailing_whitespaces: true
    start_at: beginning

This configuration outputs 10 log records, each one containing a complete 200 character record.

Collector version

v0.122.0

Environment information

Environment

OS: MacOS 15.3.2

OpenTelemetry Collector configuration

receivers:
  filelog/sap:
    include: [ auditlog.txt ]
    encoding: utf-16le
    multiline:
      line_start_pattern: '([23])[A-Z][A-Z][A-Z0-9]\d{14}00'
    preserve_trailing_whitespaces: true
    start_at: beginning
exporters:
  file/debug:
    path: debug.json
service:
  pipelines:
    logs:
      receivers:
        - filelog/sap
      exporters:
        - file/debug

Log output

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski @andrzej-stencel

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

(Triage): removing needs-triage as the issue is described clearly, together with detailed instructions for reproduction. Adding waiting-for-code-owners

[djaglowski]

I agree this is a problem. I was able to reproduce it in a test case:

func TestUTF16LEMultiline(t *testing.T) {
	t.Parallel()

	tempDir := t.TempDir()
	cfg := NewConfig().includeDir(tempDir)
	cfg.StartAt = "beginning"
	cfg.Encoding = "utf-16le"
	cfg.SplitConfig.LineStartPattern = "([23])[A-Z][A-Z][A-Z0-9]\\d{14}00"
	operator, sink := testManager(t, cfg)

	// Create a file with UTF-16LE encoded content
	temp := filetest.OpenTemp(t, tempDir)

	// These should be written directly to the file as UTF-16LE and without delimiter
	testLogs := [][]byte{
		[]byte("2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        "),
		[]byte("2AUK20250227000000002316500018D110.102.8BATCH_ALRI                      SAPMSSY1                                0501Z91_VALR_IF&&Z91_VAL_PLSTATUS                                   10.122.81.29        "),
	}

	// convert to utf-16le
	var content []byte
	for _, testLog := range testLogs {
		for _, b := range testLog {
			content = append(content, byte(b), 0)
		}
	}

	_, err := temp.Write(content)
	require.NoError(t, err)

	require.NoError(t, operator.Start(testutil.NewUnscopedMockPersister()))
	defer func() {
		require.NoError(t, operator.Stop())
	}()

	// The pattern should match and we should get the full line
	sink.ExpectTokens(t, testLogs...)
}