[Filebeat] CEL input panic without 'redact' config

[andrewkroh]

The CEL input produces a panic when the redact option is not specified.

• Version: main (a86e699)
• Code:

  beats/x-pack/filebeat/input/cel/input.go

  Line 229 in 7ed34ff

  log.Debugw("request state", logp.Namespace("cel"), "state", redactor{state: state, mask: cfg.Redact.Fields, delete: cfg.Redact.Delete})

{"log.level":"error","@timestamp":"2023-08-21T19:09:33.023-0400","log.logger":"input.cel","log.origin":{"file.name":"compat/compat.go","file.line":132},"message":"Input 'cel' failed with: input.go:130: input config-123-watcher failed (id=config-123-watcher)\n\tinput panic with: runtime error: invalid memory address or nil pointer dereference\n\tgoroutine 111 [running]:\n\truntime/debug.Stack()\n\t\truntime/debug/stack.go:24 +0x64\n\tgithub.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).runSource.func1()\n\t\tgithub.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:143 +0x48\n\tpanic({0x1065799e0, 0x1090b5fc0})\n\t\truntime/panic.go:884 +0x204\n\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1()\n\t\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:229 +0x29c\n\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel.periodically({0x106c82f40?, 0x140017699a0}, 0x38?, 0x140017ab7c0)\n\t\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:480 +0x38\n\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run({_}, {0x14001778a40, {0x140001594a0, 0x4c}, {{0x1057728d1, 0x8}, {0x1057728d1, 0x8}, {0x10576c0d7, 0x6}, ...}, ...}, ...)\n\t\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:204 +0x5d4\n\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.Run({_}, {0x14001778a40, {0x140001594a0, 0x4c}, {{0x1057728d1, 0x8}, {0x1057728d1, 0x8}, {0x10576c0d7, 0x6}, ...}, ...}, ...)\n\t\tgithub.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:108 +0xd4\n\tgithub.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).runSource(_, {0x14001778a40, {0x140001594a0, 0x4c}, {{0x1057728d1, 0x8}, {0x1057728d1, 0x8}, {0x10576c0d7, 0x6}, ...}, ...}, ...)\n\t\tgithub.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:168 +0x2f8\n\tgithub.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).Run.func1()\n\t\tgithub.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:122 +0x14c\n\tgithub.com/elastic/go-concert/unison.(*MultiErrGroup).Go.func1()\n\t\tgithub.com/elastic/go-concert@v0.2.0/unison/multierrgroup.go:42 +0x68\n\tcreated by github.com/elastic/go-concert/unison.(*MultiErrGroup).Go\n\t\tgithub.com/elastic/go-concert@v0.2.0/unison/multierrgroup.go:40 +0x8c","service.name":"filebeat","id":"config-123-watcher","ecs.version":"1.6.0"}

Input crashed with: input panic with: runtime error: invalid memory address or nil pointer dereference
goroutine 111 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x64
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).runSource.func1()
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:143 +0x48
panic({0x1065799e0, 0x1090b5fc0})
	runtime/panic.go:884 +0x204
github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run.func1()
	github.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:229 +0x29c
github.com/elastic/beats/v7/x-pack/filebeat/input/cel.periodically({0x106c82f40?, 0x140017699a0}, 0x38?, 0x140017ab7c0)
	github.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:480 +0x38
github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.run({_}, {0x14001778a40, {0x140001594a0, 0x4c}, {{0x1057728d1, 0x8}, {0x1057728d1, 0x8}, {0x10576c0d7, 0x6}, ...}, ...}, ...)
	github.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:204 +0x5d4
github.com/elastic/beats/v7/x-pack/filebeat/input/cel.input.Run({_}, {0x14001778a40, {0x140001594a0, 0x4c}, {{0x1057728d1, 0x8}, {0x1057728d1, 0x8}, {0x10576c0d7, 0x6}, ...}, ...}, ...)
	github.com/elastic/beats/v7/x-pack/filebeat/input/cel/input.go:108 +0xd4
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).runSource(_, {0x14001778a40, {0x140001594a0, 0x4c}, {{0x1057728d1, 0x8}, {0x1057728d1, 0x8}, {0x10576c0d7, 0x6}, ...}, ...}, ...)
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:168 +0x2f8
github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.(*managedInput).Run.func1()
	github.com/elastic/beats/v7/filebeat/input/v2/input-cursor/input.go:122 +0x14c
github.com/elastic/go-concert/unison.(*MultiErrGroup).Go.func1()
	github.com/elastic/go-concert@v0.2.0/unison/multierrgroup.go:42 +0x68
created by github.com/elastic/go-concert/unison.(*MultiErrGroup).Go
	github.com/elastic/go-concert@v0.2.0/unison/multierrgroup.go:40 +0x8c

---

filebeat.inputs:
  - type: cel
    id: config-123-watcher
    interval: 1m
    resource:
      url: file:///Users/jdoe/code/beats/x-pack/filebeat/filebeat.yml
    program: |
      file(state.url).as(content, content.sha256().hex().as(hash, {
          'url': state.url,
          'cursor': {
            'sha256': hash,
          },
          'want_more': false,
          'events': has(state.cursor) && has(state.cursor.sha256) && state.cursor.sha256 == hash ? [] : [{
            'file': {
              'path': state.url.trim_prefix('file://'),
              'hash': {
                'sha256': hash,
              }
            },
            'related': {
              'hash': [hash],
            },
            'message': string(content),
          }],
      }))
    publisher_pipeline.disable_host: true

output.console.pretty: true

The docs do advise using this, but it is not mandatory.

- type: cel
  redact:
    fields: ~

A secondary thing I noticed is that the same message is logged consecutively in the log.

{"log.level":"warn","@timestamp":"2023-08-21T19:09:33.016-0400","log.logger":"input.cel","log.origin":{"file.name":"cel/config.go","file.line":70},"message":"missing recommended 'redact' configuration: see documentation for details: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_redact","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-08-21T19:09:33.020-0400","log.logger":"input.cel","log.origin":{"file.name":"cel/config.go","file.line":70},"message":"missing recommended 'redact' configuration: see documentation for details: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_redact","service.name":"filebeat","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I'm not sure what we can do about the consecutive logging while keeping the log message in the config validation. We could move it into the run code, which would change it to being non-consecutive, but still naggy.

[efd6]

@andrewkroh This is closed as fixed for the panic. If you still feel that the repetition in the log needs to be addressed, please re-open.