Filebeat input v2 API

[urso]

This meta issue tracks all PRs, issues and tasks for a new input API in Filebeat. The new API also introduces a generic key-value store as registry, for reuse by other inputs. Development targets the feature branch feature-new-registry-file. Related issues/PRs will be labeled the Project:Filebeat-Input-v2.

Initial Phase

• Generic Registry as key-value store:
  • Introduce statestore package Init package libbeat/statestore #19117
  • Registry migration: Filebeat new registry #12908
  • Filebeat registrar using new registry type: Filebeat new registry #12908
• State Store
• Define and implementation API: Statestore prototype #14079
• Testing
• Input API
  • Define API for inputs: [WIP] Filebeat input v2 #14229
  • Input Loading support for new style inputs:
  • Integrate with config reloading and autodiscovery
  • Integrate with Agent/Fleet status reporting
  • Testing
• Registry garbage collection support

Next

• Convert existing inputs to the new API
  • kafka [Filebeat] kafka v2 using parsers #27335
  • redis
  • syslog
  • udp
  • TCP
  • aws s3 Migrate S3 Input to Filebeat Input V2 #20005
  • aws cloudwatch Support running multiple log groups in cloudwatch input #29695
  • azure eventhub
  • gcp pubsub

The inputs log, container, docker and stdin are not going to be moved. Instead these should be replaced by filestream or dropped altogether.
- [ ] log/container/docker/stdin
- [ ] factor out file watcher emiting events
- [ ] update to new API
- [ ] split up into independent inputs

• Add existing Beats as new inputs
  • Journaldbeat
  • Winlogbeat

We decided to keep log input around so users are not forced to update.
- [ ] Remove old input API and supporting architecture
- [ ] remove filebeat/channel package
- [ ] remove state support from input/file package
- [ ] remove registrar package

[ph]

Thanks @urso for pushing this effort!

[jlind23]

Needs to be closed by @kvch

[kaiyan-sheng]

I'm adding aws-cloudwatch, gcppubsub and azureeventhub input into the Convert existing inputs to the new API list.

[jlind23]

@rameshelastic it seems that the missing input are on your plate, am I correct?

[rameshelastic]

@jlind23 ..hmm..I'm not sure.

• @masci ..could you help clarify?

[jlind23]

@rameshelastic you are right, not all of them.

• redis must belong to you.
• syslog must belong to @elastic/security-external-integrations

All the three below belongs to @elastic/obs-cloud-monitoring

• aws cloudwatch
• azure eventhub
• gcp pubsub

[kvch]

We are moving to the new input system, so I think this can be closed. @jlind23 WDYT?