[kafka_actions] Add plugin architecture for formats and compression codecs

[piochelepiotr]

What does this PR do?

Adds a pluggable interface for message format handlers and app-level payload compression codecs to the kafka_actions check, discovered via Python entry-point groups:

• datadog_kafka_actions.formats — FormatHandler subclasses (one per message format)
• datadog_kafka_actions.compressions — CompressionCodec subclasses (one per compression algorithm)

Built-in formats (json, string, raw, bson, avro, protobuf) are now first-class plugins registered by this wheel via the same entry points — no special-cased dispatch in MessageDeserializer.

Two new config keys, value_compression and key_compression, decompress payloads before format dispatch. No compression codecs ship in core; install a plugin wheel to add them. The companion plugin wheel datadog-kafka-deserializers (separate PR in integrations-extras) provides msgpack plus gzip / zlib / snappy / lz4 / lz4_dd_hdr / zstd.

Motivation

A growing set of Kafka producers across dd-go and dd-source ship payloads in formats and codecs that the check doesn't currently support — most notably MessagePack and a handful of app-level compressions (gzip, zstd, and a custom lz4 framing used by xray-converter). Hard-coding more handlers into core makes each new format a kafka_actions change with a full release cycle, and pulls dependency churn (msgpack, lz4, zstandard, python-snappy, ...) into a check that doesn't otherwise need them.

The plugin model lets us:

1. Keep the core check small and dependency-light.
2. Move format/codec specifics into separate wheels owned by the teams that produce those payload formats.
3. Allow customers to ship their own deserializers (e.g. proprietary framing) without forking core.

Additional Notes

• MessageDeserializer's public surface is unchanged. Existing callers and tests continue to work.
• _bootstrap_format_handlers() direct-registers the built-in handlers on import so the check works in source-mode test runs without pip install.
• Compression runs before format dispatch and after skip_bytes, so existing skip_bytes semantics are preserved.
• The changelog entry is currently named plugin-architecture.added; will rename to <PR#>.added once this PR has a number.

Review checklist (to be filled by the PR author)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see docs)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81ca3f7783

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Add compression keys to action configuration schema

Wire-up in read_messages now reads value_compression/key_compression, but this commit does not add those fields to the integration configuration contract (kafka_actions/assets/configuration/spec.yaml) or generated model (kafka_actions/datadog_checks/kafka_actions/config_models/instance.py). In Remote Configuration flows, unsupported keys are rejected or dropped before reaching the check, so the new compression feature is effectively not configurable in production even though deserialization now expects it.

Useful? React with 👍 / 👎.

[codecov]

Codecov Report

❌ Patch coverage is 64.78873% with 125 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.79%. Comparing base (e529a0b) to head (371cd79).

Additional details and impacted files

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[datadog-prod-us1-6]

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 64.79%
• Overall Coverage: 76.70% (-10.54%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 371cd79 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

All 20 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and Codecov settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run