Remove message reading from kafka_consumer

[piochelepiotr]

What does this PR do?

Removes the Data Streams live messages feature from the kafka_consumer integration. This includes:

• The data_streams_live_message flow in the check and all helpers (_messages_have_been_retrieved, _mark_messages_retrieved, resolve_start_offsets).
• All message deserialization helpers (deserialize_message, build_schema, build_avro_schema, build_protobuf_schema, and the protobuf/avro/json decoders).
• The live_messages_configs instance option and its validation in KafkaConfig.
• Client methods that exist solely to read and decode messages (start_collecting_messages, get_next_message, delete_consumer_group).
• The fastavro and protobuf runtime dependencies.
• The associated unit and integration tests.

Motivation

Message reading is now owned by the kafka_actions integration, so the duplicate implementation in kafka_consumer is no longer needed.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

🤖 Generated with Claude Code

[github-actions]

⚠️ Major version bump
The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

[datadog-datadog-prod-us1]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

PR | test / upload-coverage     

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration.

The action 'datadog/install-datadog-ci-github-action@v1' is not allowed due to enterprise policy restrictions.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e4cd0ac | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and Codecov settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run