Skip to content

Comments

cluster/partition: Reword description for kafka_request_bytes_total.#25499

Merged
BenPope merged 1 commit intoredpanda-data:devfrom
BenPope:core-9575/reword_kafka_request_bytes
Mar 25, 2025
Merged

cluster/partition: Reword description for kafka_request_bytes_total.#25499
BenPope merged 1 commit intoredpanda-data:devfrom
BenPope:core-9575/reword_kafka_request_bytes

Conversation

@BenPope
Copy link
Member

@BenPope BenPope commented Mar 21, 2025

This metric probably has the wrong name, the equivalent internal metric is cluster_partition_request_bytes_total, but it's been around a long time now.

Reword the description to make it clearer.

The description is the same for both produce and consume because only the first one is returned on the Prometheus endpoint.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.1.x
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@BenPope BenPope requested review from a team March 21, 2025 16:33
@BenPope BenPope self-assigned this Mar 21, 2025
@BenPope BenPope requested review from pgellert and removed request for a team March 21, 2025 16:33
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 21, 2025

CI test results

test results on build#63524
test_id test_kind job_url test_status passed
partition_balancer_simulator_test_rpunit.partition_balancer_simulator_test_rpunit unit https://buildkite.com/redpanda/redpanda/builds/63524#0195b98f-2405-4646-8131-713e8657f897 FLAKY 1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade ducktape https://buildkite.com/redpanda/redpanda/builds/63524#0195b9ea-ae44-41ac-9481-40738fb7146f FLAKY 1/2
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_conflicting_names ducktape https://buildkite.com/redpanda/redpanda/builds/63524#0195b9e8-46c6-423c-9493-a7d5dec1c772 FLAKY 1/2
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=False.params=.cancellation.dir.in.stage.preparing.use_alias.True ducktape https://buildkite.com/redpanda/redpanda/builds/63524#0195b9e8-46c6-423c-9493-a7d5dec1c772 FLAKY 1/2
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=.cancellation.dir.in.stage.preparing.use_alias.False ducktape https://buildkite.com/redpanda/redpanda/builds/63524#0195b9e8-46c7-4f4b-b764-d16ecfae24c9 FLAKY 1/3
rptest.tests.datalake.compaction_test.CompactionTest.test_compaction.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/63524#0195b9ea-ae45-4557-8669-99065df5ebe9 FLAKY 1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic ducktape https://buildkite.com/redpanda/redpanda/builds/63524#0195b9ea-ae43-4186-b6a5-50a62b958f01 FLAKY 1/2
test results on build#63591
test_id test_kind job_url test_status passed
rptest.tests.availability_test.AvailabilityTests.test_recovery_after_catastrophic_failure ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-441b-880e-9747f48e5abb FAIL 0/1
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-441b-880e-9747f48e5abb FLAKY 1/2
rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_migrated_topic_data_integrity.transfer_leadership=True.params=.cancellation.dir.in.stage.preparing.use_alias.True ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-48ad-8b41-2400f54a6c67 FLAKY 1/2
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a708-491e-ad7a-6ef5a3f22fac FLAKY 1/14
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-48ad-8b41-2400f54a6c67 FLAKY 1/5
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c96b-c222-4fa5-8908-7b64cca67819 FLAKY 1/6
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-48ad-8b41-2400f54a6c67 FLAKY 1/3
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a708-491e-ad7a-6ef5a3f22fac FLAKY 1/2
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-441b-880e-9747f48e5abb FLAKY 1/12
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c96b-c222-4fa5-8908-7b64cca67819 FLAKY 1/3
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a708-491e-ad7a-6ef5a3f22fac FLAKY 1/9
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-48ad-8b41-2400f54a6c67 FLAKY 1/10
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c96b-c222-4fa5-8908-7b64cca67819 FLAKY 1/2
rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-441b-880e-9747f48e5abb FLAKY 1/3
rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=False ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c952-a709-48ad-8b41-2400f54a6c67 FLAKY 1/2
rptest.tests.maintenance_test.MaintenanceTest.test_maintenance_sticky.use_rpk=True ducktape https://buildkite.com/redpanda/redpanda/builds/63591#0195c96b-c222-4fa5-8908-7b64cca67819 FLAKY 1/2

pgellert
pgellert previously approved these changes Mar 24, 2025
Copy link
Contributor

@pgellert pgellert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thanks!

Comment on lines 323 to 325
sm::description(
"Total bytes read from or written to partitions for a topic "
"(fetched bytes may not all be returned to the client)."),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (ignore unless you end up changing the wording based on docs feedback): you could extract the ss::description to a variable to show that they are shared+identical.

@@ -309,7 +309,9 @@ void replicated_partition_probe::setup_public_metrics(const model::ntp& ntp) {
sm::make_total_bytes(
"request_bytes_total",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This metric probably has the wrong name, the equivalent internal metric is cluster_partition_request_bytes_total.

I'm not following this part of the commit message. The change of reusing the description makes sense, but I am not following why the metric would have the wrong name. It seems reasonable to me, since it is the procuce+fetch total (with a label to allow splitting out either of them). Am I missing something here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This metric probably has the wrong name, the equivalent internal metric is cluster_partition_request_bytes_total.

I'm not following this part of the commit message. The change of reusing the description makes sense, but I am not following why the metric would have the wrong name. It seems reasonable to me, since it is the procuce+fetch total (with a label to allow splitting out either of them). Am I missing something here?

It's bytes written to and read from the partition, not necessarily what's returned to the user. It's confusing because it has kafka prefix, but actually measures disk activity. Kafka traffic is measured with redpanda_rpc_sent_bytes{redpanda_server="kafka"} and redpanda_rpc_received_bytes{redpanda_server="kafka"}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thank you

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a slight rewording of the first sentence? Also maybe it would be clearer to lose the brackets and add another sentence?

Total number of bytes read from or written to the partitions of a topic. The total may include fetched bytes that are not returned to the client.

asimms41
asimms41 previously approved these changes Mar 24, 2025
Copy link
Contributor

@asimms41 asimms41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of suggestions.

This metric probably has the wrong name, the equivalent internal
metric is cluster_partition_request_bytes_total, the metric is
measuring disk activity, not kafka network traffic.

Reword the description to make it clearer.

The description is the same for both produce and consume because
only the first one is returned on the prometheus endpoint.

Signed-off-by: Ben Pope <ben@redpanda.com>
@BenPope BenPope dismissed stale reviews from asimms41 and pgellert via 97ecb4b March 24, 2025 16:48
@BenPope BenPope force-pushed the core-9575/reword_kafka_request_bytes branch from ce05fb4 to 97ecb4b Compare March 24, 2025 16:48
@BenPope BenPope requested review from asimms41 and pgellert March 24, 2025 16:50
@BenPope BenPope enabled auto-merge March 24, 2025 16:53
@vbotbuildovich
Copy link
Collaborator

Retry command for Build#63591

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/availability_test.py::AvailabilityTests.test_recovery_after_catastrophic_failure

@BenPope BenPope merged commit 1416c04 into redpanda-data:dev Mar 25, 2025
19 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v25.1.x

@vbotbuildovich
Copy link
Collaborator

/backport v24.3.x

@vbotbuildovich
Copy link
Collaborator

/backport v24.2.x

@vbotbuildovich
Copy link
Collaborator

/backport v24.1.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants