Skip to content

DNM: log segment locking #26256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
andrwng wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

DNM: log segment locking #26256

andrwng wants to merge 2 commits into from

Conversation

@andrwng
Copy link
Contributor

@andrwng andrwng commented May 27, 2025

Debugging

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.1.x
  • v24.3.x
  • v24.2.x

Release Notes

  • none

andrwng added 2 commits May 27, 2025 13:20
@andrwng
ssx: add a logging_rwlock
c6a6a31 
This commit adds a wrapper around ss::rwlock that logs when lock
acquisition takes longer than a default of 10s. It is meant to be a
drop-in replacement, though this commit only adds just enough of the
ss::rwlock API to replace current usages in the context of segment
locks.

The following logging will occur:
- trace message is logged when lock attempt is started
- trace message is logged when lock is acquired
- trace message is logged when lock is dropped
- error message is logged when a lock takes longer than 10s to acquire

The ss::rwlock::holder abstraction is also copied with a
ssx::logging_rwlock::holder, which includes a holder ID that is also
logged, allowing for easier correlation between log messages.

This will be useful in debugging deadlocks among the storage layer.
@andrwng
storage: replace rwlock with logging_rwlock
6a0016c 
Acquisition of segment locks span the raft, storage, and archival
subsystems, making it very difficult to determine what fiber is holding
what lock or whether slowness is caused by the waiting on a lock.

To remedy this, this replaces the segment locks with a logging_rwlock.
@andrwng andrwng force-pushed the logging-rwlock-segment-lock branch from 41026db to 6a0016c Compare May 27, 2025 20:20
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented May 28, 2025 •
edited
Loading

Retry command for Build#66457

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/blocked_catalog_test.py::DatalakeBlockedCatalogTest.test_block_cloud_retention_before_translation@{"cloud_storage_type":1,"with_spillover":true}
tests/rptest/tests/datalake/blocked_catalog_test.py::DatalakeBlockedCatalogTest.test_block_cloud_retention_before_translation@{"cloud_storage_type":1,"with_spillover":false}

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented May 28, 2025 •
edited
Loading

CI test results

test results on build#66457
test_class test_method test_arguments test_kind job_url test_status passed reason
AccessControlListTestUpgrade test_upgrade_sasl ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55a-4d92-ba3f-ba02a0c472c9 FLAKY 20/21 upstream reliability is '97.60191846522783'. current run reliability is '95.23809523809523'. drift is 2.36382 and the allowed drift is set to 50. The test should PASS
DataTransformsRpcUpgradeTest test_upgrade_one_node ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55a-4d92-ba3f-ba02a0c472c9 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DataTransformsTest test_identity {"transactional": true, "wait_running": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55a-4d92-ba3f-ba02a0c472c9 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713c1-855d-46eb-bed0-6f65848da219 FLAKY 14/21 upstream reliability is '100.0'. current run reliability is '66.66666666666666'. drift is 33.33333 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55a-4d92-ba3f-ba02a0c472c9 FLAKY 7/21 upstream reliability is '100.0'. current run reliability is '33.33333333333333'. drift is 66.66667 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713c1-855d-47ef-b71e-ccc10b2195a1 FLAKY 6/21 upstream reliability is '100.0'. current run reliability is '28.57142857142857'. drift is 71.42857 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55b-44bd-be4e-3c08a6a601a2 FLAKY 4/21 upstream reliability is '100.0'. current run reliability is '19.047619047619047'. drift is 80.95238 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713c1-855c-435b-9669-b65c7cd0bb2e FLAKY 15/21 upstream reliability is '100.0'. current run reliability is '71.42857142857143'. drift is 28.57143 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b559-46d0-8b69-70ea62650a1c FLAKY 14/21 upstream reliability is '100.0'. current run reliability is '66.66666666666666'. drift is 33.33333 and the allowed drift is set to 50. The test should PASS
LeadershipPinningTest test_leadership_pinning_sanctions ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55a-4d92-ba3f-ba02a0c472c9 FLAKY 20/21 upstream reliability is '98.09160305343512'. current run reliability is '95.23809523809523'. drift is 2.85351 and the allowed drift is set to 50. The test should PASS
PartitionBalancerTest test_maintenance_mode {"kill_same_node": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713c1-855d-47ef-b71e-ccc10b2195a1 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
PartitionReassignmentsTest test_add_partitions_with_inprogress_reassignments ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713c1-855d-47ef-b71e-ccc10b2195a1 FLAKY 19/21 upstream reliability is '89.47368421052632'. current run reliability is '90.47619047619048'. drift is -1.00251 and the allowed drift is set to 50. The test should PASS
PartitionReassignmentsTest test_add_partitions_with_inprogress_reassignments ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b559-46d0-8b69-70ea62650a1c FLAKY 19/21 upstream reliability is '89.48905109489051'. current run reliability is '90.47619047619048'. drift is -0.98714 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713c1-855d-47ef-b71e-ccc10b2195a1 FLAKY 19/21 upstream reliability is '100.0'. current run reliability is '90.47619047619048'. drift is 9.52381 and the allowed drift is set to 50. The test should PASS
RedpandaOIDCTest test_admin_revoke ducktape https://buildkite.com/redpanda/redpanda/builds/66457#019713d1-b55a-4d92-ba3f-ba02a0c472c9 FLAKY 20/21 upstream reliability is '99.24242424242425'. current run reliability is '95.23809523809523'. drift is 4.00433 and the allowed drift is set to 50. The test should PASS
test results on build#66532
test_class test_method test_arguments test_kind job_url test_status passed reason
AccessControlListTestUpgrade test_upgrade_sasl ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b9-485e-86ea-e63bc84f891b FLAKY 20/21 upstream reliability is '97.73869346733667'. current run reliability is '95.23809523809523'. drift is 2.5006 and the allowed drift is set to 50. The test should PASS
AccessControlListTestUpgrade test_upgrade_sasl ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4674-abd9-94020057d24e FLAKY 20/21 upstream reliability is '97.73869346733667'. current run reliability is '95.23809523809523'. drift is 2.5006 and the allowed drift is set to 50. The test should PASS
AccessControlListTestUpgrade test_upgrade_sasl ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4dc1-a85a-bafb144f7886 FLAKY 20/21 upstream reliability is '97.73869346733667'. current run reliability is '95.23809523809523'. drift is 2.5006 and the allowed drift is set to 50. The test should PASS
AccessControlListTestUpgrade test_upgrade_sasl ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4edb-9343-05dd09b68d21 FLAKY 20/21 upstream reliability is '97.73869346733667'. current run reliability is '95.23809523809523'. drift is 2.5006 and the allowed drift is set to 50. The test should PASS
CloudRetentionTest test_cloud_retention {"cloud_storage_type": 2, "max_consume_rate_mb": null} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-4723-bc48-ddd005ef828e FLAKY 11/14
CloudStorageTimingStressTest test_cloud_storage_with_partition_moves {"cleanup_policy": "compact,delete"} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c9-4bd2-a9b3-ad3a8ba586eb FLAKY 20/21 upstream reliability is '96.91119691119691'. current run reliability is '95.23809523809523'. drift is 1.6731 and the allowed drift is set to 50. The test should PASS
ConsumerOffsetsRecoveryToolTest test_consumer_offsets_partition_count_change ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4674-abd9-94020057d24e FLAKY 20/21 upstream reliability is '98.14814814814815'. current run reliability is '95.23809523809523'. drift is 2.91005 and the allowed drift is set to 50. The test should PASS
ConsumerOffsetsRecoveryToolTest test_consumer_offsets_partition_count_change ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4edb-9343-05dd09b68d21 FLAKY 20/21 upstream reliability is '98.15384615384616'. current run reliability is '95.23809523809523'. drift is 2.91575 and the allowed drift is set to 50. The test should PASS
ControllerLogLimitPartitionBalancerTests test_partition_balancer_with_limits ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b7-4d0e-9f1b-29224234c75c FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DataMigrationsApiTest test_migrated_topic_data_integrity {"params": {"cancellation": {"dir": "in", "stage": "executed"}, "use_alias": true}, "transfer_leadership": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bc-44a2-840a-7d4ad7e6d10a FLAKY 20/21 upstream reliability is '99.57983193277312'. current run reliability is '95.23809523809523'. drift is 4.34174 and the allowed drift is set to 50. The test should PASS
DataMigrationsApiTest test_migrated_topic_data_integrity {"params": {"cancellation": {"dir": "in", "stage": "executing"}, "use_alias": false}, "transfer_leadership": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4f7d-88cf-5fbd6ac6a801 FLAKY 20/21 upstream reliability is '98.97610921501706'. current run reliability is '95.23809523809523'. drift is 3.73801 and the allowed drift is set to 50. The test should PASS
DataMigrationsApiTest test_migrated_topic_data_integrity {"params": {"cancellation": {"dir": "out", "stage": "executed"}, "use_alias": false}, "transfer_leadership": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bf-4cf6-b5a4-92ce41d43e4e FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DataMigrationsApiTest test_migrated_topic_data_integrity {"params": {"cancellation": {"dir": "out", "stage": "prepared"}, "use_alias": false}, "transfer_leadership": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4dc1-a85a-bafb144f7886 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DataTransformsRpcUpgradeTest test_upgrade_one_node ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4674-abd9-94020057d24e FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DataTransformsRpcUpgradeTest test_upgrade_one_node ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4edb-9343-05dd09b68d21 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_after_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b5-4afa-aa01-f1e6a328c860 FLAKY 15/21 upstream reliability is '100.0'. current run reliability is '71.42857142857143'. drift is 28.57143 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_after_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b6-48e0-8afd-37a2a9e838b4 FLAKY 16/21 upstream reliability is '100.0'. current run reliability is '76.19047619047619'. drift is 23.80952 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_after_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b6-4b1b-89ba-4e8203b7cadb FLAKY 12/21 upstream reliability is '100.0'. current run reliability is '57.14285714285714'. drift is 42.85714 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b8-4247-b682-80955b5ea2dc FLAKY 11/21 upstream reliability is '100.0'. current run reliability is '52.38095238095239'. drift is 47.61905 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b9-47f5-a06a-6d36a4f6ddb3 FLAKY 10/21 upstream reliability is '100.0'. current run reliability is '47.61904761904761'. drift is 52.38095 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4674-abd9-94020057d24e FLAKY 8/21 upstream reliability is '100.0'. current run reliability is '38.095238095238095'. drift is 61.90476 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ba-4a3e-9ef3-385c597459db FLAKY 10/21 upstream reliability is '100.0'. current run reliability is '47.61904761904761'. drift is 52.38095 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c9-429c-abb6-bab09073194b FLAKY 9/21 upstream reliability is '100.0'. current run reliability is '42.857142857142854'. drift is 57.14286 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c9-4fca-9fa9-ee1309ca5b9f FLAKY 12/21 upstream reliability is '100.0'. current run reliability is '57.14285714285714'. drift is 42.85714 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61ca-48d3-9df0-a128a53b9e0d FLAKY 14/21 upstream reliability is '100.0'. current run reliability is '66.66666666666666'. drift is 33.33333 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bb-432c-ae12-4fb0fcc01eb6 FLAKY 8/21 upstream reliability is '100.0'. current run reliability is '38.095238095238095'. drift is 61.90476 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bb-45f3-b6f0-8990ec0070cb FLAKY 7/21 upstream reliability is '100.0'. current run reliability is '33.33333333333333'. drift is 66.66667 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bc-4014-a814-aa790cccd327 FLAKY 6/21 upstream reliability is '100.0'. current run reliability is '28.57142857142857'. drift is 71.42857 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bc-4150-b62e-8ecb3f04048e FLAKY 8/21 upstream reliability is '100.0'. current run reliability is '38.095238095238095'. drift is 61.90476 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bc-44a2-840a-7d4ad7e6d10a FLAKY 6/21 upstream reliability is '100.0'. current run reliability is '28.57142857142857'. drift is 71.42857 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bc-4b65-8552-5616af0ecada FLAKY 4/21 upstream reliability is '100.0'. current run reliability is '19.047619047619047'. drift is 80.95238 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bd-49c7-87cb-1730332d6502 FLAKY 3/21 upstream reliability is '100.0'. current run reliability is '14.285714285714285'. drift is 85.71429 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bd-4c76-886b-5d629714a7fe FLAKY 6/21 upstream reliability is '100.0'. current run reliability is '28.57142857142857'. drift is 71.42857 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cb-4270-b0a4-0cd3ad71dbc2 FLAKY 4/21 upstream reliability is '100.0'. current run reliability is '19.047619047619047'. drift is 80.95238 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cb-4bf0-bc58-04233091cd30 FLAKY 1/21 upstream reliability is '100.0'. current run reliability is '4.761904761904762'. drift is 95.2381 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-43a7-8da6-6f5263c65b9f FLAKY 5/21 upstream reliability is '100.0'. current run reliability is '23.809523809523807'. drift is 76.19048 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-47ee-bc2a-0394df5ddaa9 FLAKY 6/21 upstream reliability is '100.0'. current run reliability is '28.57142857142857'. drift is 71.42857 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-4a56-869a-e9413a1b654e FLAKY 3/21 upstream reliability is '100.0'. current run reliability is '14.285714285714285'. drift is 85.71429 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-4c18-9d45-dcf7e19eb759 FLAKY 2/21 upstream reliability is '100.0'. current run reliability is '9.523809523809524'. drift is 90.47619 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cd-402f-ab21-c511650e1bc1 FLAKY 3/21 upstream reliability is '100.0'. current run reliability is '14.285714285714285'. drift is 85.71429 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_block_cloud_retention_before_translation {"cloud_storage_type": 1, "with_spillover": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cd-4bdc-844d-acba8e6ecb4d FLAKY 9/21 upstream reliability is '100.0'. current run reliability is '42.857142857142854'. drift is 57.14286 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b6-4b1b-89ba-4e8203b7cadb FLAKY 14/21 upstream reliability is '100.0'. current run reliability is '66.66666666666666'. drift is 33.33333 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b7-4393-8f8a-2e668dbc743d FLAKY 10/21 upstream reliability is '100.0'. current run reliability is '47.61904761904761'. drift is 52.38095 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b7-4d0e-9f1b-29224234c75c FLAKY 9/21 upstream reliability is '100.0'. current run reliability is '42.857142857142854'. drift is 57.14286 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c5-4ef2-a292-6d44f1190fc9 FLAKY 11/21 upstream reliability is '100.0'. current run reliability is '52.38095238095239'. drift is 47.61905 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-433e-aa61-12b81f1124fa FLAKY 12/21 upstream reliability is '100.0'. current run reliability is '57.14285714285714'. drift is 42.85714 and the allowed drift is set to 50. The test should PASS
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-475c-ae45-eddb2a725f85 FLAKY 10/21 upstream reliability is '100.0'. current run reliability is '47.61904761904761'. drift is 52.38095 and the allowed drift is set to 50. The test should FAIL
DatalakeBlockedCatalogTest test_trim_local_before_first_translation {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c7-4957-8d06-c39e0ed3a222 FLAKY 12/21 upstream reliability is '100.0'. current run reliability is '57.14285714285714'. drift is 42.85714 and the allowed drift is set to 50. The test should PASS
DatalakeDelayedEnablementTest test_enabling_iceberg_in_existing_cluster {"catalog_type": "rest_jdbc", "cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c5-4ef2-a292-6d44f1190fc9 FLAKY 20/21 upstream reliability is '94.67213114754098'. current run reliability is '95.23809523809523'. drift is -0.56596 and the allowed drift is set to 50. The test should PASS
RestCatalogConnectionTest test_redpanda_connection_to_rest_catalog {"cloud_storage_type": 1} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cd-402f-ab21-c511650e1bc1 FLAKY 20/21 upstream reliability is '99.24242424242425'. current run reliability is '95.23809523809523'. drift is 4.00433 and the allowed drift is set to 50. The test should PASS
MultiRestartTest test_recovery_after_multiple_restarts {"cloud_storage_type": 2} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c9-4eff-ad35-09e80356626e FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
NodePoolMigrationTest test_migrating_redpanda_nodes_to_new_pool {"balancing_mode": "node_add", "cleanup_policy": "compact", "test_mode": "no_tiered_storage"} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c3-4671-a670-79c3516b4b83 FAIL 0/1
NodePoolMigrationTest test_migrating_redpanda_nodes_to_new_pool {"balancing_mode": "node_add", "cleanup_policy": "compact,delete", "test_mode": "tiered_storage"} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cb-4bf0-bc58-04233091cd30 FLAKY 20/21 upstream reliability is '99.64028776978418'. current run reliability is '95.23809523809523'. drift is 4.40219 and the allowed drift is set to 50. The test should PASS
NodePoolMigrationTest test_migrating_redpanda_nodes_to_new_pool {"balancing_mode": "off", "cleanup_policy": "compact,delete", "test_mode": "tiered_storage"} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-49d8-a31d-3ea042eb2ac3 FLAKY 18/21 upstream reliability is '98.62068965517241'. current run reliability is '85.71428571428571'. drift is 12.9064 and the allowed drift is set to 50. The test should PASS
NodePostRestartProbeTest post_restart_probe_test ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-4c18-9d45-dcf7e19eb759 FLAKY 20/21 upstream reliability is '97.05882352941177'. current run reliability is '95.23809523809523'. drift is 1.82073 and the allowed drift is set to 50. The test should PASS
PartitionBalancerTest test_fuzz_admin_ops ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c9-4bd2-a9b3-ad3a8ba586eb FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
PartitionReassignmentsTest test_add_partitions_with_inprogress_reassignments ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-43a7-8da6-6f5263c65b9f FLAKY 17/21 upstream reliability is '89.50276243093923'. current run reliability is '80.95238095238095'. drift is 8.55038 and the allowed drift is set to 50. The test should PASS
PartitionReassignmentsTest test_add_partitions_with_inprogress_reassignments ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cd-4bdc-844d-acba8e6ecb4d FLAKY 19/21 upstream reliability is '89.48824343015215'. current run reliability is '90.47619047619048'. drift is -0.98795 and the allowed drift is set to 50. The test should PASS
RaftSlowFollowerTest test_single_slow_follower ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b7-42c6-9168-d0b76c1255f0 FLAKY 19/21 upstream reliability is '100.0'. current run reliability is '90.47619047619048'. drift is 9.52381 and the allowed drift is set to 50. The test should PASS
RaftSlowFollowerTest test_single_slow_follower ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b7-4d0e-9f1b-29224234c75c FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c2-45df-9b77-434a84f184cc FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c2-4f2a-97f6-81093d193760 FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c3-4671-a670-79c3516b4b83 FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c4-4ec9-8737-167a8368c90d FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c2-45df-9b77-434a84f184cc FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c2-4f2a-97f6-81093d193760 FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c3-4671-a670-79c3516b4b83 FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c4-4ec9-8737-167a8368c90d FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": true} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c4-4f47-b6eb-6673c5a26601 FAIL 0/2
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c4-4f47-b6eb-6673c5a26601 FAIL 0/2
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c5-47de-bb8d-73c9ec24b03f FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c5-4ef2-a292-6d44f1190fc9 FLAKY 17/21 upstream reliability is '100.0'. current run reliability is '80.95238095238095'. drift is 19.04762 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-422d-8c14-0b96b4b00f44 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-4723-bc48-ddd005ef828e FLAKY 1/5
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-475c-ae45-eddb2a725f85 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c7-4396-b2ae-a9240030828f FLAKY 16/18
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": false, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c7-4957-8d06-c39e0ed3a222 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-43a7-8da6-6f5263c65b9f FLAKY 15/21 upstream reliability is '100.0'. current run reliability is '71.42857142857143'. drift is 28.57143 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cd-402f-ab21-c511650e1bc1 FLAKY 14/21 upstream reliability is '100.0'. current run reliability is '66.66666666666666'. drift is 33.33333 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 1, "enable_failures": true, "mixed_versions": false, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c4-4ec9-8737-167a8368c90d FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": false, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c6-4723-bc48-ddd005ef828e FAIL 0/5
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cb-4270-b0a4-0cd3ad71dbc2 FLAKY 17/21 upstream reliability is '100.0'. current run reliability is '80.95238095238095'. drift is 19.04762 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cb-4bf0-bc58-04233091cd30 FLAKY 13/21 upstream reliability is '100.0'. current run reliability is '61.904761904761905'. drift is 38.09524 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": false, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-4fbe-8958-b7908eb9acd4 FLAKY 18/21 upstream reliability is '100.0'. current run reliability is '85.71428571428571'. drift is 14.28571 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c5-47de-bb8d-73c9ec24b03f FAIL 0/1
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c5-4bd6-8a24-53a8d6de9620 FLAKY 15/21 upstream reliability is '100.0'. current run reliability is '71.42857142857143'. drift is 28.57143 and the allowed drift is set to 50. The test should PASS
RandomNodeOperationsTest test_node_operations {"cloud_storage_type": 2, "enable_failures": true, "mixed_versions": true, "with_chunked_compaction": true, "with_iceberg": false} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61c7-4396-b2ae-a9240030828f FLAKY 15/17
DisablingPartitionsTest test_disable ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61be-4710-9727-0862365d738e FLAKY 17/21 upstream reliability is '77.87610619469027'. current run reliability is '80.95238095238095'. drift is -3.07627 and the allowed drift is set to 50. The test should PASS
DisablingPartitionsTest test_disable ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61bf-489d-8a22-d48aaf2ea40c FLAKY 18/21 upstream reliability is '77.87610619469027'. current run reliability is '85.71428571428571'. drift is -7.83818 and the allowed drift is set to 50. The test should PASS
OIDCLicenseTest test_license_nag {"authn_config": {"http_authentication": ["OIDC", "BASIC"]}} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b8-4247-b682-80955b5ea2dc FLAKY 20/21 upstream reliability is '97.34848484848484'. current run reliability is '95.23809523809523'. drift is 2.11039 and the allowed drift is set to 50. The test should PASS
RedpandaOIDCTlsTest test_admin_revoke ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b9-485e-86ea-e63bc84f891b FLAKY 20/21 upstream reliability is '99.59016393442623'. current run reliability is '95.23809523809523'. drift is 4.35207 and the allowed drift is set to 50. The test should PASS
SaslPlainTest test_plain_authn {"client_type": 1, "sasl_plain_enabled": false, "scram_type": 2} ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61cc-4a56-869a-e9413a1b654e FLAKY 20/21 upstream reliability is '98.85496183206108'. current run reliability is '95.23809523809523'. drift is 3.61687 and the allowed drift is set to 50. The test should PASS
TimeQueryKafkaTest test_timequery ducktape https://buildkite.com/redpanda/redpanda/builds/66532#01971920-61b9-49ba-ae44-eee784d9a737 FLAKY 20/21 upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS

@andrwng
Copy link
Contributor Author

andrwng commented May 28, 2025

/ci-repeat 10

@andrwng
Copy link
Contributor Author

andrwng commented May 28, 2025

/ci-repeat 3 dt-repeat=4 skip-redpanda-build skip-units

@andrwng
Copy link
Contributor Author

andrwng commented May 28, 2025

/ci-repeat 10 skip-redpanda-build skip-units

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented May 29, 2025 •
edited
Loading

Retry command for Build#66532

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/blocked_catalog_test.py::DatalakeBlockedCatalogTest.test_block_cloud_retention_before_translation@{"cloud_storage_type":1,"with_spillover":false}
tests/rptest/tests/datalake/blocked_catalog_test.py::DatalakeBlockedCatalogTest.test_trim_local_before_first_translation@{"cloud_storage_type":1}
tests/rptest/tests/datalake/blocked_catalog_test.py::DatalakeBlockedCatalogTest.test_block_cloud_retention_before_translation@{"cloud_storage_type":1,"with_spillover":true}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/build area/redpanda

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@andrwng @vbotbuildovich