Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

archival: Start housekeeping jobs after STM sync #14599

Merged
merged 1 commit into from
Jan 16, 2024
Merged

archival: Start housekeeping jobs after STM sync #14599

merged 1 commit into from
Jan 16, 2024

Conversation

Lazin
Copy link
Contributor

@Lazin Lazin commented Nov 1, 2023

Previously, the housekeeping jobs were started in the c-tor of the ntp_archiver. This allowed them to 'see' potentially stale manifest. To avoid this this commit forces STM sync in the 'ntp_archiver::start' and enables housekeeping jobs after that.

Fixes #14222

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.2.x
  • v23.1.x
  • v22.3.x

Release Notes

  • none

@Lazin Lazin requested a review from VladLazar November 1, 2023 13:17
@Lazin Lazin force-pushed the fix/ci-14222-delay-reuploads branch from 9577fed to 4d0b3de Compare November 1, 2023 13:22
VladLazar
VladLazar reviewed Nov 1, 2023
View reviewed changes
Copy link
Contributor

@VladLazar VladLazar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good change, but isn't this problem more general? A housekeeping job may try to replicate a command at any time. It will sync with the log before replicating, but the decisions which led to the replication of the command may have been made on stale data.

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b3e-bd67-42ad-8405-8289aaf8f68b: "rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.read_replica_e2e_test.ReadReplicasUpgradeTest.test_upgrades.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_features_test.FeaturesSingleNodeTest.test_get_features"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_move_consumer_offsets_intranode.num_to_upgrade=2"
"rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_rebalancing_node.shutdown_decommissioned=False"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=True.tick_interval=3600000"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.nodes_decommissioning_test.NodeDecommissionFailureReportingTest.test_allocation_failure_reporting"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=True"
"rptest.tests.workload_upgrade_runner_test.RedpandaUpgradeTest.test_workloads_through_releases.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.acls_test.AccessControlListTestUpgrade.test_upgrade_sasl"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_during_upgrade.empty_seed_starts_cluster=False"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default_explicit_after_upgrade.wipe_cache=False"
"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_explicit_activation"
"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_upgrade"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_empty.num_to_upgrade=2"
"rptest.tests.topic_creation_test.CreateTopicUpgradeTest.test_retention_config_on_upgrade_from_v22_2_to_v22_3.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.transactions_test.GATransaction_v22_1_UpgradeTest.upgrade_coordinator_test"

@Lazin Lazin self-assigned this Nov 1, 2023
@Lazin
Copy link
Contributor Author

Lazin commented Nov 1, 2023

@VladLazar the adjacent segment merger uses locking to avoid this issue. There is a mutual exclusion between the ASM and ntp_archiver so it will never use stale data after the initial sync.

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b3e-bd6e-48c7-9ffd-d6c3f6cc9561: "rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add"
"rptest.tests.upgrade_test.UpgradeFromPriorFeatureVersionTest.test_basic_upgrade"
"rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.upgrade_test.RedpandaInstallerTest.test_install_by_line"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=False.tick_interval=3600000"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation.delete_topic=False"
"rptest.tests.partition_movement_upgrade_test.PartitionMovementUpgradeTest.test_basic_upgrade"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_flipping_decommission_recommission.node_is_alive=False"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_recommissioning_one_of_decommissioned_nodes"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_dynamic.num_to_upgrade=2"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_node_is_not_allowed_to_join_after_restart.new_bootstrap=True"
"rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade_with_rollback.upgrade_after_rollback=True"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_synthetic_old_node_join"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_after_upgrade.empty_seed_starts_cluster=False"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default.wipe_cache=False"
"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_license_upload_and_query"
"rptest.tests.cluster_features_test.FeaturesUpgradeAssertionTest.test_upgrade_assertion"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"
"rptest.tests.pandaproxy_test.BasicAuthUpgradeTest.test_upgrade_and_enable_basic_auth.base_release=.22.3.next_release=.23.1"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_invalid_destination.num_to_upgrade=2"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_create_topic_with_single_configuration_property"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b3e-bd72-40ea-abe7-7ebe3a45d7ba: "rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_and_upgrade"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=False.tick_interval=5000"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation.delete_topic=True"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_flipping_decommission_recommission.node_is_alive=True"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_synthetic_too_new_node_join"
"rptest.tests.controller_snapshot_test.ControllerSnapshotTest.test_upgrade_compat"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_deletion_stops_move.num_to_upgrade=2"
"rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=False"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_after_upgrade.empty_seed_starts_cluster=True"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default.wipe_cache=True"
"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_rollback"
"rptest.tests.controller_snapshot_test.ControllerSnapshotPolicyTest.test_upgrade_auto_enable"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_moving_not_fully_initialized_partition.num_to_upgrade=2"
"rptest.tests.upgrade_test.UpgradeFromSpecificVersion.test_basic_upgrade"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b3e-bd6b-409e-92d8-262658de307e: "rptest.tests.cluster_features_test.FeaturesSingleNodeUpgradeTest.test_upgrade"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_crashed_node"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_rebalancing_node.shutdown_decommissioned=True"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=True.tick_interval=5000"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_recommissioning_node_finishes"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_bootstrapping_after_move.num_to_upgrade=2"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_overlapping_changes.num_to_upgrade=2"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_node_is_not_allowed_to_join_after_restart.new_bootstrap=False"
"rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade_with_rollback.upgrade_after_rollback=False"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_old_node_join"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_static.num_to_upgrade=2"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.upgrade_test.UpgradeFromPriorFeatureVersionCloudStorageTest.test_rolling_upgrade.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_during_upgrade.empty_seed_starts_cluster=True"
"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_get_features"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default_explicit_after_upgrade.wipe_cache=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.license_upgrade_test.UpgradeToLicenseChecks.test_basic_upgrade.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.pandaproxy_test.BasicAuthUpgradeTest.test_upgrade_and_enable_basic_auth.base_release=.22.2.next_release=.22.3"
"rptest.tests.offset_retention_test.OffsetRetentionDisabledAfterUpgrade.test_upgrade_from_pre_v23.initial_version=.22.2.9"
"rptest.tests.topic_creation_test.CreateTopicUpgradeTest.test_retention_upgrade_with_cluster_remote_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.transactions_test.GATransaction_v22_1_UpgradeTest.upgrade_topic_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b5d-cc35-4e6c-b731-d17d257657ad: "rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add"
"rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.upgrade_test.UpgradeFromPriorFeatureVersionTest.test_basic_upgrade"
"rptest.tests.upgrade_test.RedpandaInstallerTest.test_install_by_line"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_after_upgrade.empty_seed_starts_cluster=False"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=False.tick_interval=3600000"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation.delete_topic=False"
"rptest.tests.partition_movement_upgrade_test.PartitionMovementUpgradeTest.test_basic_upgrade"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_flipping_decommission_recommission.node_is_alive=False"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_recommissioning_one_of_decommissioned_nodes"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default.wipe_cache=False"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_license_upload_and_query"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.cluster_features_test.FeaturesUpgradeAssertionTest.test_upgrade_assertion"
"rptest.tests.nodes_decommissioning_test.NodeDecommissionSpaceManagementTest.test_decommission.single_partition=True"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_dynamic.num_to_upgrade=2"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_node_is_not_allowed_to_join_after_restart.new_bootstrap=True"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade_with_rollback.upgrade_after_rollback=True"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_synthetic_old_node_join"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"
"rptest.tests.pandaproxy_test.BasicAuthUpgradeTest.test_upgrade_and_enable_basic_auth.base_release=.22.3.next_release=.23.1"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_invalid_destination.num_to_upgrade=2"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_create_topic_with_single_configuration_property"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b5d-cc38-477d-b21b-e39d710ce410: "rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_and_upgrade"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_finishes_after_manual_cancellation.delete_topic=True"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=False.tick_interval=5000"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_flipping_decommission_recommission.node_is_alive=True"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage.cleanup_policy=compact.delete"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_synthetic_too_new_node_join"
"rptest.tests.memory_stress_test.MemoryStressTest.test_fetch_with_many_partitions.memory_share_for_fetch=0.05"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.controller_snapshot_test.ControllerSnapshotTest.test_upgrade_compat"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_deletion_stops_move.num_to_upgrade=2"
"rptest.tests.raft_recovery_test.RaftRecoveryUpgradeTest.test_upgrade"
"rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=False"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_after_upgrade.empty_seed_starts_cluster=True"
"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_rollback"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default.wipe_cache=True"
"rptest.tests.controller_snapshot_test.ControllerSnapshotPolicyTest.test_upgrade_auto_enable"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_moving_not_fully_initialized_partition.num_to_upgrade=2"
"rptest.tests.upgrade_test.UpgradeFromSpecificVersion.test_basic_upgrade"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b5d-cc32-4789-bd7e-f4129ae759c9: "rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete"
"rptest.tests.cluster_features_test.FeaturesSingleNodeUpgradeTest.test_upgrade"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_crashed_node"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=True.tick_interval=5000"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_rebalancing_node.shutdown_decommissioned=True"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_recommissioning_node_finishes"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_bootstrapping_after_move.num_to_upgrade=2"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_node_is_not_allowed_to_join_after_restart.new_bootstrap=False"
"rptest.tests.nodes_decommissioning_test.NodeDecommissionSpaceManagementTest.test_decommission.single_partition=False"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_overlapping_changes.num_to_upgrade=2"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade_with_rollback.upgrade_after_rollback=False"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_old_node_join"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_static.num_to_upgrade=2"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.upgrade_test.UpgradeFromPriorFeatureVersionCloudStorageTest.test_rolling_upgrade.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_during_upgrade.empty_seed_starts_cluster=True"
"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_get_features"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default_explicit_after_upgrade.wipe_cache=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.license_upgrade_test.UpgradeToLicenseChecks.test_basic_upgrade.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.pandaproxy_test.BasicAuthUpgradeTest.test_upgrade_and_enable_basic_auth.base_release=.22.2.next_release=.22.3"
"rptest.tests.offset_retention_test.OffsetRetentionDisabledAfterUpgrade.test_upgrade_from_pre_v23.initial_version=.22.2.9"
"rptest.tests.rpk_registry_test.RpkRegistryTest.test_produce_consume_proto"
"rptest.tests.topic_creation_test.CreateTopicUpgradeTest.test_retention_upgrade_with_cluster_remote_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.transactions_test.GATransaction_v22_1_UpgradeTest.upgrade_topic_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40238#018b8b5d-cc2f-4ca9-af1a-3a63618e703e: "rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.read_replica_e2e_test.ReadReplicasUpgradeTest.test_upgrades.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_features_test.FeaturesSingleNodeTest.test_get_features"
"rptest.tests.controller_upgrade_test.ControllerUpgradeTest.test_updating_cluster_when_executing_operations"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.acls_test.AccessControlListTestUpgrade.test_upgrade_sasl"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.num_to_upgrade=0.with_tiered_storage=False"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.num_to_upgrade=0.with_tiered_storage=False"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_cancel_ongoing_movements"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_rebalancing_node.shutdown_decommissioned=False"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_decommissioning_working_node.delete_topic=True.tick_interval=3600000"
"rptest.tests.nodes_decommissioning_test.NodesDecommissioningTest.test_multiple_decommissions"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_move_consumer_offsets_intranode.num_to_upgrade=2"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.upgrade_test.UpgradeWithWorkloadTest.test_rolling_upgrade"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_recover_after_delete_records"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.nodes_decommissioning_test.NodeDecommissionFailureReportingTest.test_allocation_failure_reporting"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.node_folder_deletion_test.NodeFolderDeletionTest.test_deleting_node_folder"
"rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=True"
"rptest.tests.workload_upgrade_runner_test.RedpandaUpgradeTest.test_workloads_through_releases.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_bootstrap_test.ClusterBootstrapUpgrade.test_change_bootstrap_configs_during_upgrade.empty_seed_starts_cluster=False"
"rptest.tests.cluster_config_test.ClusterConfigLegacyDefaultTest.test_legacy_default_explicit_after_upgrade.wipe_cache=False"
"rptest.tests.cluster_features_test.FeaturesMultiNodeTest.test_explicit_activation"
"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_upgrade"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.partition_movement_test.PartitionMovementTest.test_empty.num_to_upgrade=2"
"rptest.tests.topic_creation_test.CreateTopicUpgradeTest.test_retention_config_on_upgrade_from_v22_2_to_v22_3.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.transactions_test.GATransaction_v22_1_UpgradeTest.upgrade_coordinator_test"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b905f-4dc4-435d-a32c-ed2e82705cb8: "rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.rpk_registry_test.RpkRegistryTest.test_produce_consume_avro"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b905f-4dca-4b38-b77e-a779e8a0d24b: "rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b905f-4dcd-4b7c-8577-4dca9261d475: "rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b905f-4dc7-4a6e-98e1-b1f4b21de4da: "rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.rpk_registry_test.RpkRegistryTest.test_produce_consume_proto"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b906e-41db-45ab-9370-4c7a9a156e51: "rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_create_topic_with_single_configuration_property"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b906e-41dd-47a2-a0ec-1c8338c4ad28: "rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage.cleanup_policy=compact.delete"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b906e-41d8-4eab-ac5c-ebc985787234: "rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"

@vbotbuildovich
Copy link
Collaborator

new failures detected in https://buildkite.com/redpanda/redpanda/builds/40312#018b906e-41d5-4b62-8396-7e6d4a0fb437: "rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.rpk_registry_test.RpkRegistryTest.test_produce_consume_avro"

@nvartolomei
Copy link
Contributor

When ntp_archiver::notify_leadership is called is it guaranteed that the STM is up to date?

@VladLazar
Copy link
Contributor

When ntp_archiver::notify_leadership is called is it guaranteed that the STM is up to date?

Not as as far as I'm aware.

@nvartolomei
Copy link
Contributor

nvartolomei commented Nov 3, 2023 •
edited

@VladLazar another possible race source then? As it starts the uploader/merger loop there.

@VladLazar
Copy link
Contributor

@VladLazar another possible race source then? As it starts the uploader/merger loop there.

The jobs are enabled after the call to archival_metadata_stm::sync (change in this pr), so it should be fine at start-up.

@nvartolomei
Copy link
Contributor

@VladLazar can you point to the file:lineno where sync call is made from ntp_archiver::notify_leadership? I can't see it.

@VladLazar
Copy link
Contributor

VladLazar commented Nov 6, 2023 •
edited

@VladLazar can you point to the file:lineno where sync call is made from ntp_archiver::notify_leadership? I can't see it.

ntp_archiver::notify_leadership signals _leader_cond. This will wake up ntp_archiver::upload_until_abort. When that happens, we check for leadership via may_begin_uploads and then call archival_metadata_stm::sync. That's on dev. In this PR a new sync call is added in ntp_archiver::start.

VladLazar
VladLazar reviewed Nov 6, 2023
View reviewed changes
src/v/archival/ntp_archiver_service.cc
// can only see up to date manifest.
auto sync_timeout = config::shard_local_cfg()
.cloud_storage_metadata_sync_timeout_ms.value();
co_await _parent.archival_meta_stm()->sync(sync_timeout);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nvartolomei👋

@nvartolomei
Copy link
Contributor

@VladLazar assuming upload loop is fine, notify_leadership also re-enabled the merger and it doesn't have a call to sync (this PR adds one only at archival start). Struggle to reason about these code paths a bit.

@Lazin Lazin force-pushed the fix/ci-14222-delay-reuploads branch from 4d0b3de to 9a378e6 Compare December 21, 2023 19:02
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Dec 21, 2023 •
edited

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e10-6018-4346-8a7b-1e750869e9f3:

"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_with_leadership_transfers"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_bootstrap_with_recovery"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e08-e9d5-44bc-86ae-1439494777fc:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_basic_controller_snapshot_restore"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_recover_after_delete_records"
"rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e08-e9d9-4edd-8313-60370b594c2d:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact"
"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingManyPartitionsTest.test_many_partitions_recovery"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_bootstrap_with_recovery"
"rptest.tests.consumer_group_recovery_test.ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e08-e9df-4ac5-9e13-0188c9faf589:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_leader_acks"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_with_leadership_transfers"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_recover"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e08-e9dc-4785-979e-fbdd9b55cd80:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage.cleanup_policy=compact.delete"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_create_topic_with_single_configuration_property"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e10-6022-4ae3-b53d-fc4385e231ad:

"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_basic_controller_snapshot_restore"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e10-601f-4a6e-82ad-b824c0185aa2:

"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/43254#018c8e10-601c-4558-99a2-70234aaa26bb:

"rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.consumer_group_recovery_test.ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_create_topic_with_single_configuration_property"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda55-fc59-4bc6-baec-008a32b913d2:

"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_with_leadership_transfers"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_bootstrap_with_recovery"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda55-fc55-4d8b-850d-6f4d30402fc2:

"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.recovery_mode_test.RecoveryModeTest.test_recovery_mode"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_basic_controller_snapshot_restore"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda55-fc5c-42c5-bdff-c0afaa8e9230:

"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.consumer_group_recovery_test.ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda55-fc5f-4c7f-b9cc-7c615cc0e3af:

"rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.recovery_mode_test.DisablingPartitionsTest.test_disable"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"
"rptest.tests.topic_creation_test.CreateTopicsTest.test_create_topic_with_single_configuration_property"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda66-e74d-4e37-bf56-b49551177956:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.num_to_upgrade=0.with_tiered_storage=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=False"
"rptest.tests.e2e_iam_role_test.ShortLivedCredentialsTests.test_short_lived_credentials"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_leader_acks"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.usage_test.UsageTestCloudStorageMetrics.test_usage_manager_cloud_storage"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_basic_controller_snapshot_restore"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_recover_after_delete_records"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_high_watermark"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda66-e750-4028-bfb7-74302c15324d:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.e2e_shadow_indexing_test.ShadowIndexingManyPartitionsTest.test_many_partitions_recovery"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cluster_recovery_test.ClusterRecoveryTest.test_bootstrap_with_recovery"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage.cleanup_policy=compact.delete"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_basic_follower_fetching.read_from_object_store=True"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.consumer_group_recovery_test.ConsumerOffsetsRecoveryTest.test_consumer_offsets_partition_recovery"
"rptest.tests.multi_restarts_with_archival_test.MultiRestartTest.test_recovery_after_multiple_restarts.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.recovery_mode_test.RecoveryModeTest.test_recovery_mode"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.services_self_test.BucketScrubSelfTest.test_missing_segment.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_bounds_checking.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=at_segment_boundary"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_policy_change.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_with_transactions.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda66-e749-42bc-ac08-1bb38688c567:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact.delete"
"rptest.tests.topic_creation_test.CreateSITopicsTest.topic_alter_config_test"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=20.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_iam_role_test.STSRoleFetchTests.test_write"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_write.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_with_leadership_transfers"
"rptest.tests.internal_topic_protection_test.InternalTopicProtectionLargeClusterTest.test_schemas_topic"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_recover"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_compacted_topic.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_empty_or_missing_topic_or_partition.cloud_storage_enabled=True"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=random_offset"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_topic_start_delta.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/43484#018cda66-e753-42f1-9b32-f722f16b167c:

"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.FAST_MOVES.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=node_add.test_mode=TestMode.NO_TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.node_pool_migration_test.NodePoolMigrationTest.test_migrating_redpanda_nodes_to_new_pool.balancing_mode=off.test_mode=TestMode.TIRED_STORAGE.cleanup_policy=compact"
"rptest.tests.cloud_retention_test.CloudRetentionTest.test_cloud_retention.max_consume_rate_mb=None.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_iam_role_test.AWSRoleFetchTests.test_write"
"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage.cleanup_policy=delete"
"rptest.tests.cloud_storage_usage_test.CloudStorageUsageTest.test_cloud_storage_usage_reporting_with_partition_moves"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestCompactedTopic.test_compacting_during_leadership_transfer.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.follower_fetching_test.FollowerFetchingTest.test_follower_fetching_with_maintenance_mode"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=2.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTest.test_reset_spillover.cloud_storage_type=CloudStorageType.S3"
"rptest.tests.shadow_indexing_compacted_topic_test.ShadowIndexingCompactedTopicTest.test_upload.cloud_storage_type=CloudStorageType.ABS"
"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_segment_deletion.cloud_storage_enabled=True.truncate_point=one_below_high_watermark"

@Lazin Lazin force-pushed the fix/ci-14222-delay-reuploads branch from 9a378e6 to 8c8551a Compare January 5, 2024 15:04
@Lazin
archival: Start housekeeping jobs after STM sync
f4cbe6c 
Previously, the housekeeping jobs were started in the c-tor of the
ntp_archiver. This allowed them to 'see' potentially stale manifest. To
avoid this this commit forces STM sync in the 'ntp_archiver::start' and
enables housekeeping jobs after that.
@Lazin Lazin force-pushed the fix/ci-14222-delay-reuploads branch from 8c8551a to f4cbe6c Compare January 15, 2024 13:23
@andijcr andijcr self-requested a review January 15, 2024 15:44
@vbotbuildovich
Copy link
Collaborator

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/43771#018d0d9f-6ae6-499e-8ef6-484dbf9aa574

@Lazin Lazin merged commit 252d47b into redpanda-data:dev Jan 16, 2024
19 checks passed
@Lazin
Copy link
Contributor Author

Lazin commented Jan 16, 2024

/backport v23.3.x

This was referenced Jan 16, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@VladLazar VladLazar VladLazar left review comments

@andijcr andijcr andijcr approved these changes

@nvartolomei nvartolomei Awaiting requested review from nvartolomei

Assignees

@Lazin Lazin

Labels
area/redpanda
Projects
None yet
Milestone
No milestone
5 participants
@Lazin @vbotbuildovich @nvartolomei @VladLazar @andijcr